What Kinds of Tokens Benefit from Distant Text? An Analysis on Long Context Language Modeling
Authors: Yutong Hu, Quzhe Huang, Kangcheng Luo, Yansong Feng


Reviewer paper summaries: This paper discusses the influence of context length on language modeling. The authors find that a longer context provides smaller perplexity, which kinds of words benefit from the long context, and if the confidence of language modeling increases with long context.
Summary of strengths: This paper explores an interesting and important research question: how does the manner of the language model change if the input context is long? Three aspects are discussed by comparing the model manner with that of humans, by researching words in different types, and by observing the confidence change.

The experiments cover different models and context lengths.
Summary of weaknesses: First, the choice of the metric is a bit problematic. Even if perplexity can reveal the language modeling condition on text, it’s not suitable to combine it with context length. Specifically, we calculate the perplexity of one same sentence, but condition 1 only contains that sentence, while condition 2 contains many background/similar text to that sentence. Apparently, condition 2 would probably have lower perplexity since the model is more familiar with the writing styles, word usages with the sentence. This is the experiment setting in the paper, where text from the same document is used to measure the input length. Then, it is obvious that the perplexity will decrease with a more similar context. And it does not matter with the length at all. If the authors wan to evaluate the context length, sentences from other documents should be used. But I guess the perplexity would increase then.

Second, still corresponding to the first point, the findings in Figure 2 can reveal that the perplexity of nouns decreases significantly. This may just because that longer context helps the model to capture the semantic background information and get more familiar with the text style e.g., word usage. Also, I guess if the context contains sentences from other documents, the results might reverse.

Overall, I think the results of this work cannot support the point that the authors want to make. I would suggest that the authors to do more experiment (e.g., increasing context using some independent text) or rewrite the story.
Comments, suggestions and typos: Line 173: whitespace before “After…”
Overall assessment scores: 2