Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigates how language models allocate their probability mass, with an emphasis on rare sequences that are part of the 'heavy tail' of the distribution of natural language sequences. The authors use a language model to define a target distribution with access to samples and ground-truth probabilities. A language model is trained on an empirical estimate of the target distribution, and the authors study the gap between the learned model's and target distribution's probability assignments. 

Using this methodology, the authors uncover various interesting phenomena: the model systematically assigns lower probabilities than the target distribution, but assigns unusually high probabilities to unnatural, perturbed sequences (suggesting an explanation for where the probability mass moved to). The authors include several fine-grained analyses with additional interesting findings.
main_review: **Interesting background**. The connections between productivity, low-probability sequences, and the limitations of perplexity were clearly written and interesting.

**Well-executed experiments and methodology**. The idea of using a language model as a ground-truth distribution was interesting and well-suited for the analysis done here. The experiments were easy to follow and the analysis was clear.

**Interesting findings**. The main findings related to underestimation, comparison of training-set and test-set dynamics during training, and dependence on training data were interesting and not obvious. The finding that perturbed, unnatural sequence received unusually high probabilities (while test samples receive unusually low probabilities) [Figure 4] was especially interesting. 

**Clarity**. The paper was well-written and enjoyable to read. The authors articulated how their work fits in with related work, and the concepts (e.g. the LNRE zone), methodology, and results are clearly written. Well done.

**Additional experiments**: A good addition would be measuring the impact of model size with models larger than the target distribution. What would GPT2-XL's or GPT-3's probability assignments look like on a target distribution from GPT2-medium?
summary_of_the_review: This investigatory paper defines the problem that they are studying, develops a methodology for studying it, and clearly analyzes the results. The investigation yields interesting findings related to language models. This paper would make a great addition to ICLR - accept.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes to examine the probability mass that a language model trained as usual places on frequent/likely and rare/unlikely sentences by learning from a distribution that is another neural language model itself so instance-level probabilities/NLLs of sentences can be compared. It turns out that all models in all tested circumstances *underestimate* the probabilities of rare events---and an ablation study shows that much of that missing mass can be found in sequences that are also rare and unlikely, but that weren't likely under the origin model.
main_review: The paper overall is exceptionally well-written and clear and tables and plots are beautiful and clear---multiple times did I go "ooh that's a nice way of laying it out." Kudos! The abstract tells the entire story and the paper substantiates the claims nicely making what is shown feel easy and obvious, which I consider a great feat.

Concerning content, I am equally enthusiastic about the idea and evaluation and agree that this can be a very interesting piece of evidence that nicely connects to a number of past threads of NLP research (as laid out throughout and in the nice related work section, though I don't think I know enough related literature to truly safely say that the paper is as novel as it claims to be). That said, there are three major methodological concerns that I would like to discuss through with the authors before recommending acceptance:

1. We do not know whether neural-LM-defined origin distributions are "strange"---or really, structurally different from actual human natural language distributions that we believe exist---in some systematic way that leads to the results in this paper. One potential intuition here is that a neural LM being a very imperfect model makes very erratic choices for what low-frequency/low-likelihood it assigns probability mass to---erratic choices that may be very hard to learn/emulate with the student LM that the authors train. Maybe real natural language distributions have more "sensible tails" that are actually not all that hard to learn and maybe not even underestimated. To be clear, this is a fundamental issue that the authors have no way of *resolving* easily under this paradigm and I think the paradigm is still a worthwhile idea! What I am asking is that this limitation is acknowledged and discussed with the same clarity that is afforded to the other content.

2. Relatedly, currently there is no way to know whether the trained models are actually reasonably trained, overall well-working models or not: the paper is missing test ppl/bpc for both the origin model (on English test data) and for the student model (both on the original English data and on test data sampled from the origin model). The authors make a good point that these scores are not as informative as what this paper presents, but the reason these are crucial is so we know whether the effects the authors find are a property of well-trained models or whether these models (through subpar training or another issue) are just poor models that have little connection to the models we actually use in reality. A secondary idea to easy this concern might be to actually use a pretrained GPT model as the origin model to inspire trust that that at least is a good model---I'm not sure why this road wasn't chosen unless it truly did not matter and this "home-made" model performs just as well, in which case, again, I would like to see some quantitative substantiation of that claim.

3. The paper is not really clear on how the sampling/teaching procedure and the tempering of the softmax interact: Footnote 2 claims pure (right?) ancestral sampling is used (as opposed to top-k/nucleus?), but then Footnote 4 says the softmaxes are all cooled to T=0.85, and yet Section 4.5 again implies that no tempering took place up to that point in the study. This is not only slightly confusing messaging, but I am worried that it may belie the promise of fair evaluation, especially because it isn't clear to me whether the tempering was accounted for in the calculation of probabilities as defined at the end of section 3.2. Specifically, if the probability of a sentence under the origin model does not take into account this local tempering, then it is no surprise that the student model learns to underestimate rare events because they just weren't sampled thanks to the locally cooled distribution! I would be somewhat reassured if the probability calculation in section 3.2 takes the tempering into account, but my understanding of the paper is that it currently fails to do so. As slight reassurance, Figure 5 does seem to tell us that even without any tempering (yes?) the described effect is visible, so I think this too should not be an issue that sinks the paper, but I do see it as a critical flaw the way it is right now.

Beyond those three, I see some minor concerns that may be worth addressing:
- "autoregressive neural language models" should also cite the Mikolov RNN paper, perhaps in place of one of the two GPT papers
- not to Schmidhuber, but citing only Melis et al. (2020) for LSTMs looks a little odd (citing Hochreiter & Schmidhuber or Sundermeyer et al. may be less surprising though I understand the desire to cite the *actual* model and training specification used and agree it should be present)
- concerning citations, Shiran Dudy & Steven Bedrick's work may be nice to also connect to (despite its relatively low visibility)
- it may be worth pointing out that the perturbations in section 4.4 may individually also make sentences *more* grammatical and likely and that is just a relatively rare outcome (I don't think much more than that needs to be said on that)
- I did not quite see what was "[p]erhaps unsurprisin[g]" in Section 4.5, but that's a minor nitpick 
- Appendix A.1 measures events, okay... but are those words (as the hapax legomena theory usually assumes) or are they sentences (as I think this experiment does)? If they are sentences, we should not be surprised to see very high probabilities of seeing novel events, after all sentences are rarely the same (excluding formulaic language and "Resumption of the session" style sentences). I also would have liked to see some Poisson distributions or other distributions that were more interesting than uniform n-sided dice in Figure 6, but that is more wish than necessity.
summary_of_the_review: Before author response:

My comparatively low score is a consequence of my concerns and very much given as a temporary score until we can discuss the three major concerns I have---if they are addressed satisfactorily, I anticipate raising my score significantly to champion the paper if needed as I found this paper a joy to read and thought-provoking in a good way (even if that lead to criticism).


After author response:

Thank you for all the clarifications and edits, they are greatly appreciated. I would strongly like to see this paper accepted.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper conducts experiments to evaluate whether neural sequence models such as LSTMs and Transformers are able to correctly assess the probability of rare sentences, which collectively constitute a large probability mass in natural language productions (heavy-tail phenomenon). In order to do so, it performs experiments in a controlled synthetic environment where a first language model $P_L$ is trained on a corpus of natural sentences, and a second model $P_M$ is trained to emulate the first model. The authors observe that the second model systematically tends to underestimate the probability of rare $P_L$ sentences, the more so the rarer such sentences are, and show that some artificially corrupted sentences tend to receive higher probability from $P_M$ than from $P_L$, partly explaining where the missing probability mass over rare well-formed sentences went.


main_review: * Strengths: A very well written and clear paper, addressing a focussed question through a well-designed set of experiments on synthetic data allowing to precisely control experimental conditions.

* Weaknesses: No major weakness, but some mild issues and areas for improvement, see the comments and questions below.

* Comments and Questions:

    * (Minor Comment) I did not find the description of the LNRE Zone in section 2 to bring much to the discussion. In particular, characterizing this zone through the condition that $\mathcal{P}_N$ is not __null__ is almost empty of content for natural language, because it would in particular mean that we could precisely delimit what constitutes a possible "natural" sentence, and that the LNRE Zone should cover all such sentences. Here the formal description as given does not clarify anything.
    * (Question) In some of your experiments, you train a $P_M$ GPT2-medium model on 30M sentences sampled from the $P_L$ GPT2-medium model, that is, $P_M$ here uses the same architecture as $P_L$ (if I understand correctly). However, the $P_M$ model still underestimates the probabilities relative to $P_L$. Do you think this is a symptom of 30M sentences being too few, or more a symptom that $P_M$ is stuck in some local minimum during the optimization ? After all, (again, if I understand the setup correctly), using the same architecture, $P_M$ would make it possible, in principle, to just use the same parameters as $P_L$, and then the mismatch could not occur. Perhaps this would be worth a note in the text?
    * (More important Question/Comment) In section 4.4, you perform some perturbations, and observe that they lead to higher $P_M$ than $P_L$. First, I think it would be worth defining these perturbations in more detail than what you do. Second, you do not discuss your choice for these perturbations nor analyze why they have (relatively) high probability relative to $P_M$. However, the fact that, for instance, swapping two words or deleting a word in a natural sentence is not severely penalized by $P_M$ is an interesting but specific fact, which may have to do with a special tolerance of $P_M$ for such operations, rather than for others. To give one example, if the perturbations consisted solely in replacing a word in the natural sentence by a _rare_ word in the training set, would you observe such a difference between $P_M$ and $P_L$? Overall, could you discuss this aspect of your experiments?


summary_of_the_review: A nice experimental paper addressing the difficulty of neural LMs to approximate the probability of rare events from an underlying "teacher" LM. The paper could be improved by more analysis and discussion of some of the results.

*After authors' response*: thank you for your detailed answers to my questions and for your additions to the paper and to experiments. I aim raising my score and hope the paper will get accepted.
review_title: null

Meta-review: This paper presents a very interesting study of using an artificial language (generated using a specific algorithm via a transformer model) and training SOTA transformer and LSTM language models on that language;  the authors show that these LMs underestimate the probability of sequences from this language and overestimate the probability of ill formed sentences, among other observations.  This is a very interesting study that captures the behavior of recent LMs.  All reviewers are supportive of accepting this paper and it is good to see the engagement between reviewers and authors of this paper.
Decision: Accept (Poster)