Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new self-terminating Language Model (LM). The authors aim to address the issue of non-termination in current LMs and compete with previously proposed Monotonic Self-Terminating LM by relaxing the monotonically increasing condition. To achieve the said relaxation authors propose a new parametrization technique (in place of softmax) for LM's classifier and encourage the termination probability to converge to 1. Further, the authors prove that under this relaxation, the proposed method still prevents non-terminating sequences resulting from incomplete probable decoding algorithms. 
Experimentation is conducted by training RNN and LSTM on WikiText-2 and fine-tuning GPT-2 on WikiText-103 and show that the proposed method works better than previous Self-Terminating (ST) LM.
strength_and_weaknesses: Strengths:

+ The motivation is well-founded.
+ Proposed method is novel and builds upon previously proposed ST LM.
+ Experiments show that the proposed method works better than baselines and authors also show that non-monotonicity can better model the termination probability.
+ The ideas in the paper are very well presented and the writing is clear.

Weaknesses / Questions:

I feel the proposed method is quite well formulated and presented and I have no reason to reject this work. However, I feel some of the points should be addressed to further improve the quality of this work and make it well-rounded. It is also understandable that asking authors to perform a huge amount of experiments during rebuttal is unfair. Hence, not performing the experiments proposed below will not affect the paper's rating negatively. 

- Experimentation is done for sequence completion tasks, and it shows that the proposed method can perform better than vanilla models with the correct choice of hyperparameters. However, I would further like to know the performance of self-terminating language models on other language tasks such as Machine Translation, Question Answering. The motivation behind this inquiry is that since the probability of distribution of the vocabulary is being changed, it is important to see how this affects the model's performance in tasks where it has to be faithful to the given context.
- It would also be interesting to see how well this method integrates with some of the guided decoding algorithms such as [Krause et al., Findings 2021](https://aclanthology.org/2021.findings-emnlp.424).
 
clarity,_quality,_novelty_and_reproducibility: Overall, this paper is of high quality and presents the ideas with clarity which makes it easy to understand and enjoyable to read. The proposed method is novel and works well in practice.
summary_of_the_review: In this work, the authors proposed a novel non-monotonic self-terminating language model and through extensive experiments show that the proposed method handles the problem of non-terminating sequences better than the baselines. Overall, I feel this paper is of high significance to the language model literature and thus I vote for accepting the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a non-monotonic self-terminating language model (NMST). Language models are known not to be consistent -- not guaranteed to generate `<eos>` -- w.r.t. different sampling strategies including greedy decoding, top-k sampling, nucleus sampling, beam search etc. Previous work, self-terminating language models (ST), proposed monotonically increasing the probability of `<eos>` over time which is problematic as language doesn't necessarily terminate monotonically. The authors propose an extension of this work by making sure that the termination probability (probability of `<eos>` at any given time step *t*) reaches to $1$ as $t \rightarrow \inf$. They formulate the termination probability as a convex combination of $1$ and $1-(1-\epsilon)^t$ where the combination coefficient is derived from the inner product between embeddings of `<eos>` and latent vector at *t*. As $t \rightarrow \inf$, the probability is guaranteed to reach $1$. On two language modeling benchmarks, the authors show that NMST improves consistency when sequence length *L* increases. They show that with a carefully chosen $\epsilon$, NMST slightly improves perplexity as well. Using other decoding algorithms, the difference between vanilla GPT and NMST gets smaller.
strength_and_weaknesses: **Strengths** I find the non-monotonic termination an interesting problem and the convex combination with a monotonically increasing lower-point a nice way of formulating the problem.

**Weaknesses** There are a few places that need more clarification.

1. Could you discuss how practical is your approach given that all recent LLMs excel at generating sequences, as also GPT-2 results being close to NMST? I would be interested in seeing if the same model also helps a downstream task such as summarization but I also believe that is not necessarily within the scope of your work.

2. What is the intuition behind using a sigmoid interpolation rather than using a modified softmax where you use the probability of `<eos>` as the coefficient? Like, In Eq. (10), you can use `1-softmax(<eos>)` and `softmax(<eos>)`. This formulation could be more natural as the language model is already trained to optimize for the softmax distribution.

3. Could you apply your termination adjustment as a post-processing step? Using the above `softmax(<eos>)` based interpolation should be doable as a post-processing step without any training.

4. I think setting $\epsilon$ properly is critical for the model to perform well but it is not clear how. I also found sentences discussing / analyzing $\epsilon$ confusing. For example, in Table-1 you mention the results with $\epsilon=1.0 x 10^{-5}$ is competitive but in text you mention it performs better. You also mention that the same $\epsilon$ in Table-2 gives better perplexity but the intervals between VA+ and NMST+ overlap. Could you please explain how you derived that conclusion?

5. Could you also discuss if your objective over-estimates the probability of `<eos>` for sequences of different lengths? I think this would be a bigger issue in ST as it is monotonic but I am just curious if this is something that you observed. A histogram of *predicted sequence length* vs *ground truth sequence length* would help.

6. "resepct" --> "respect"
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and I found the idea to be novel. There is no accompanied code but I find the convex combination easy to implement in LLMs.
summary_of_the_review: Using a non-monotonic self-termination is important to be consistent with popular decoding algorithms, such as greedy decoding, and capture natural language better. I think the paper is easy to follow and implement. There are also a few clarifications needed.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose non-monotonic self-terminating (NMST) decoding algorithm, improving upon [Welleck,et. al. 2020] ST algorithm.
The paper first show that a language model is non-monotonic if it is trained on dataset that has the same prefix with two different lengths samples.
It also points out that the monotonicity requirements can engender validation perplexity degradation.
In addition, the same requirement may also produce non-terminating sequences.

Thus it introduces a decoding algorithm that is a convex combination of two curves (lower and upper bound) that is non-monotonic, but with the lower bound monotonically increasing but the combination function is not). The algorithm encourages termination probability of each sequence to converge to 1, even it’s not always increasing. 

The author tested it on RNN, LSTM, GPT-2 on vanilla softmax, [Welleck,et. al. 2020] ST softmax, and NMST softmax.
The experiments measure perplexity and varying non-termination ratio, through different mixing contants for lower and upper bound curves.
For the first experiment with WikiText-2 with RNN and LSTM, ti shows that NMST shows better the convergence characteristic, and better validation perplexities for both RNN and LSTM.
The second experiment with larger data WikiText-103 and model GPT-2 also show similar result.
The experiment also goes into the details of the behavior of NMST. One is NMST avoid repetitive tokens and ending with <eos> to create better sequences.
Another is showing how show the non-monotonic nature of the decoding algorithm when an end of sentence is encountered, allows for much more natural terminating spikes.

strength_and_weaknesses: Strengths: well written paper, the intro and equations.

Weaknesses: the evaluation section has 2 experiments, but only 2 very insightful detailed examples. The paper can use a few more examples to illustrate more differences of the output sequences. This would allow the reader to internalize how the non-monotonicity in a deeper way.


Questions:
In details, how does the decoding algorithm actually avoid repetitions?
In other way, how does other models actually degrade validation perplexity using their decoding algorithm?

Typos, Grammar, etc.:
Page 7, section 4.2, par. 2: the callout to table 5 should go to table 3, instead.
Page 7, section 5, last par.: figure 6 callout is not directing properly 

clarity,_quality,_novelty_and_reproducibility: Paper is clear. Even for a person in a different part of NLU (I am in encoding part), the reader would gain a lot of understanding of the problems. More examples, would definitely help, however.
summary_of_the_review: The paper as is is a valuable read. I am adding a few suggestions to improve upon what it already has.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a self-terminating language model, which constrains the family of learnable distributions over sequences to distributions with termination probabilities that converge to 1 as a function of sequence length (Definition 6). The proposed model is evaluated on WikiText-2 (RNN, LSTM architectures) and WikiText-103 (GPT2-small).
strength_and_weaknesses: The proposed non-monotonic self-terminating model (Definition 6) is interesting! I would have liked to see a more thorough discussion and motivation of this model. Most of the methods section is dedicated to background and re-iteration of ideas developed in Welleck et al. (2020) and little room remains to expand upon the core contribution of this work.

The results on WikiText-2 are encouraging. Self-termination is achieved without a sacrificing perplexity. This contrasts with the monotonic models, which achieve self-termination at some cost to perplexity. The WikiText-103 evaluation with GPT-2 is more nuanced, because the 1000-token sequence length appears to be too short to properly evaluate self-termination. That said, again on WikiText-103 we see that perplexity is maintained compared to the baseline model: again, encouraging.

==== Questions ====

How well calibrated is the proposed model to sequence lengths? Let's ask this question for WikiText-2 because I understand that the limited context of gpt2 makes this difficult in the WikiText-103 experiments. Figure 2 suggests to me that choosing epsilon is a strong bias towards generating sequences of a particular length. Intuitively, I also find this behavior plausible just thinking about Definition 6.

If you are not well-calibrated, then following up on the previous question: if you choose epsilon to calibrate the expected sequence length to the data distribution, do you still achieve reasonable perplexities? And furthermore, if you are able to calibrate the expected sequence length, is the variance of generated sequences reasonable compared to the data distribution or does it cluster tightly around the mean? Basically I am concerned about whether Definition 6 imposes a strong bias towards generating sequences of a particular length. If that is the case, I feel that it would warrant acknowledgement and more nuanced discussion.

Edit: My concerns raised in these questions have been thoroughly and satisfactorily addressed by the authors. I have updated my score to reflect this response.
clarity,_quality,_novelty_and_reproducibility: Novelty. This work extends the work of Welleck et al. (2020), which imposed the constraint that termination probabilities converge monotonically to 1 (Definition 5). In both cases, these constraints are achieved by altering the softmax parameterization of the model family. This work is that it is a relatively small incremental advance over the work presented in Welleck et al. (2020): the methodological proposal is a variant of this previous work, and the experimental setup is the same. That said, as I do believe that Definition 6 is a valuable contribution to the community.

Reproducibility. The model is empirically evaluated on WikiText-2 RNN and LSTM architectures and WikiText-103 using the Transformer architecture (GPT2-small) with baseline comparisons to the monotonic model proposed by Welleck et al. (2020) and the standard autoregressive softmax parameterization. The models are evaluated for test-set perplexity and non-termination ratio under several decoding algorithms: greedy, beam search, top-k, and top-p (nucleus). I am confident in the reproducibility of these results.

Clarity. Overall I found the structure of the paper clear, and generally well-written. Perhaps too much space was dedicated to summarizing previous works that could have been better-spent elaborating on the proposed model. Some of the more technical writing in Section 2 and 3 is more formal than the level of rigor justifies. For example, Theorem 2 and 3 are not a rigorous: a further condition needs to be made about the underlying model p_theta beyond Definition 1: e.g. that p_\theta(<eos>|history) is bounded away from zero (e.g., the assumption made in Welleck et al. (2020)). It's not a big deal, but generally when I see claims made in the form of Theorem statements, I would expect these statements to be airtight.
summary_of_the_review: An interesting methodological contribution (Definition 6) with rigorous--but not groundbreaking--experiments.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors present a new non-monotonic self-terminating language model that significantly relaxes the constraint of monotonically increasing termination probability in the originally proposed self-terminating language model by Welleck et al. (2020).  The authors provide experiments on sequence completion tasks using various architectures.

Strengths:  There is consensus that the paper is well written, well founded, and has strong empirical results.  In addition, it was good to see lengthy, productive back and forths with reviewers.

Weaknesses:  There aren't that many weaknesses mentioned by the reviewers as there is strong consensus regarding its strengths.  Some suggestions include running experiments on non-sequence-completion tasks and one reviewer asking for illustrative examples, 
justification_for_why_not_higher_score: I think the paper is quite good with a novel methodology and strong results--however, I did not find to be area changing and hence a poster presentation should be appropriate.
justification_for_why_not_lower_score: The paper clearly deserves acceptance.
Decision: Accept: poster