Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper investigates the repetition issue in natural text generation. The described loop issue was caused by the maximization-based decoding algorithm we use. Machine generated text has higher chance to be repetitive compared to human corpus. The paper starts with analysis on individual cases and then some more scalable experiments with a few standard metrics. It introduces a “self-reinforcement effect” where the model will be more likely to repeat if the similar context has shown before. The more times of previous occurrences, the more likely it will repeat. To tackle the problem, the paper proposes a simple and intuitive training objective, DITTO, to reduce the repetition. The paper conducts experiment on open ended text generation and text summarization on CNNDM. 

strengths_and_weaknesses: The paper tackles a core challenge in NLG. The “loop” of the paper is complete and convincing. The proposed technique is inspired by experiments and analysis. The idea of the paper is generally clear and easy to follow. 
The proposed technique is relatively simple to implement and use. 

It compares with some prior work and shows its effectiveness. Human evaluation is also provided to validate the proposed technique. 

The empirical results are good. The proposed method is competitive compared to prior work including unlikelihood training and StraightGradient.

Weakness

I have a major question regarding the function of DITTO (Equation 1).
Equation 1: 
-math.log( 1 - abs(cur - alpha * prev))
Let’s set alpha = 1 for now. 
If the model is well-trained to avoid repetition, the current probability for a target token v P(x_{n,l} = v) is lower than the probability if shows before P(x_{n-1,l} = v), then this is an ideal scenario that the model did great. However, the equation actually penalizes this situation. 
Here is my experiment:
(Good case) If cur=0.1, prev=0.5, alpha=1, the loss is 0.51.
 cur=0.1;prev=0.5;alpha=1
-math.log( 1 - abs(cur - alpha * prev ))
0.5108256237659907

(Bad case) If cur=0.9, prev=0.5, alpha=1, the loss is still 0.51
cur=0.9;prev=0.5;alpha=1
-math.log( 1 - abs(cur - alpha * prev ))
0.5108256237659907

This doesn’t make sense to me that the loss is the same for both cases. In the “good” case, the model decrease the chance of repetition so it should be rewarded rather than penalized. 

If the alpha is set to be <1, it’s better than the case described when alpha=1, but still I think it’s unfair to penalize a model to not be repetitive. 
cur=0.1;prev=0.5;alpha=0.5
-math.log( 1 - abs(cur - alpha * prev ))
0.16251892949777494
cur=0.9;prev=0.5;alpha=0.5
-math.log( 1 - abs(cur - alpha * prev ))
1.0498221244986778


If I understand the equation correctly, should it be max(_, 0) instead of abs(_)?

cur=0.1;prev=0.5;alpha=1
-math.log( 1 - max(cur - alpha * prev,0 ))
-0.0

cur=0.9;prev=0.5;alpha=1
-math.log( 1 - max(cur - alpha * prev,0 ))
0.5108256237659907
To conclude, the designed equation penalizes the model even the current pred has less “overlap” with previous pred. I am not sure if there is some misunderstanding of the equation. I apologize if there is, and I am open to discussion. Authors and reviewers are welcomed to comment on this. 


The coverage of the experiment could be further improved. There are only two datasets used in this paper, Wiki103 for LM and CNNDM for summarization. It would be better if the paper can provide more evidence on more datasets. 

questions: The equation 1 is designed to penalize “one-hop” similarity. Let’s say you have a sequence: The orange is blue. The orange is red. The orange is blue. The orange is red. [repetition] 
In this case, every two sentences are different, but 2-sentence repetition is not a problem here. Is it possible to boost the equation so you can cover all the previous sentences?
limitations: The DITTO equation tackles limited scenario of repetition. Some data augmentation or contrastive learning techniques can be applied to make the training less “synthetic”.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work proposes a novel way of mitigating sentence-level repetition in autoregressive sequence generation: they form an artificial set of negative samples by repeating the same sentence from training corpus up to the max length of prediction. After that they use a loss function to penalize those instances from being produced during fine-tuning. They compared their method with multiple variants from previous work and concluded the superior quality of generated sequences. They have done both automatic and human evaluation for open-ended generation and automatic evaluation for directed generation.
strengths_and_weaknesses: # Originality

## Strengths

* Authors proposed a novel way of viewing inter-sentence repetition which is a more global way compared to n-gram repetition (which can also be inter-sentence for large enough n-gram length)

## Weaknesses

* Many claims around idea justification are written in a vague way, which I will touch later in the questions section.

# Quality

## Strengths

* Authors uploaded a well-done code base with experiments and instruction of how to run it. The code is based on fairseq which makes it not hard to use in the future work by other researchers.

* Authors composed comparison with other methods which avoids this type of degeneracy in addition to the comparison with the baseline method.

## Weaknesses

* Related work section could be improved. For instance, there is a recent work (not related with myself in any way) about the ability of the LM to break the reinforcing loop: https://arxiv.org/pdf/1905.10617.pdf . In other words, there exists previous work directly related and opposite to observations and claims for this work. I want to stress that this is totally fine, but authors should consider discussing such results and providing some explanation of how the setup is different etc.

* Experiment comparison is not very transparent. For instance, sequence-level UL models are trained to penalize continuations generated by up to 150 tokens. It was observed earlier that after such 150 tokens it may tend to repeat again (looks like this is exactly what happens in Fig.4). In other words, the length of generation influences the ability of the model to not repeat up to such extent. In DITTO the model is being penalized by the max length of generation. You may have done the experiment where DITTO would be trained on negative sentences of similar length to match the UL training protocol. If you are not going to do that, then at least clarifying this is necessary, because it addresses your claim that UL does not break the reinforcing loop. Because it may do so if you train UL model in similar fashion (with longer generated seqs).

# Clarity

## Strengths 

* The experimental protocol is clear.

## Weaknesses

* The section I was very curious about 2.2 is extremely vague and does not provide any answers (I will touch this later in the questions)

# Significance

## Strengths

* I believe that this general method provides a significant contribution for future work beyond this specific use case: using an external set of negative samples which are easy to form and optimize. 

## Weaknesses

* On the other hand from the the strong point #1, the loss function is not well justified (I touch this again in the questions) as in why such construction (with modulo difference) is better than eg the margin based version.

* Even though Contrastive framework for text generation was discussed (it was mistakenly categorized as decoding based method while in fact it is training based too), it was not used for experimental comparison. 
questions: Here I will write both my comments and questions:

Line 30: [30] is also training based method.

Line 40-41: How does your analysis differ from one in Holtzman et al. work? I can see the introduction of several metrics to measure the rate of self-reinforcing loop, but other than that it doesn't look different. If you want to stress the point about analyzing sentence-level repeat, then it is a good place to do so.

Line 52-56: When you say "The cause ... may be that ...", then the overall claim does not sound strong at all. If you do not have sufficient evidence (e.g. by showing how this effect does not hold on some other test sets without repeats or so, then I am not sure how this is different from all other existing observations about n-gram level repeat.

Line 61, 63, 64: Please avoid using "can" or "may". if it is not always the case, then quantify how often this happens. After rewriting these sentences the overall reading will be improved.

Line 63-65: Where is the justification for this statement? It sounds incomplete without any reasoning afterwards.

Line 89: what does it mean to "statistically investigate"? Other wording will improve this sentence.

Line 123: "To our surprise": why is it surprising? All previous work was showing that this is indeed a large problem. I agree that random token sentence repeat is more surprising, but authors don't say that.

Lines 125-132: I do not see the answer to the question "Why sentence repetitions occur?". I see observations, but no verified hypothesis of the reason behind it. For instance, Contrastive Generation framework paper [30] specifically introduce the hypothesis of very similar embeddings of repeating tokens which they further verified. I do not see any similar process here. Perhaps authors had some valid answer, but it is not presented here.

Lines 133-137: Where is the answer for the question? The observation of self-reinforcing loop on artificial corpus is not an answer. Moreover, there is previous work showing that LM has the ability to break such a loop: https://arxiv.org/pdf/1905.10617.pdf I would be happy to see if authors can elaborate more about this.

Lines 144-153: Due to the unclear claims in the section 2.2 the statements in "Analyses" paragraph do not sound strong.

Line 167: why did you choose the loss function to look like this? It is interesting to see if there is some theoretical benefit of using such construction of log(1-|p1-p2|). For instance, the gradient of the loss may show some benefits. Alternatively you could have optimized the difference between log-probabilities: max(margin, logp(x_n) - logp(x_{n-1})) . Some justification around the loss function design would help readers to understand the motivation better.

Lines 190-193: At least for UL-seq it is not true that it penalize any repetitions. In fact it is specified by the length of ngram. In other words, if n=15, then it will only penalize those sentence-a-like ngrams. 

Lines 256-258: DITTO is specifically designed to mitigate repeats up to max generation lengths, while other methods you compared to did not. In other words, other methods could have a chance to show better performance if they match the design of negative samples similarly to DITTO. And the same holds in opposite direction I believe: if DITTO would be trained for e.g. 150 tokens only, then it may fall in the same repetition loop again after generation that number of tokens. I think this is very important point to mention in this section.

Lines 305-207: As I said earlier, this can be a result of not 100% fair comparison between training and evaluation protocols. 
limitations: Relevant related work which is missing (sorry if it is there and I missed it):

https://arxiv.org/pdf/1905.10617.pdf

https://arxiv.org/abs/2007.14966

https://www.semanticscholar.org/paper/High-probability-or-low-information-The-paradox-in-Meister-Wiher/49a7515b1c0250a12c0e6b6fa0a8b6d46fc42adf

https://arxiv.org/abs/2002.02492
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents an empirical study showing the problem of repetitions in text generation (especially sentence repetitions), and proposed a training method called DITTO to address this problem where they add a training objective to penalize the probabilities of sentence level repetitions for the synthetically created examples (repeated sentences). Authors evaluated their method on different tasks and datasets, open domain text generation and summarization, and showed their method has much less repetition while still achieves competitive performance on other metrics. 

strengths_and_weaknesses: Strength: 
The paper presents a good empirical analysis for the repetition problem in text generation.  Authors defined a few metrics, and evaluated on different datasets demonstrating the problem.  Though the community is aware of such problems, this is the first time I see such an analysis systematically showing the empirical results. 

Based on the repetition pattens authors identified (higher probabilities when repetitive sentences are included in the context), authors proposed a training method to penalize such repetition. The proposed training objective is well motivated. 


Weakness: 

For the repetition problem, though the empirical analysis is good, I feel the authors are overclaiming slightly. They do show the sentence repetition happens, and models has a self-reinforcement effect. However, they say that’s the reason that sentence repetition occur (I.e., the model has seen the previous sentence in the context). I feel this is not exactly the explanation yet. The underlying cause of this phenomenon still needs additional analysis, probably more inside the neural networks. 


questions: In sec 3:
Authors mentioned "We find that, appending another sentence sampled from the training corpus randomly as the prefix to the pseudo repetitive data x can achieve better and more stable performance." 
Can you clarify this?  is this providing some context to the repetitive sentences? but if so, why do you need to randomly sample another sentence, rather than just the previous sentence? 

Fig 4:  I’m a bit confused by the results for UL. 
Earlier tables showed it has much lower repetition rate than other methods, which seems to be inconsistent with the results presented here. 



limitations: Authors didn't discuss this.
review_title: null

Meta-review: metareview: This paper investigates the source of repetition in text generation from a language model and presents a training method to mitigate this problem. Their experiments show the proposed method not only reduces repetition, but also improves generation quality. I think this is a good paper. All reviewers agreed with me so I recommend acceptance to NeurIPS.
confidence: Certain
Decision: Accept