Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposes contrastive token learning in order to alleviate the degeneration problem in text generation tasks. In addition to the conventional token-wise cross entropy loss, the loss penalizes negative tokens in the similar manner as done in cross entropy loss, but sum  over negative tokens and the true token in the denominator. The negative tokens are chosen from the past history in a window but allowing duplicates by changing from a set to a multiset. Experiments are carried out on language modeling and dialogue generation tasks and the proposed method achieves lower repetition rates with more distinctive outputs preserving comparable perplexities when compared with SOTA baselines.
strengths_and_weaknesses: Strengths
* The proposed method based on the unlikelihood training by penalizing negative tokens, but employs contrastive loss for softly penalizing negative tokens. The idea is simple yet sound.
* The use of tokens in a window and multiset sounds the major contribution in this work.
* Experiments are well designed and the effectiveness is demonstrated by the systematic comparison. Analysis is also convincing both qualitatively and quantitatively.

Weaknesses
* One of the main contributions is the carefully designed negative token set and the set could be easily employed in other approaches, e.g., UT and SimCTG. I'd like to see the comparisons with those SOTA approaches with the proposed negative token set.
questions: None
limitations: * Currently, the proposed approach is experimented on language modeling and dialogue generation, but it is not clear whether the approach is also effective to conditional language model setting.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper introduces a simple contrastive objective for text generation models to alleviate text degeneration. The proposed objective contrasts the ground-truth tokens with the negative tokens (previous $M$ context tokens), thus suppressing the model from generating those tokens during inference. Empirical results show the proposed approach can reduce the repetition significantly.
strengths_and_weaknesses: **Strengths**
- The proposed approach is simple and intuitive.
- The proposed approach effectively reduces the repetition in the generated text.
- The analysis is extensive and interesting.

**Weaknesses**
- Some confusion about the description.
    + The authors keep claiming that Unlikelihood training (UL) does not consider the relationship between the label tokens and unlikely tokens but they never give details about how the proposed approach (CT) considers this.
    + I don't understand why UL does not contrast the negative tokens to other tokens. Essentially we are looking at the probability distribution. So when the probabilities for negative tokens are reduced because of UL, the probabilities for other tokens are relatively boosted compared to negative tokens. Is this not considered as contrastive?
    + The authors claim that the fact that UL considers all the previous context tokens as negative tokens introduces too much noise and results in sub-optimal repetition reduction. However, in the proposed approach, the same strategy is used to define negative tokens with the exception of an additional window of size $M$, which I believe can be applied to UL directly. And how is this not sub-optimal compared to UL?
    + The main intuition in this work is to dynamically promote/suppress tokens based on three categories (positive, negative, and irrelevant). Similar stuff has been observed in existing work [1] and it is very relevant to this work. The author should at least discuss this regard.
- Experiments
    + Hyper-parameters. How is the hyper-parameter used in UL? Apparently, if UL is applied more aggressively, the repetition could be reduced more obviously although the ppl could be harmed. However, there is no any description of the hyper-parameter choice of UL. Similarly, I cannot really agree with the claim that the CT outperforms the top-$k$/$p$ in terms of reducing repetition. Once a larger value of $k$ or $p$ is used, the repetition is almost guaranteed to be reduced. But here again, no different values of $k$ and $p$ are considered. 
    + I disagree with the claim that CT even outperforms humans according to rep-* and unique token counts. The repetition is not always the less the better, and the number of unique tokens is not always the more the better, as the good text requires a certain level of repetition to retain the coherence.
    + Human evaluation. Human evaluation is somewhat biased and subjective. Even major vote is used, it will be more convincing to report some correlation between humans.
- Line 265: analyses --> analysis


[1] Lin et al., "Straight to the Gradient: Learning to Use Novel Tokens for Neural Text Generation"
questions: - In figure 3, I wonder what would UL-T look like? And do you think the uncertainty in the later steps can be solved by introducing $M$ to UL?
limitations: Yes
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper propose a learning-based method for alleviating the problem of text degeneration.  The proposed method is simple. I combines unlikelihood training and contrastive learning.  The training objective is to promote label tokens while suppressing preceding tokens. Experimental results show that the proposed method can reduce repetitions in generated text.
strengths_and_weaknesses: Strengths: 
1. The method is simple.
2. The experiments are elaborate.
3. The proposed method seems effective in reducing repetitions. 

Weaknesses: 
1. The idea to penalize any repetitions in a short window is somewhat problematic. Some repetitions are reasonable in human-generated text. I think the critical question is how to properly construct the negative token set.
2. The analysis of experimental results is problematic. For rep-1/2/3/4, dist-1, uniq-1, I think the best results should be the ones that are closest to human-generated text. Like the above, humans generate repetitions naturally. NOT all repetitions are evil. 
questions: My major concern: for rep-1/2/3/4, dist-1, uniq-1, I think the best results should be the ones that are closest to human-generated text. if so, you have to redo the analysis of experiment results.
limitations: n/a
review_title: null

Meta-review: metareview: The paper proposes a simple contrastive objective punishing the degeneration in the text generation. There is an agreement in the reviewers that the method itself is somewhat incremental and lacks very clear justification. It could be seen as a straight-forward extension to the SimCTG algorithm and there have been a handful of publications in the field. I therefore do not recommend acceptance of this paper to NeurIPS.

confidence: Certain
Decision: Reject