Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper aims to solve the degeneration problem using a contrastive training objective and a contractive search. The proposed contrastive training objective encourages the model to learn isotropic representation for the tokens. In addition, the proposed contrastive search algorithm scores hypotheses by discriminating the candidate tokens and previous context tokens. Experiments show improved ppl and acc on language model trained with the contrastive objective and enhanced generation qualities with contrastive search.

strengths_and_weaknesses: Strength: 1) The proposed methods are relatively simple and general, which can potentially be applied to any text generation model. 2) The results strongly support the methods. 3) Well written.

Weakness: 1) The proposed methods - contrastive training objective and contrastive search - are two independent methods that have little inner connection on both the intuition and the algorithm. 2) The justification for isotropic representation and contractive search could be more solid.

questions: LN158: Is there any study demonstrating the effectiveness of the metric on coherence?

Table 1: Why does contrastive search work so much better for SimCTG than for MLE and Unlike?


limitations: NA
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: NLG models rely on the maximum likelihood objective during training and a related decoding strategy. In the most vanilla formulation, this combination of training objective + inference strategy results in degenerate text. One reason is the anisotropic distribution of the token embeddings learned by the underlying models. Inspired by contrastive representation learning, this paper proposes SimCTG a new approach that incorporates an additional term in the training formulation and a modified inference time strategy during decoding. The combination induces diversity (reducing degeneration) in the generated text while maintaining relevance to the input.    
strengths_and_weaknesses: Strengths: 
1. The paper is well-written and easy to understand.
2. Applicable to wide variety of NLG tasks.
3. Good performance on Wikitext-103, LCCC, and Dialog Daily.
4. Comprehensive strategy analysis and generation analysis (section 6).
5. Fair comparison against competitive baselines and readable code in supplementary.

Weaknesses: 
1. [Very Minor] While the contributions of this work are certainly novel, the formulation is a simple variation of previously known schemes, one of which had been explored for encoder-only models. The training strategy is a variation of TaCL [19]  + Self-similarity [6]. The decoding objective is a variation of the unlikelihood objective.
2. Human Evaluation for informativeness is vague.
3. Evaluation of non-pre-trained models is missing. This would help in understanding if the performance gains are only restricted to pre-trained models or not.

questions: 1. The paper mentions in Line 112 that the formulation is architecture agnostic. Would the same strategy work in the case of encoder-decoder models, where the decoder might learn the isotropic embeddings while the encoder might remain more or less unaffected? Or should the objective also involve constraining the encoder representations as well?
2. Can you please elaborate on the Informativeness evaluation in human studies? The description given in the appendix is also difficult to parse.
3. How does the formulation perform on a vanilla transformer network / standard LSTM model?
limitations: The authors have addressed the limitations in the appendix however they should discuss the potential downsides of using the GPT2 model - offensive content generation. Since the decoding process involves the generation of words that are different from the word appearing before them (contrastive search), it might help to show some cases where the model diverges too much from the content and starts rewarding offensive content (or at least mention it).

Missing citation:
"Towards Transparent and Explainable Attention Models (Mohankumar et al., ACL 2020)"

One evaluation metric for checking token dissimilarity is proposed in "Towards Understanding the Geometry of Knowledge Graph Embeddings (Chandrahas et al, ACL 2018). Kindly evaluate that since self-similarity is very similar to the objective that is being tried to optimize in the SimCTG method.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this work, the authors investigate the model degeneration problem in neural text generation, they show that the main reason for model degeneration is the anisotropic distribution of token representation. Hence, they use a contrastive training objective to enlarge the distance between tokens and propose a new decoding method of contrastive search to keep the coherence in the generated text. This paper is well-written and it shows its advantages compare to other methods.
strengths_and_weaknesses: Strengths:  
1. This paper is well-written, methods are clearly explained.  
2. The given storyline and corresponding experiments are logically consistent.
---------
Weakness:  
1. The contrastive learning method has been widely used to solve the representation degeneration problem, the method is not novel.  
2. The proof is not convincing enough, it is not enough to use the token cosine similarity to reveal the model degeneration problem is caused by the token representation degeneration.  
3. It needs more experiments to illustrate the effectiveness of the methods, it needs to add some representative NLG experiments, e.g. machine translation. 
questions: 1. The anisotropic distribution of token embeddings is considered from a global perspective, why Figure 1 can reveal the phenomenon? I think the phenomenon can not be explained by anisotropic.  
2. In Table 1, for the metric of language modeling quality, what decoding strategies are used for different models?  
limitations: The method proposed by the authors prefers to generate tokens that have not appeared in the previous context, it will influence the performance of high-frequency tokens. At the same time, this method is more suitable for tasks with high degrees of freedom.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Current language models suffer from the issue of degeneration, which makes their outputs dull and repetitive. This paper argues that this degeneration in text generation is due to the anisotropic distribution of LMs. Prior work [1] has shown that LM representations are anisotropic --- the cosine similarity between the representation vectors for *different* tokens in a sentence is very high, up to 0.95. This paper designs training + inference time algorithms to fix this issue, and report improvements in text generation quality.

This paper proposes the SimCTG algorithm, which helps make representations more isotropic during training. The key idea is to use an extra loss function which pushes away representations of different tokens. During inference, the paper proposes the use of contrastive search, a decoding objective which encourages the generation of tokens whose representations are dissimilar to one another. Contrastive search can be applied without the contrastive training on any existing LM.

The paper evaluates their approach on open-ended dialogue and document generation, covering both English and Chinese datasets. Extensive automatic and human confirm that the proposed approach beats baselines like nucleus sampling & unlikelihood training.

[1] - https://aclanthology.org/D19-1006.pdf
strengths_and_weaknesses: **Strengths**

1. The paper works towards fixing an important issue in current text generation systems --- their tendency to produce degenerate text. The paper draws an nice connection to prior work [1] which has found LM representations are anisotropic.

2. The paper presents simple training and inference time algorithms to make language model representations more isotropic. The inference time algorithm (contrastive search) can be flexibly applied to any existing LM without further training, and seems to improve over nucleus sampling for dialogue generation even without contrastive training.

3. The paper evaluates their algorithm on three datasets spanning two tasks and two languages. Extensive automatic evaluation (using effective metrics like MAUVE) and human evaluation is conducted to confirm the efficacy method.

4. The main experiments are complemented with good analysis experiments discussing qualitative aspects of improvements with model generations, timing analysis, effect of varying hyperparameters, isotropy of SimCTG representations.

**Weaknesses**

I have one major concern with the paper. All experiments have been conducted on a very small language model, GPT2-small which has 117M parameters. However, GPT2-small is a weak language model compared to much larger open-source alternatives like GPT2-large / XL [6], T5 variants fine-tuned for causal language modeling [2], OPT [5] etc. For dialogue generation you could simply fine-tune larger T5 variants or BART.

Testing out novel text generation ideas at a larger scale is important --- as the LMs get bigger, generative quality of MLE baselines significantly improves. It's unclear how anisotropic larger LMs are, and if yes, whether this anisotropy affects generation quality. A related informal tweet - [7].

**Minor**

It will be good to discuss the relations to [3, 4] in the related work since they use contrastive learning for text generation as well. These two papers have come out in the last month (contemporary to this paper), so no direct comparison is necessary.

In the introduction, explicitly mentioned that the anisotropic representation issue was first noticed by [1].

In Table 2 / 3, keep a column which aggregates scores and mentions the fraction of data points where all three metrics simultaneously score a 4 or 5. This will give a good idea about the overall performance of methods.

line 173: confusing --> confused

**Overall** --- This paper has good originality, clarity, quality, but moderate significance. My main concern is that experiments were only conducted on GPT2-small, a very small language model compared to the state-of-the-art. Nevertheless, I'm leaning accept due to interesting idea as well as good automatic & human evaluation conducted in the paper.

---

**After Rebuttal** - Thank you for your comprehensive reply and addressing the weakness I pointed out. I've raised my score to 7.

[1] - https://aclanthology.org/D19-1006.pdf  
[2] - https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#lm-adapted-t511lm100k  
[3] - https://arxiv.org/abs/2205.09726  
[4] - https://arxiv.org/abs/2205.14690  
[5] - https://arxiv.org/pdf/2205.01068.pdf  
[6] - https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf  
[7] - https://mobile.twitter.com/_jasonwei/status/1526589104758042624
questions: None
limitations: Good limitations section in Appendix
review_title: null

Meta-review: metareview: All four reviewers sided to accept the paper, as the proposed contrastive search approach to mitigating text degeneration problem is simple and effective and has applications to a variety of NLG tasks. Its evaluation is quite comprehensive and includes competitive baselines, human evaluation, and evaluation of both LM/generation quality on Wikitext-103 and effect on a downstream task (dialog). Two of the reviewers were more hesitant (borderline accept), but one of them was quite satisfied with the author response and the other reviewer didn't raise any major issue. The one remaining concern is that experiments with GPT-2 were base on the "small" model, but the rebuttal shows that the findings of the paper mostly hold with bigger language models (medium and large) but become relatively small with XL. We suggest including these additional experiments in the next version of the paper, along with further discussions of these smaller differences.
confidence: Certain
Decision: Accept