Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: - Overall comments

This paper propose a principled method to generate "hard" positive and negative samples based on conditional likelihood for contrastive learning of seq2seq models, and it shows significant improvements in training conditional text generation tasks compared to naïve approach with random negative samples. Overall, the idea is interesting, and the experiments are well-conducted. However, I still have some detailed questions regarding to the method and experiment as follows:

- Methods:

(1) I am a bit confused with Eq(2). What is $\bf{M}$? Do you mean $x_i$ is the source sentence, $y_i$ is the corresponding target sentence? Is it meaningful to "match" the hidden representation between source and target sentence especially for tasks such as summarization?
Also training with Eq(2) did not involve any decoding process, nor supervising how to decode a sentence. Some form of MLE training (also noted in Eq (9)) seems to be unavoidable which in some sense still relies on teacher forcing..

(2) The proposed method to create positive/negative examples is related to virtual adversarial training (VAT) in NLP:
 *Miyato, Takeru, Andrew M. Dai, and Ian Goodfellow. "Adversarial training methods for semi-supervised text classification." arXiv preprint arXiv:1605.07725 (2016).*
It would be nice to include for discussion or comparison.

(3) For Sec 3.3 & 3.4:
(a) How do we know the perturbed hidden states $\bf{H}$ still lay in the manifold of valid sentences? It is possible the hidden states may not be corresponded to any sentences. 
(b) Using the conditional likelihood over the original target sentence to measure the negative samples may also be misleading. For example, it is also possible to get a very different sentence with the same semantic meanings with the target sentence.
(c) What is $\hat{y}$ and $\bar{y}$ in Eq (6) and (7)? Are they different target sentence? Where are they from as the proposed methods did not seem to include decoding.

- Experiments

(1) It seems that all experiments are initialized with T5. Does it mean that the proposed method only works with large scale pre-training? It would be more important to show results with training from scratch.
(2) The results on WMT16 RO-EN do not seem to be too low especially with T5 pre-training which makes the improvement difficult to tell.
(3) For many tasks, the improvements of the proposed method are actually marginal. It may improve the paper by include discussion of statistical significance.
(4) There are also methods such as Reinforcement learning which also aims to overcome the problem of teacher forcing. It should be also discussed in experiments. 
review_title: Interesting idea to generate "hard" positive and negative samples for contrastive learning

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a method for conditional text generation tasks that aims to over the "exposure bias" problem through contrastive learning where negative examples are generated by adding small perturbations to the input sequence to minimize its conditional likelihood, and positive examples are generated by adding  large perturbations while enforcing it to have a high conditional likelihood. Experimental results on machine translation, text summarization and question generation show the effectiveness of the proposed approach.

My only concern is that compare to MLE, the improvements either on Table 1 or on Table 2 are relative small. The study in the paper by Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, Laurent Charlin, Language GANs Falling Short, ICLR 2020 shows that the "exposure bias" problem for text generation by MLE appears to be less of an issue, and simple "temperature sweep" in the softmax significantly boosts the performance and gives pretty good results that beat all language GANs. So I think in the experiments, all results should be compared using the trick of "temperature sweep". Moreover, if diversity is an issue, the results should be compared in the quality-diversity space as did in Language GANs Falling Short paper. Hopefully the authors can address my concern in the rebuttal period. 
review_title: This paper presents a method for conditional text generation tasks that aims to over the "exposure bias" problem through contrastive learning where negative examples are generated by adding small perturbations to the input sequence to minimize its conditional likelihood, and positive examples are generated by adding  large perturbations while enforcing it to have a high conditional likelihood. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: ========================

Paper Summary:

This paper proposes to add contrastive learning to the sequence-to-sequence generation problem. More specifically, the authors apply a contrastive loss on the globally pooled hidden representation of the generated hidden states. The key novelty is to apply adversarial gradients to obtain both hard negative and hard positive examples. The proposed method can improve a state-of-art pretrained transformer model (T5) on 3 tasks: machine translation (WMT16 En-Ro), abstractive summarization (XSum), and question generation (SQuAD).

==========================

Overall review

Although the proposed method seems to be effective and new, the concerns outweighs the contributions in my opinion. I am leaning towards rejection for now. Please try to address my concerns during the rebuttal period.

Pros

-	The idea of using adversarial gradients to generate hard negative/positive is novel, at least for contrastive learning and sequence generation problems.
-	Improvement is demonstrated on a strong pre-trained transformer model (T5).
-	This method is experimented on 3 tasks and could possibly be extended to any seq2seq generation.

Cons

-	The sdfgclaim of solving the ‘exposure bias’ is somewhat exaggerated.
-	The proposed method is somewhat straight-forward and lacking theoretical insights/guarantees.
-	The method is only applied on a small version of T5, which is limited. How about other pretrained (potentially larger) models? How about non-pretrained models such as randomly initialized Transformers/LSTMs?

==========================

Detailed Review

The authors claimed to mitigate the exposure bias problem for sequence generation. However, the original `exposure bias’ problem refers to not seeing incorrectly generated text tokens as training input, which leads to train-test mismatch. In this work, the model does not see any self-generated negative tokens as input, but only pooled adversarial hidden states. It does not mitigate train-test mismatch at all. Therefore, the current presentation may be misleading. It might benefit the paper to also compare and contrast to adversarial training for NLU such as SMART/FreeLB. 

Moreover, this work does not provide new theoretical insights. The hard negatives/positives do not have theoretical guarantee. It is not clear to me why Eqn (6) & (7) will be a distant positive. If g = f, then H = H_bar. Moreover, in MT and Sum., adversarial step size eta and epsilon are set the same. This is inconsistent with the intuition of near negative and distant positive claimed in the paper.


==========================

Other Questions / Suggestions

-	In Eqn. (2), why not use 2-layer MLP as in SimCLR?
-	In experiments, maybe add a CLAPS w/o negative so that readers would know which is more important.
-	Why not train as how you generate Table 3 example? This will then better solve the train-test mismatch (exposure bias), although maybe at a cost of slow training.
-	Some human evaluation on a larger set of generated examples would help. For example, how many hard negatives are actually being recognized as negative by human.

review_title: Official Review by R4

final_score: rating: 6: Marginally above acceptance threshold
review_content: # Summary
Proposes contrastive learning method for conditional text-generation. Here we maximize similarity (of representations) between source and target sequences (positive) while minimizing similarity with false targets (negative). Additional positives and negatives are created in the sequence representation space by adding perturbations to decoder (output) hidden states to minimize/maximize conditional likelihood p(y|x). It is shown this works a lot better than the naive contrastive approach of sampling random non-target sequences.

The full model is based on T5-small (Raffel et al) and combines contrastive objective with regular MLE objective by simple addition. Modest improvements over T5-small are observed on Translation, Summarization, and Question-generation seq2seq tasks.

# Pros
1. Diversity of seq2seq tasks, with consistent improvements over baseline T5-MLE (small).
2. Possibly improves the exposure bias issue of regular MLE seq2seq training.
3. Complementary to seq2seq MLE training and can be used to improve it in general, not just text generation.

# Cons
1. The improvements are consistent but appear to be modest. It is unclear whether the improvements would persist on the larger T5 model sizes. Would it be possible to study this (e.g. medium size)? 
2. Please add SOTA results in the tables for the various tasks for reference.
3. Please discuss effect on training/inference speed.
4. Since this is generation, more non-cherry-picked example decodes would be informative to have in the appendix. 
5. Even better would be some basic human evaluation of generated outputs to verify whether meaningful quality improvements are made.
6. Scheduled Sampling (Bengio et al) should be discussed and perhaps compared as it is a well-known method for addressing exposure bias.
7. Should discuss relationship to Virtual Adversarial Training (Miyato et al)

# Clarifications
1. Are all the models initialized with T5-MLE or are they trained from scratch on C4 for the same number of steps as T5-MLE?
review_title: Interesting technique, but significance of the improvements over baselines unclear

Meta-review: This paper proposes a new method for conditional text generation that uses contrastive learning to mitigate the exposure bias problem in order to improve the performance. Specifically, negative examples are generated by adding small perturbations to the input sequence to minimize its conditional likelihood, while positive examples are generated by adding large perturbations while enforcing it to have a high conditional likelihood. 

This paper receives 2 reject and 2 accept recommendations, which is a borderline case. The reviewers have raised many useful questions during the review process, while the authors has also done a good job during the rebuttal to address the concerns. After checking the paper and all the discussions, the AC feels that all the major concerns have been solved, such as more clarification in the paper, more results on non-pretrained models, and small-scale human evaluation.  

On one hand, reviewers found that the proposed method is interesting and novel to a certain extent, the paper is also well written. On the other hand, even after adding all the additional results, the reviewers still feel it is not super-clear that results would extend to better models, as most of the experiments are conducted on T5-small, and the final reported numbers in the paper are far from SOTA. 

As shown in Table 1 & 2, the AC agrees that the final results are far from SOTA, and the authors should probably also study the incorporation of CLAPS into stronger backbones. On the other hand, the AC also thinks that T5 is already a relatively strong baseline to start with (though it is T5-small), and it may not be necessary to chase SOTA. Under a fair comparison, the AC thinks that the authors have done a good job at demonstrating its improvements over T5-MLE baselines. 

As a summary, the AC thinks that the authors have done a good job during the rebuttal. On balance, the AC is happy to recommend acceptance of the paper. The authors should add more careful discussions to reflect the reviewers' comments when preparing the camera ready. 
Decision: Accept (Poster)