Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

The augmentation of NLP samples is an important task with no clear "applicable to all" mechanism. This is in sharp contrast to computer vision where techniques like rotation, modification of hue, saturation as well as umpteen other techniques exist. This work tries to address the issue by proposing a technique that carefully amalgamates multiple previously known approaches to generate diverse label preserving examples. The experimental results on RoBERTa highlight the applicability and importance of this data augmentation approach on the downstream task of text classification (GLUE).

Strengths:

1. Empirical results. Performance better than previous approaches (although minor).
2. Paper Clarity
3. Each formulation is backed by a strong intuitive understanding.
4. Contrastive training (negative sampling) is one of the crucial contributions of this work. It seems to be making every previously known augmentation approach better. 
Please feel free to highlight other major contributions.

Weaknesses (Minor):

1. Ad-hoc regularization parameter selection is necessary for getting performance gains. This makes it difficult to conclusively prove that this is an "applicable to all" data augmentation scheme.
2. It would have been better to see the performance gains on more difficult text-classification tasks (non-GLUE), or underperforming models (non-BERT based). Since the gains are not much. It becomes difficult to fathom if the gains are actually due to good objective function or a case of chance for choosing better examples.


Comments/Questions:

1. What is the augmentation size being used in the setup? I suspect the size plays an important role in such setups and this hasn't been discussed much in the paper. Also, please show the performance trends based on different augmentation sizes.

2. How do you measure the diversity (as mentioned in the paper title) in the generated samples? 

3. Rather than using the ad-hoc approach for selecting which augmentation "stacking" scheme is helpful, it would have been better to compare/use an approach highlighted in "Learning to Compose Domain-Specific Transformations for Data Augmentation" [NeuRIPS 2017].


Correction:

1. Related Work: Contrastive learning - Under an unsupervised setting, ontrastive -> contrastive

Overall:

This work highlights the importance of incorporating contrastive training for data augmentation.

Please let me know if I have misunderstood something(s)
review_title: Official Blind Review #1

final_score: rating: 7: Good paper, accept
review_content: Paper proposes a contrastive learning-based approach to combine different data augmentation techniques for NLP tasks. While the widely used consistency loss focuses on a single example, the proposed contrastive objective allows capturing the relationships among all data samples which helps in producing diverse and informative examples. For experiments, the paper explores 5 data augmentation approaches with Roberta-large as the classification model. Empirical results on the standard GLUE benchmark leads to an impressive 2.2% average improvement. Authors also found that back-translation and adversarial training combination leads to better performance than other DA combinations.  


Strengths:
1. The proposed framework can be applied with any text data augmentation methods. It's a solid work that will help the NLP community in developing better DA techniques. For example, [Kumar et al. 2020] shows that any pre-trained model can be used for data augmentation. I believe seq2seq model like T5, BART based augmentation combined with CoDA, will further push the state of the art for text DA. 
2. Paper provides clear motivations and describes their methods, experiments in detail. Authors study DA in both low-resource and rich-resource setting. Ablation studies have been conducted to investigate gains from different components.  
3. Authors plan to release their code which is good for reproducibility.  

Weakness: 
My understanding is that all numbers reported in the paper are from a single experiment. As a reader, I would like to see some variance with the results. Apart from this, I don't see any major issues with the paper.   

Questions:
1. Since one of your goals is to improve the diversity of the augmented data, have you tried replacing more words in the c-bert model? By nature, c-bert is bound to replace max 15% of the tokens while maintaining the sentence length. Methods such as back-translation or seq2seq models do not have such restrictions.  Also, have you considered using a pre-trained seq2seq model for DA as in [Kumar et al. 2020]
2. Fig 5, back-translation, and adversarial training have similar performance. This result is intriguing.  Do you have some further insights into it? 

Typos:
- Sec2.2. "the first term correspond" -> corresponds 
- Sec 4, Contrastive Learning para, "ontrastive learning" -> "Contrastive learning"

References (additional DA citations):
1. Kumar, V., Choudhary, A., & Cho, E. (2020). Data Augmentation using Pre-trained Transformer Models. ArXiv, abs/2003.02245.
review_title: A contrastive learning based framework for NLP Data augmenation 

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes a novel data augmentation framework, which explores different combinations of isolated label-preserving transformations to improve the diversity of augmented samples.  The authors find that stacking distinct label-preserving transformations produces particularly informative samples.  The paper also introduces a contrastive learning objective to capture the global relationship among the data points in representation space.  

In my opinion, the exploration of different combinations of isolated label-preserving transformations is the major contribution of this paper, which may inspire future works for data augmentation. However, the contrastive regularization object is a bit incremental, and I cannot see a big difference compared with Moco or SupCon.   

Strength:

+ The idea of stacking distinct label-preserving transformations is intuitive.
+ The integration of the consistency training objective and the contrastive regularization objective is interesting.

Weakness:

- Lack of novelty, the contrastive regularization object is a bit incremental, and this object is very similar to MoCo or SupCon.
- The model has first to obtain the augmented samples, which is computation expensive for large-scale datasets and may hinder the practical application of the model. Moreover, the overall improvements are relatively small compared with R3F, and there is a lack of variance analysis.

Questions:

What is the computational complexity of CoDA?

Why using MMD distance in section 3.1?

Is stacking distinct label-preserving transformations the default setting for CoDA in your GLUE experiments? What if other strategies (mix, random) work better in datasets like QNLI, RTE. MRPC, and so on. Why not report results on those datasets?

What is the major difference between your contrastive regularization and MoCo or SupCon?

As the improvements are relatively small, could you please provide the test of statistical significance？

What if you stack cut first and then back? Does the order affect the performance?
review_title: Comprehensive empirical evaluations and interesting ideas.

Meta-review: This paper concerns data augmentation techniques for NLP. In particular, the authors introduce a general augmentation framework they call CoDA and demonstrate its utility on a few benchmark NLP tasks, reporting promising empirical results. The authors addressed some key concerns (e.g., regarding hyperparameters, reporting of variances) during the discussion period. The consensus, then, is that this work provides a useful and relatively general method for augmentation in NLP and the ICLR audience is likely to find this useful.
Decision: Accept (Poster)