Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: In this paper, the authors present a new adversarial training algorithm and apply it to the fintuning stage large scale language models BERT and RoBERTa. They find that with FreeLB applied to finetuning, both BERT and RoBERTa see small boosts in performance on GLUE, ARC, and CommonsenseQA. The gains they see on GLUE are quite small (0.3 on the GLUE test score for RoBERTa) but the gains are more substantial on ARC and CommonsenseQA. The paper also presents some ablation studies on the use of the same dropout mask across each ascent step of FreeLB, empirically seeing gains by using the same mask. They also present some analysis on robustness in the embedding space, showing that FreeLB leads to greater robustness than other adversarial training methods

This paper is clearly presented and the algorithm shows gains over other methods. I would recommend that the authors try testing their method on SuperGLUE because it's possible they're hitting ceiling issues with GLUE, suppressing any gains the algorithm may yield.

Questions,
-  In tables 4 and 5, why are only results on RTE, CoLA, and MRPC presented? If this is because there was not noticeable difference on the other GLUE datasets, please mention it in the text.
- I realize that this method is meant to increase robustness in the embedding space, but did you do any error analysis on the models? Did they make different types of errors than models fine-tuned the vanilla way?

Couple typos,
- Section 2.2, line 1: many -> much
- Section 4.2, GLUE paragraph: 88 -> 88.8 
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: 
-	This paper modifies and extends the recent “free” training strategies in adversarial training for representation learning for natural language.  The proposed “Free” Large-Batch Adversarial Training is well motived, in comparison with plain PGD-based adversarial training and the existing methods like FreeAT and YOPO, which virtually enlarges the batch size and minimize maximum risk at every ascent step. The contributions are solid. 

-	The proposed methods are empirically shown to be effective, in addition to being aligned with some recent theoretic analysis.  The models achieve SOTA on GLUE (by time the paper was submitted; it is not the best model now but that does not affect the contributions), ARC, and the commonsenseQA dataset.

-	The paper conducted good analysis demonstrating the effectiveness of the proposed components, including detailed ablation analysis. 

-	The paper is well written. It is well structured and easy to follow.  A minor suggestion (just a personal view) is that the author(s) may consider using “natural  language” instead of just “language” in the title and may consider using more specific words like “representation” instead of “understanding”. But this is minor. 

I recommend an accept. 

review_title: Official Blind Review #1

Meta-review: The paper proposes a new algorithm for adversarial training of language models.  This is an important research area and the paper is well presented, has great empirical results and a novel idea. 
Decision: Accept (Spotlight)