Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper addresses the problem of speeding up adaptation training by introducing an adversarial training algorithm that is compatible with large batch training. Previously, while being able to significantly speed up adaptation training, large batch training has been shown to harm the model's ability to generalize. Adversarial training algorithm helps, but the inner optimization steps brings a major computation overhead over other optimization algorithms. Motivated by these insights, this paper proposes ScaLA, which (1) sticks to the adversarial solution but reduces gradient computation steps to 1, and (2) introduces a delay in perturbation injection, in order to bring down the computation overhead. Experiments show that the proposed training regime is faster than previously-proposed solutions like FreeLb and LAMB, while also improving the performance of the model, which indicates stronger generalization capabilities.
strengths_and_weaknesses: Strengths:

- The research problem is well-motivated by the related work survey and the preliminary analyses.
- The presentation of the solution is mostly clear, with the benefit of each part being clearly stated.
- Experiments show clear benefit over existing methods in terms of adaptation training speed, while also showing strong performance.

Weaknesses:

- It is unclear to me if any of the modification here is specifically targeted to speed-up the adaptation training process (as opposed to pre-training).
- There are a few non-ideal presentation choices, see below.

Some more detailed comments:

- Eqn. 2: the choice of using $x$ and $y$ as model parameters and perturbed embeddings is a very confusing one, especially since they were used as instances from the training data in (1).
- L130: It would be nice to introduce what exactly with and without communication means, or provide some citations for it. I have trouble understanding how would the non-communication case work.
- Algorithm 1 L9: I don't understand $\prod_{\omega}$ means here.
questions: Please respond to the points made in the weaknesses & detailed comments part above.
limitations: Adequately addressed to the best of my knowledge.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a ScaLA method to improve the computational efficiency of the fine-tuning of pre-trained language models while preserving generalization. ScaLA uses a large batch size to lower the total training iterations and employs adversarial training to mitigate the generalization drop brought by using a large batch size. This idea is quite counter-intuitive to me as it aims to lower the computational overhead, but introduces a more time-consuming adversarial training part. However, by using techniques such as one-shot adversarial examples and delayed perturbation injection, ScaLA is able to accelerate the adversarial training part and thus lower the overall computational overhead, which is empirically justified by experiments over a range of natural language understanding tasks.
strengths_and_weaknesses: Strengths:
1. The paper is well-written and the method is clearly presented.
2. The empirical result looks strong.

Weakness:
1. The proposed method looks like a naive combination of two existing techniques and thus to some extent incremental.
2. It would help if the authors can include a more in-depth analysis of why large batch size does harm to generalization and how this issue can be addressed (or even best addressed) by adversarial training. The current version focuses more on empirical validation.

===========After Rebuttal===========

Thank the authors for the response. After reading all the reviews and responses, I decided to keep my rating unchanged.

questions: Kindly refer to my suggestion in the weakness part.
limitations: I do not see any potential negative social impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper aims to improve the efficiency of finetuning pretrained large transformers through adversarial training. It resorts to large mini-batch training and proposes several methods to reduce the computational overhead and address the optimization challenges of large mini-batch training. The paper provides an extensive empirical analysis confirming similar challenges raised by recent works. The proposed ScaLA algorithm combines several optimization tricks such as delayed and one-shot perturbations. On the BLUE benchmark, ScaLA substantially reduces the overhead of finetuning BERT and RoBERTA without hurting accuracy. Convergence analysis of the algorithm is provided. The ablation study is sound.

The paper reads like a collection of techniques mostly proposed by previous works. For example, as the paper points out, several key ideas are already established in computer vision applications, and now are adopted to textual data by this work. To be clear, I have no doubt that such efforts are of great use in real-world applications of large models. Just that I am not sure this paper is a right fit for the NeurIPS community. 

====

After author response: with the additional results and analysis, as well as the experiments planned out, the paper is definitely in a better shape. I have increased my rating to 6.

strengths_and_weaknesses: Strengths:
- Addressing the challenges of large mini-batch training is important and timely.
- Technical aspects of the paper are reasonably clear.
- The proposed method is simple and effective.

Weaknesses:
- The paper heavily builds on the findings of existing works. It is hard to argue for technical novelty.
- I think the paper could be better organized. The current presentation has several separate ideas floating around lacking a clear central theme. Besides, I find the detailed discussion of previous work in the middle of the method section very distractive. 
- The paper should include experiments on other pretrained transformers, as well as other datasets.

questions: - The notation around Eqs. 1 and 2 is confusing to me. What are $x$, $y$, $Q$, etc.? Shouldn’t $r$ be $\mathbb{X}\times\mathbb{Y}\rightarrow\mathbb{R}$? Why is g’s domain $\mathbb{X}\times\mathbb{Y}$?
- Could the authors explain what PGA is?

limitations: The paper does not explicitly discuss the limitations and potential negative societal impact.
review_title: null

Meta-review: metareview: This paper presents ScaLA, a method to improve the efficiency of finetuning of a pretrained language model using adversarial training. Experiments confirm that ScaLA allows faster finetuning compared to standard approaches without reducing accuracy. 

I think this is a nice approach that can have a lot of impact for those who would like to finetune large language models. However, the reviewers have concerns regarding novelty of the paper and would like to see the proposed method to be tested for other larger scale pretrained models as well as more analysis. I encourage the authors to incorporate these changes and resubmit to another conference.
confidence: Less certain
Decision: Reject