Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a method to mitigate the difficulty of training deep adversarial auto-regressive generators in the ELECTRA self-supervised framework, by extracting the hidden representation from each layer of the deep generator network and feeding it through a mixture of representations function. The resulting model, named adversarial mixture of signals (AMOS), combined with the use of Gumble-Softmax relaxation, effectively stabilized GAN-style adversarial training for ELECTRA-style frameworks. This enabled the possibility of learning better generators and discriminators, by making it possible to feed gradients from the discriminator directly to the generator. The paper provided an extensive set of experiments for self-supervised language learning using GLUE and SQuAD tasks, for which AMOS showed state-of-the-art results compared to similarly-sized alternatives. Furthermore, extensive ablation studies are provided to show that both curriculum setup (as in the ELECTRA-style frameworks) and adversarial setup (as in AMOS, which is new in the paper) improve the results. These studies also verify the effectiveness of stabilizing training using the mixture-of-hidden-representations setup, especially for deeper generators.
main_review: There are several strengths from the paper that make me believe it is a good paper to be published in ICLR 2022:
1. The paper made a significant contribution to idea of using adversarial training as part of the self-supervision signal for language learning.
2. The paper made a impactful finding for practicing adversarial training, that mixture of signals at different depth of of the generator can stabilize ELECTRA-style models trained adversarially using Gumble-Softmax relaxation.
3. The experiments in the paper demonstrated the superiority of adding adversarial training to a self-supervision framework, in that significant improvements can be obtained for similar-sized networks.
4. Good set of ablation studies to show that each component of the model is necessary, especially because the entire model already has many moving parts in addition to adversarial training.

With the strengths being said, I hope to also point out that the paper's application of adversarial training is one attempt in many possibilities, and in many cases it is not clear where the improvements come from. For example:
1. The paper pointed out that ELECTRA framework [1] explored the idea of using REINFORCE [2] as the the way of adding adversarial training signals to the model but observed degenerated results. Compared to this, the paper made 2 changes to the model: 1) using Gumble-Softmax instead of REINFORCE, and 2) using mixture-of-signals instead of straightforward gradient back-propagation. It is hard to know here which one of these actually made the adversarial setup useful. Is it possible to run an ablation study using the combination of REINFORCE and mixture-of-signals to verify whether Gumble-Softmax relaxation is the reason for it to work?
2. There are many past papers that apply the discriminator to some internal representations of the generator instead of on the softmax outputs of the discrete text signal (for example, [3][4][5]). In the practices of GAN for text, these are proven to be more stable than both Gumble-Softmax relaxation and REINFORCE. What are the reasons for the paper to choose Gumble-Softmax relaxation instead? Please discuss.

References:

[1] Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning, ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators, ICLR 2020

[2] Lantao Yu, Weinan Zhang, Jun Wang, Yong Yu, SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient, AAAI 2017

[3] Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, Lawrence Carin, Adversarial Feature Matching for Text Generation, ICML 2017

[4] Jake Zhao (Junbo), Yoon Kim, Kelly Zhang, Alexander M. Rush, Yann LeCun, Adversarially Regularized Autoencoders, ICML 2018

[5] Sandeep Subramanian, Sai Rajeswar Mudumba, Alessandro Sordoni, Adam Trischler, Aaron C. Courville, Chris Pal, Towards Text Generation with Adversarially Learned Neural Outlines, NeurIPS 2018
summary_of_the_review: Novel idea of stabilizing ELECTRA-style language learning using a mixture of signals at different depth of generator hidden representations. Significant contribution to using adversarial training as part of the signal for self-supervised language learning. Good experimental results of the proposed model. Good set of ablation studies for each part of the model. Some unclear reasoning in the choice of adversarial training techniques compared to previous papers.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper present a method for pretraining language model using generator-discriminator training. Similar to ELECTRA model which used a MLM model as generator, which corrupts input text and replace some tokens with Masked token, and to train a different language model to detects the replaced tokens. Different from ELECTRA, they proposed to use multiple MLM generator models, which replace tokens at different difficulty levels. This way, the discriminator is forced to learn higher level of language. In order to allow gradient back-propagation from discriminator to multiple generator, they employed gumble-softmax. The evaluation results on GLUE and Squad indicates better performance than previous arts. 
main_review:   Questions:

1- Figure 1: the generator with more layers (8) gains lower training loss during pretraining, meaning it can better recover the masked token, which means during sampling it should recover the original token. In other words, the replaced-token sequence should have less replaced tokens and it should be an easier task for the discriminator!

2- Are the MLM heads shared across different layers of generator? 

3- Table 2: the ablation on stop-gradient is only evaluated on MNLI and Squad which should degrade in performance. Is this ablation studied on other GLUE tasks as well?

4- the author proposed using gumble-softmax to enable gradient backpropagation from discriminator. In ELECTRA paper, reinforcement learning is used to leverage this. Is the proposed approach can be used using RL too? what is the performance on downstream task?

5- The author mentioned that the discriminator loss is used to train the mixture weights of MLM output to combine a  more difficult signal for discriminator, but it will not update the MLM embeddings. This is confusing, because generator is trained jointly with discriminator, and MLM embeddings are trained via two gradients, one from discriminator, and the other from MLM pretraining of generator! Moreover, in Figure 2, it is shown that discriminator gradient are backpropagated through generator too. In table 2, it is shown that using adv. MLM hurts MNLI matched performance by 0.3 points, whereas other tasks are untouched. what is the performance of this setting on other GLUE tasks?

6- Table 1: why the number of parameters in Base++ setting is larger than Base one?

7- Table 2: all the ablation results are evaluated on MNLI and squad tasks. however, I think all GLUE tasks should be considered in this study to understand the actual contribution of different components of the proposed AMOS.

8- Table 2: in layer switch and random layer configuration, are all MLM layers are pretrained equally? 

9- Table 2: w. separate MLM gen. configuration shows only 0.1 performance decrease on MNLI-matched compared to -stop grad setup. what is the performance on this setting on other GLUE tasks?

10- Figure 3(b): the discriminator accuracy with 4-layer generator have a smooth increase with higher performance than AMOS, which does not use any adversarial training, which seems an intuitive learning curriculum too. what is the explanation on this?
summary_of_the_review: The proposed adversarial approach combined with the multi-generator signal which is composed from different layers of a generator provides a curriculum learning approach to train a better discriminator for downstream tasks. Despite better results on GLUE and Squad tasks compared to previous arts, the extent to which the different component of AMOS contribute to the results are less studied in the ablation. Only two tasks (MNLI and Squad) are selected for ablation of adversarial learning, mixture weights, gradient stopping, etc. The evaluation results can be extended to SuperGLUE as well.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents an extension to Electra-like pre-training strategies that use MLM generator heads positioned at different layers throughout the networks and a single discriminator. Mixture-weights for different MLM outputs are also learned end-to-end via the gumbel-softmax estimator. The authors present evidence that generator-discriminator dynamics are impacted significantly by the generator’s capacity and architecture. The proposed idea, therefore, is to have the discriminator operator on top of a “mixture” of discriminators of different capacities. The authors make this mixture efficient by sharing the generator backbone and having separate MLM heads at different layers of the network and only learning the mixture weights of the representations produced at each layer.
main_review: Strengths:

1. A clever approach to mitigate generator-discriminator dynamics in end-to-end training of Electra-like models by using a computationally efficient mixture of generators.
2. Strong empirical results on many downstream NLU tasks outperforming strong baselines.
3. Fairly thorough ablations for different model components.

Weaknesses:

1. It is unclear to me why the discriminator cannot just optimize gammas to be high for lower layers, thereby making it easier to optimize its own objective. Wouldn’t it, therefore, be better to have the gammas affect the MLM loss as well to help the lower layers learn better.
2. Following up on the previous point 1) Fig 3 (a) is counter-intuitive as to why layer 8 has the highest coefficient followed by 4 and then 6. I would have expected the discriminator to learn weights in the order 4 > 6 > 8 since the only gradient for these parameters comes from the discriminator loss and not the generator MLM loss (b) how robust is the mixture weights to restarts with different seeds and what if we trained from scratch with fixed mixture weights using what was obtained at the end of 120k steps in Fig 3 (a)?
3. Although computationally expensive, it may be useful to establish a performance “upper bound” by training M different generators.
summary_of_the_review: This is a meaningful contribution to Electra-like pretraining approaches and provides fairly significant downstream improvements.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposed a new framework AMOS to enhance ELECTRA-style pretraining. The new framework includes adversarial learning and curriculum learning. Instead of direct adverbial learning via Gumbel-softmax, the author proposed a mix-of-signals framework to mix the signal from different layer of the generator. The authors show the overall performance outperforms ELECTRA and more-recent SOTA COCO-LM by a reasonable margin. 
main_review: The paper is clear written and relatively easy to follow. The paper idea is majorly from 3 aspects: curriculum learning, adversarial training via Gumble-softmax, layer wise understanding of pretrained models. These ideas are reasonable. The results show on GLUE dev and SQUAD 2.0 also looks significant. 

There are two parts of the work I think needs further clarification. First, the author states that the model is based on mix of training signal generator, however, the work is actually using different layer mixture of signals. To make the statement solid, I think another experiment of independent generators with same layers as generators should be conducted. Also, the meaning of vector v in weight calculation is not clearly discussed. Second, the adversarial training does not include re-sample the masked tokens, which may be a strong signal in adverbial training. 

Besides this, as this is a pretraining work. I would expect to see the performance on GLUE test set instead of just dev set which should make the results more convincing. Also, the training time/speed/performance comparison with ELECTRA may be better shown in the work. 
summary_of_the_review: Overall, I think this is a reasonable work by directly improving of the ELECTRA framework. The proposed ideas are clear, straightforward, and sound. The results can be further improvement.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents an adversarial learning framework for pre-training text encoders following ELECTRA-style architecture. The use of single auxiliary MLM broken into sub-MLMs and then combined together with a mixture parameter is novel. Authors spent great deal of effort on the ablation study to show that the proposed framework and design choices perform better than existing state-of-art methods and other alternatives on several downstream NLP tasks.

The paper is clearly written and easy follow. The conclusion of the inductive learning curriculum is well supported by experimental results.
main_review: Strengths:

The paper is well written and easy to understand. Since it builds on the well known ELECTRA architecture, it is easy to identify the main contribution, which is adding more inductive learning curriculum to ELECTRA training. The authors achieved this by using multiple auxiliary MLMs and combining their outputs to generate the input to the discriminative text encoder. To improve training efficiency and further streamline the architecture, all the auxiliary MLMs are derived from the same base MLM with outputs of multiple layers used as its own sub-MLM realization. Finally, the mixture parameter is adversarially learned using the negative of the discriminator gradient. The ablation study is exhaustive providing empirical evidence of superiority of the proposed architecture.

Weaknesses:
There is a typo “date-centric” --> "data-centric"
The paper applied the MLM to only 3 layers (4/6/8) of the 8-layer generator. Do you have an intuition on the effect of increasing MLM heads on the discriminator performance on the downstream tasks? It was shown that using just one MLM head irrespective of the layer is worse, but is there an optimal number of of MLM heads for an N-layer generator network?
summary_of_the_review: I recommend this paper for publication given a satisfactory explanation of the raised weakness. Overall, I think the concepts explored in this paper are novel and will be of interest to the NLP community, and might spur new research in this direction.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a framework for pre-training transformer based LMs a la ELECTRA-Generator-Discriminator style by leveraging a mixture of generators and using signal from the discriminator to guide the learning of the generator. Specifically, the generator is modified to have multiple MLM heads specific to different layers (under the intuition that the deeper layers will act as a stronger generator), and the replacement predictions from each of the layers are combined using learnable weights as the input to the discriminator. To aid learning and ensure the different MLM heads of the generator to function as "independent generators", the gradient is not back-propagated across the entire generator, but only within individual blocks between the MLM heads. For back-propagating the discriminator loss for learning the mixture of generators, the Gumbel-Softmax trick is used to back-prop the loss to the weights (the discriminator-guided selection of different generators can be intuitively linked with GANs). Empirical evaluation is performed on GLUE and SQuAD, and the 12-layer AMOS pre-training (with both the standard base and base++ pretraining configurations) is able to outperform previous strong baselines like ELECTRA and COCO-LM by an average of 1+ absolute point. Ablation studies have been performed to show the empirical benefits of different components: the weights v/s random and simple->hard 3 model curriculum, back-propagating discriminator loss only to the weights and not to the MLM prediction heads, etc.

main_review: Strengths:
1) The paper is very well written and easy to follow. The paper is well-placed w.r.t recent developments in pre-training by presenting similarities and differences with related work.

2) The idea of using a committee of generators to create the replaced token for training the discriminator, so as to provide a diverse signal for learning different complexity levels of RTD is natural and logical (this also can be linked with curriculum learning). The neat trick utilized for modelling AMOS is to create multiple generators from the same single monolithic generator by means of individual MLM heads at different layer depths which : (i) reuses computation and is efficient, and (ii) exploits underlying representations learned by lower layers to make the deeper layer MLMs stronger. 

3) The empirical results of the AMOS pretraining approach are strong. Despite several discussions in the NLP community on using GLUE as an evaluation benchmark, a 1 absolute point improvement using a 12-layer base model is significant over the previously strong baseline of COCO-LM (which additionally uses the InfoNCE loss).

4) The ablation studies on MNLI and SQuAD help justify the modeling decisions made in the paper w.r.t the learnable weights, the use of stop-gradient between layers, not using the adversarial discriminator loss to train the learnable weights, etc.  


Weaknesses:
1) The ablation studies on MNLI and SQuAD are provided without standard deviation error bars. The improvements of AMOS over the ablated components is rather small, and thus it is important to present the error bar estimates to ensure the statistical significance of the results. 

2) For Table 3, the experiments are performed using independent 4,6 and 8 layer generators to establish the point that the pretraining signals provided are diverse. A more appropriate experiment would have been to perform the pre-training of AMOS once, and then use the 4,6 and 8 -layer MLMs (having common transformer blocks) and then see whether this diversity in probing performance exists. The objective is to show that the 4,6 and 8 layer generator with the shared MLM have diverse probing performance on tasks.


3) While extensive, the experiments are limited to pretraining a 12-layer transformer model. Since the paper presents arguments such as the 12-layer generator being very strong, it is natural to question whether the improvements from AMOS will translate to larger architectures (24 layers) using the same mixture of multiple generators approach. 


4) I think the paper should include a societal impact section and discuss the extensive compute resources that have been consumed for the pretraining runs and the ablation studies. Furthermore, the reproducibility statement is missing details of the compute infrastructure used : Number of GPUs, types of GPU/TPUs, etc.


Questions:
1) How was the \lambda in Equation-(4) chosen for the experiments? Was some form of cross-validation performed?

2) Instead of random masking, do you have any high level thoughts/intuitions on whether the AMOS approach will also provide empirical improvements with targetted masking approaches (For example: REALM that masks specific POS entities for retrieval-augmented MLM)?

3) While the different MLMs in AMOS have different number of layers, they all have the same structural transformer block, and thus are from the same "family" of generators. Since the goal is to increase the diversity and complexity of signal from the generator for improving the strength of the discriminator, it will be interesting to experiment with a different "family" of generators (based on LSTM, word-CNN, etc.). Any thoughts/comments on this?

4) The paper presents the AMOS multi-generator guided pretraining approach for a general k number of generators, but all the experiments are performed using k=3. While I understand that each pretraining run is computationally expensive, I was intrested to know if the authors had any intuitions/initial results from experiments by changing the number of MLM heads k?
summary_of_the_review: The paper makes a novel addition to the ELECTRA-style LM pretraining direction by using multiple generators to guide the learning of the discriminator, and in turn use the loss from the discriminator to enhance the generation capability of the generators. The idea is simple, easy to comprehend and evaluated on the standard benchmarks used for LM pretraining research. The results are strong and show a 1 point absolute improvement in average GLUE score over previous baselines.

The results of the ablation study are not very conclusive, and raise some doubts about whether certain modeling decisions are essential (and if they are making the AMOS modeling too complex). There are some weaknesses in the experiments to justify the need for different number of layers of generators, and concerns over generalization to different values of k and the size of the transformer architecture.
review_title: null

Meta-review: This paper received six reviews, consisting of three 8s two 6s and one 3.
The reviewers generally felt that the proposed Electra-like pretraining provided fairly significant downstream improvements.
Additional ablations were provided to during the author response period and other author responses were sufficient to cause scores to rise during the discussion period.
The vast majority of reviewers recommended accepting this paper and the AC also recommends acceptance.
Decision: Accept (Poster)