Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The current research piece introduce combines existing pretrained language models as DeBERTa and ELECTRA into a single model.
In addition, authors introduce GDES, a technique that the shared embeddings from the generator and discriminative models interfere with each other by avoiding propagating the gradients due to the discriminative loss. 
The model is evaluated in a series of standard NLU dataset, particularly GLUE benchmark, plus other dataset for specific tasks such as QA, NLI and NER varying models sizes showing the effectiveness of their proposed method. 



strength_and_weaknesses: Strength 
The paper is sound and make sense.
DeBERTa and ELECTRA are both top level PLMs, but with significantly different training regimes so its combination makes sense to lead to further improvements.

Weaknesses
Novelty is weak. 
Combining (any) models might the actual contribution here. Picking two SOTA models and combining them is not clear to have enough novelty.
The GDES idea is interesting, but doing a thoughtful ablation study with any pairs of PLMs instead will prove authors point. 

clarity,_quality,_novelty_and_reproducibility: 

Experimentation is problematic. A fair comparison against DeBERTA and ELECTRA alone should involve training both models, again with the same training data. Authors do not state (at least not clearly) that has been the case. 
summary_of_the_review: Despite the jump in performance shown by the experimentation the paper do not provide enough novelty..
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to enhance the DeBERTa model with the more sample-efficient replaced token detection (RTD) pre-training task. 
To further improve the training efficiency and performance of the vanilla embedding-sharing strategy (ELECTRA) for the DeBERTa + RTD setting, a new gradient-entangled embedding-sharing (GDES) alternative is leveraged. Experiments across different model sizes and downstream tasks demonstrate the effectiveness of the combination of DeBERTa and RTD and the newly introduced GDES strategy.
strength_and_weaknesses: Strengths:
1. A new pre-trained language model is proposed, which can combine the merits of DeBERTa and ELECTRA by enhancing DeBERTa with the RTD pre-training task and an extra GDES strategy to improve the vanilla embedding-sharing strategy from ELECTRA.
2. A few strong pre-trained backbone alternatives will be released that can achieve very competitive performance across different model sizes, downstream tasks, and settings.

Weaknesses: 
1. I am not convinced that the newly introduced GDES can improve the training efficiency of the pre-trained models. It has been well-proven that RTD is a sample-efficient pre-training task with thorough comparisons of pre-train FLOPs to other pre-trained language models. However, this paper only utilizes the MLM loss curve of the generator to demonstrate the efficiency of the whole pre-training procedure of both the generator and the discriminator. In real applications, we mainly utilize the larger discriminator for downstream task fine-tuning.
2. The technical contribution is limited. RTD is borrowed from ELECTRA, and I am not sure whether the GDES strategy can generalize well to other MLM + RTD setting (e.g., ELECTRA), except for DeBERTa + RTD.
clarity,_quality,_novelty_and_reproducibility: This paper is well-written, with adequate training details for reproducibility.
This paper presents a novel and effective embedding-sharing strategy for the DeBERTa + RTD setting.
summary_of_the_review: The newly proposed pre-trained language model achieves very competitive performance with the existing RTD pre-training task and a newly presented embedding-sharing strategy.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper trained a new DeBERTa model (named NewModel) which incorporates two additional ingredients: 1) The ELECTRA-style training; 2) The token embedding sharing mechanisms, where the authors proposed gradient-disentangled embedding sharing. The author empirically demonstrated that the existing token sharing is not efficient and hurts model performance, and then show that by not propagating RTD loss to generator, the training can be much better. 3)The author also pre-trained a multi-lingual model and observe larger improvement over strong baselines compared to English models.
strength_and_weaknesses: The strength of the paper is the following:
1) The goal of the paper is clear. The authors wanted to improve over existing DeBERTa model, with new training approach.
2) The idea of "gradient-disentangle" is solid and novel. The author drew the conclusion based on 3 evidences: the average cosine similarity of token embeddings, the convergence of MLM loss, and the downstream task comparison.
3) The experiment results demonstrating the effectiveness on NLU tasks are comprehensive and solid.

The weakness of the paper:
1) The name "NewModel" could be improved -- this name appears to have nothing to do the DeBERTa, and does not accommodate model backbone at all, so it will be confusing to keep track of.
clarity,_quality,_novelty_and_reproducibility: The paper is very well written, easy to follow, and the argument/evidence/experiment results are solid.
summary_of_the_review: I recommend acceptance of this paper, for the following reasons:
1) The gradient-disentangled embedding sharing idea is novel and well supported by three aspects of evidences.
2) The improvement of the model performance is solid and significant, both from the English model and multilingual model.
3) The paper has excellent quality and clarity.
4) While ELECTRA is a known idea, the paper does not merely using the method as it is.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper shows that embedding sharing in ELECTRA hurts the training dynamics and proposes a gradient-disentangled embedding sharing method. This method achieves better results in downstream tasks than both embedding sharing and separate embeddings. The paper then uses this ELECTRA objective on a DeBERTa model architecture, showing best in class performance for models of similar size and often surpassing bigger models across many NLU tasks. They also show that this outperformance extends to smaller model settings and to multilingual models.

The main technical contribution of the paper is in identifying the ELECTRA gradient-sharing issue and in proposing an effective scheme to replace it. While the tests of the combination of ELECTRA and DeBERTa done in the rest of the paper presents less novelty, the high performance of these models for these size means they could be useful to the wider community and so this empirical contribution should not be disregarded.

strength_and_weaknesses: Pros:
- The embedding sharing issue and its fix is discussed clearly and well supported by the author's experiments.
- The results of "NewModel" are empirically quite strong.
- The paper's writing and structure is quite solid. See below for minor typos/comments. 
- The authors have done a good job at providing experimental details.

Cons:
- The name "NewModel" across the paper is really not ideal. There are three possibilities: 1) The authors really do intend to adopt this name 2) a macro failed 3) A name was not found in time for the deadline and the authors feared that the DeBERTaV3 name would be more likely to raise comments about the lack of novelty than "NewModel". The "NewModel" will then be replaced upon acceptance/finding a new name. I am not very comfortable with the latter as I feel like it games the system. I will give the benefits of the doubt in this instance.
- I am not a fan of the comparison to larger Megatron models. While it is a larger model, its performance is abnormally low for a model of this size. I think the other comparisons are enough to make your performance clear.
- Minor: Can you confirm that the step time for GDES is only marginally different than ES? It would be great to have this in the paper as further context for Fig 2.


Minor writing issues:
* where the generator’s and the discriminator’s objectives *interference* with each other and slow down the training convergence -> interfere
* section 4.1 first sentence: you likely mean GDES not DES. Also I would spell out DA as disentangled attention as all readers might not remember what it stands. Alternatively, use the abbreviation in Section 2.2.
* page 1 remove space "(Clark et al., 2020) ."

clarity,_quality,_novelty_and_reproducibility: * Clarity and reproducibility should not be an issue
* Quality-wise the paper is well-executed. 
* I found the GDES contribution insightful. I could see criticism about it being the only technical contribution but that was not too much of an issue for me.
summary_of_the_review: The GDES contribution is useful and the final model in this paper achieves very good size/performance tradeoffs which will be useful for the wider community. Despite limited novelty and some minor issues highlighted in the Cons, I think it is well executed and slightly above the threshold.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The submission proposes a new pre-training technique that combines ideas from DeBERTa and Electra. Making these idea work together further required the addition of a novel 'gradient disentangled embedding layer'. 

The approach is a straightforward but logical evolution of existing ideas, and achieves very strong results. The authors have done a good job with the response and updated submission, which addresses most of the main concerns.

The main weakness is a somewhat limited technical contribution, however overall this is useful work that pushes the current state-of-the-art. 
justification_for_why_not_higher_score: The technical contribution here is quite limited, and the paper is mostly a combination of existing ideas.
justification_for_why_not_lower_score: While the ideas here are mostly familiar, the execution is good and some novel contributions were required to allow this work to push the current state of the art.
Decision: Accept: poster