Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: A parse-tree based curriculum learning framework to learn a pre-train model.
strength_and_weaknesses: *strength: 
1. A working curriculum algorithm for vanilla masked language modeling.
2. parse-tree based masking seems novel to me (but I'm not familiar with prior works)

*weakness:
1. the experiment is a bit limiting, as only a bert-Base is tested. It's unclear if this method can scale up to larger model and datasets.
2. The scheduling of the curriculum learning seems a bit arbitrary and not systematically investigated.
3. unclear how the method can generalize to languages where a parse-tree is not available.
clarity,_quality,_novelty_and_reproducibility: *clarity: the paper's algorithm is clear, except for how the training corpus is actually parsed (which parser is used)
*quality: the result is solid, but it is lacking some ablations studies to really understand what caused the improvement (the curriculum of vocabulary or the parse tree).
*novelty: seems quite novel to me
*reproducibility: I didn't find how the training corpus is actually parsed, other than that it seems reproducible.
summary_of_the_review: The work is novel and has good result on a small model and dataset. The main weakness is the lack of ablation study and testing on more models and datasets.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper improves pre-training of MLM models (e.g. BERT) with a curriculum learning strategy, where not frequent words are substituted with their constituent labels. Rarer words are then introduced in pretraining in subsequent pretraining stages, until all words are substituted back to their original form.  Their approach shows minor improvements across multiple tasks including NER, QA, POS tagging and Parsing.
strength_and_weaknesses: Strengths: their approach is novel and shows consistent results over multiple benchmarks,

Weaknesses: results are very close between models in multiple benchmarks, it would be good to show whether they are within the range of noise, especially models trained on the same data.  Table 4 for example is extremely close.
clarity,_quality,_novelty_and_reproducibility: 
- This is very unclear in the abstract: "This is achieved by substituting syntactic constituents for rare words with their constituent labels." The same it is much clearer a the end of page 1.

- Section 2.1 could be just a citation.

- The results of BERT (Devlin et al. 2019) are not really necessary given that you are reporting results with your BERT reimplementation.




summary_of_the_review: Some additional qs:

- Have you thought about using simpler synonyms for rare words instead of the constituent approach?

- Have you thought about using sentence with simpler syntactic structure in the initial phases of pretraining?  

- Would be good to cite relevant work on syntactic information inherently present in BERT like models: https://aclanthology.org/N19-1419/, https://aclanthology.org/2021.emnlp-main.118.pdf , and potentially apply these probes to see whether they actually improve.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a curriculum learning for language model pretraining: training with high-frequency tokens first, and low-frequency tokens later. They are 4 stages of pretraining:
- 1. First 3 stages: training with the text whose low-frequency tokens are replaced with their syntactic label, for example, NP for noun, CD for number, etc. 
By changing the replacement frequency threshold, the pretrain task difficulty can be controlled. This paper introduces 3 thresholds and split the first 600k training steps into 200k respectively. (the word frequency thresholds are <0.5k, 0.5k~3k, 3k~18k)

- 2. Final stage: training with the unplaced original text. This stage is the final 400k steps of the training.

Experiments show the BERT trained with their proposed method can outperform the BERT trained without it. 
Evaluation tasks include the GLUE benchmark, CoNLL03, SQuAD 1.1 and 2.0, and WSJ pos-tagging and constituency parsing.
strength_and_weaknesses: Strength: 

- 1. The idea is intuitive and well-described.
- 2. The results show the improvements clearly and support the method.

Weakness:

- 1. See the originality concern in the next section.
- 2. Whether the proposed method is robust to different hyper-parameters of the curriculum training is not answered. For example, the current setup is 200K steps per stage, and 400K for the final stage. I would suggest showing the impact of these hyperparameters.
- 3. Reported finetuning results should be averaged across 3 or 5 times of finetuning-evaluating. In the current submission, I cannot find details of this part and I assume that the result of each task is from a single run.
- 4. Only experiment with the BERT base, which is quite small (110M) in the context of pretraining in 2022/2023. Even for non-pretrained LM papers, 110M is a quite small model. For example, [1] trains 257M models. [2] trains 261M models. 

[1] Bai, He, et al. "Better Language Model with Hypernym Class Prediction." Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.

[2] Press, Ofir, Noah A. Smith, and Mike Lewis. "Shortformer: Better Language Modeling using Shorter Inputs." Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.
clarity,_quality,_novelty_and_reproducibility: This idea of two-stage curriculum training and low-frequency replacement has been introduced by Bai et al in ACL 2022. This ACL paper finds that two-stage curriculum training can help language modeling, by replacing the low-frequency words with labels, training LM first with the replaced text, and finally training the LM with the original text. The method is basically the same. The differences are: 
- Bai et al use WordNet's hypernym relation to find the replaced label for the low-frequency tokens, while this paper uses a parser to get the labels for the low-frequency tokens.
- Bai et al train an LM and evaluate the perplexity with WikiText103 and Arxiv, while this paper pretrain an LM and evaluates with downstream tasks. 

In summary, in this submission, I did not find any discussion of this ACL paper. I think they are quite similar in terms of the method and read like an extension of Bai et al. Furthermore, even though this submission focus on pretraining, the model size is smaller than Bai et al.
Some claims of originality and contributions need to be rephrased and corrected.

[1] Bai, He, et al. "Better Language Model with Hypernym Class Prediction." Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.

[2] Press, Ofir, Noah A. Smith, and Mike Lewis. "Shortformer: Better Language Modeling using Shorter Inputs." Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.
summary_of_the_review: Overall, I like this idea and am happy to see this idea works on BERT pretraining. 

However, this submission misses a key related work that first proposes and uses the same "low-frequency word replacement" curriculum method for LM training. This submission needs to rephrase and correct its claim of originality. For example, the originality is how to replace words with labels (with a parser instead of the wordnet).

It would be great to add experiments to show the different word-replacing method is the key to BERT training. (the syntactic parser v.s. WordNet Hypernym relation)

Also, since this paper tries to show the curriculum training works for BERT, an analysis of the curriculum hyperparameters should be added.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a curriculum learning (CL) method to train BERT, which is treated as blackbox (no need to modify the original implementation). The CL method consists of several stages, each of which is to replace rare words / phrases with syntactical tags (e.g. 1000 with CD). By doing that, BERT will only need to learn frequent words first, then rare words later. The paper shows that CL method helps mitigate the representation degeneration problem of BERT (i.e. BERT is biased towards frequent words). Also, it demonstrates that BERT trained with the CL outperforms the traditional training way on several down-streaming tasks. 
strength_and_weaknesses: The motivation of the paper is quite unclear. Specifically, it is difficult to see, in the paper, the reason CL was chosen. It is true that "there is a more salient discrepancy between the current PLM training and the language learning process of humans", but why is closing this gap helpful? What're specific aspects that the paper is looking for when employing CL? One thing I can see here in the paper, is to deal with rare words, but then the original BERT training already has a quite effective mechanisms, which is the tokenization (resulting in quite a small size vocab, 29k tokens).

The idea of employing syntactic tags to replace rare words / phrases is nice. But CL is not the only way to do. In fact, we can do a simple augmentation: randomly replace words / phrases with their syntactical tags during training BERT. By doing that way, trained BERT will also be aware of syntax and phrasal structures. Following this point, the fact that the new BERT outperforms BERT trained with traditional method could be due to the fact that the new BERT is more aware of syntax, rather than anything related to word frequency. 

The experiments do not reflect fully the intention of the paper, which is targeting LM in general. In fact, the paper demonstrates on only one LM which is BERT-based-cased. 

The analysis in section 4.1 is unconvincing. In Fig 3, we can see that for both BERT 1M and BERT-CL 1M, there is a rare-word cluster that is away from the rest, and mixture of rare-word and frequent-word clusters. It is difficult to see why BERT-CL mitigates the problem of representation degeneration. Should it be clear if methods of Gong 2018 or Gao 2019 are used for a more thoughtful analysis? 




clarity,_quality,_novelty_and_reproducibility: The paper is easy to read but the overall structure is weak because the motivation, and any claims the paper wants to make do not seem highlighted. Especially, the paper can be improved by more convincing arguments why CL is really needed here. 

Although using CL for pretraining LMs is not new, the proposed CL is simple yet original. 
summary_of_the_review: I like the idea of the paper, but the paper should be improved to meet the conf. standard:
1. the motivation is unclear, 
2. the experiment doesn't support the claim (only BERT-base-cased is demonstrated rather than LM in general) 
3. the analysis of mitigating the degeneration problem is unconvincing
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: CLARIFICATION

This was a borderline paper. We conducted a virtual meeting with the reviewers. Two reviewers did not take part in the virtual meeting.

SUMMARY

The paper addresses language model pretraining. It takes a
curriculum learning approach, focusing on frequent subwords
first, rare subwords later.  In the early stages of
training, rare subwords are replaced by part-of-speech tags.
Extensive experimentation shows that the new method modestly
(but clearly) outperforms prior work as reimplemented /
rerun by the authors. The strongest results (even compared
to state of the art for models of this size) are for
"syntactic" tasks (and, oddly, for squad). This makes sense
as the model is trained on syntactic signals not available
in prior work.


STRENGTHS

The idea of replacing input words by part-of-speech tags and
thereby training a model with better syntactic capabilities
is good. It is surprising that nobody has done this
before. If that's true (i.e., there is no prior work on
this), then this would be the strongest reason for accepting
the paper.

(This:
https://arxiv.org/abs/2211.05344
is somewhat similar, but was published on 2022-11-11, so
not relevant for this decision.)


WEAKNESSES

This paper:

Bai, He, et al. "Better Language Model with Hypernym
Class Prediction." Proceedings of the 60th Annual Meeting of
the Association for Computational Linguistics (Volume 1:
Long Papers). 2022.

is closely related. Instead of initially replacing words
with POS tags, Bai et al initially replace words with
hypernyms. Similar to the submitted paper, there is a lot of
replacement in the beginning of training and no replacement
at the end.

So in a sense the submitted paper does what Bai et al do,
except that the "abstraction" introduced in the beginning of
pretraining is syntactic, not semantic.

This limits the novelty of the submitted paper. But doing
syntactically what Bai et al did semantically still clearly
is a novel contribution.

The fact that the authors did not cite and discuss Bai et al
does not instill trust in the paper. I would be inclined to
reject it for that reason.

Another issue with the paper is that there are other ways of
implementing the idea of "abstraction" in pretraining. As
one of the reviewers points out, one could just randomly
replace words (or maybe just rare words) with tags. Why
would that not work as well as curriculum learning?

For this reason, I see the "POS abstraction" as the key
contribution. But, as pointed out above, this contribution
is limited since a similar type of abstraction in a similar
experimental setup was already proposed and evaluated by Bai
et al.

Finally, instead of the simpler idea of replacing words with
POS tags, they actually do a more complex version: they use
a complete parse of the sentence and then replace phrases
with tags, as opposed to words. Unfortunately, there is no
comparison to the baseline of just replacing words. So we
have no evidence allowing us to tell whether the more
complex parsing-based setup makes a contribution.

justification_for_why_not_higher_score: The basic contribution is good, but there are too many problems with this submission. See above.
justification_for_why_not_lower_score: n/a
Decision: Reject