Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper studies the effectiveness of self-pretraining, meaning applying pretraining *objectives* like masked language modeling to *datasets* specific for a single task. Remarkably, self-pretraining often achieves performance comparable to the conventional approach of pre-training on generalist web-scale text. This casts doubt on the proposition that knowledge transfer from large text corpora is at the core of why pre-training is so effective in NLP.

Contributions:
- The paper compares self-pretraining to using an off-the-shelf model like ELECTRA, and finds that the two often achieve comparable results.
- Pre-training on other task-targeted datasets (rather than generalist web corpora) is also shown to provide significant gains
- Self-pretrained models are observed to have a different distribution of errors compared to off-the-shelf pretrained models, but simple ensembling is not able to take advantage of this to get better results
strength_and_weaknesses: Strengths:
- Experiments include a variety of text domains and dataset sizes, and the core findings of the paper hold across all data conditions
- Both off-the-shelf and randomly-initialized models are included as points of comparison in the experiments, providing appropriate context for interpreting the effectiveness of self-pretraining
- Understanding the reasons underlying the effectiveness of pre-training can have far-reaching implications given the current state of the field. Providing concrete evidence that casts doubt on the popular claims of "knowledge transfer" contributes to a experimentally-grounded understanding of the topic.

Weaknesses:
- The ensembling results appear to have ensembling take place over probabilities, rather than discrete predictions -- at least that appears to be the case given that only two models are being ensembled. Is there any way to apply majority-vote style ensembling instead, to get around the issues with uncalibrated model confidence levels?
- The discussion on how to construct artificial long-term dependencies is very interesting, but it also is a little bit orthogonal to the remainder of the paper; at least with the current framing. The paper would be stronger if it could provide a more cohesive narrative in terms of incoporating this section.
- The use of BERT-base scale models naturally leaves open the question of whether anything changes at larger scale. While the fine-tuning paradigm becomes effective at these scales, recent research suggests that zero-shot capabilities tend to emerge at larger model sizes. Given that zero-shot capabilities are naturally connected to investigating the role of knowledge transfer in pre-training, there is a natural unresolved question of whether self-pretraining would fall behind if the experimental setup is upscaled by at least an order of magnitude. This is, of course, understandably hard to assess without enormous compute expenditures. 
clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow and the writing is clear. A comprehensive investigation of self-pretraining is (to my knowledge) novel in the NLP literature, and the related work is addressed in the paper.
summary_of_the_review: Overall the paper contains a thorough and comprehensive experimental evaluation of self-pretraining, and the result that self-pretraining can be as effective as off-the-shelf models can have important implications for our understanding of why the pre-training paradigm is as effective as it is.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper investigates the use of downstream datasets as pre-training corpora and finds that for small ELECTRA models, as well as RoBERTa-base, downstream datasets are competitive to large-scale general-purpose pre-training corpora. Furthermore, some downstream datasets (esp. Yahoo Answers) are good pre-training corpora for most considered downstream tasks. Experiments show that their pre-trained models tend to make different errors than off-the-shelf models after fine-tuning. Furthermore, the authors find that sentence contiguity is important when pre-training on sentence-pair datasets, and experiment with using TF-IDF to construct better pre-training examples.
strength_and_weaknesses: Strengths:
* The paper is well written and easy to follow.
* The authors conduct a thorough empirical study of pre-training on 8 downstream datasets. During fine-tuning, they compare their models to random initialization, off-the-shelf pre-trained models and task-adaptive pre-training. The authors find that in most cases the downstream datasets can also serve as a good task-specific pre-training datasets.
* While most of the paper focuses on a small ELECTRA model, the authors consider how the results transfer to a larger RoBERTa-bert model, and show that similar observations can be made.

Weaknesses:
* In the introduction the authors make the strong claim that “[...] the benefits from pretraining in this case cannot be attributed to knowledge transfer”, while in Section 2 the authors are more nuanced and allow for the possibility that “sufficient knowledge is [...] present in the downstream dataset”. It seems possible that the models can acquire linguistic and factual knowledge from these datasets during pre-training, which cannot be learnt efficiently from only predicting the downstream task labels. Since “knowledge transfer” arguments are concerned primarily with the type of knowledge transferred, rather than the source of knowledge, the authors would have to show that certain probing results cannot be replicated in their pre-trained models to support the claim from the introduction.
* The observation that small task-specific pre-training corpora can bring strong benefits for a particular fine-tuning task was already made by Yao et al., 2022, which is not referenced in the paper. It is also relevant to Section 7, as Yao et al. construct examples by concatenating top-k BM25 retrieved documents, which is similar to using TF-IDF. Another relevant reference in Section 7 is Levine et al., 2022, who propose a similar retrieval heuristic for constructing better pre-training examples.
* The name “self-pretraining” seems unfortunate since (a) it evokes parallels to self-training, where a model is used to generate pseudo-labels (see Amini et al,. 2022), and (b) confounds general, task-agnostic pre-training with task-specific pre-training.
* Regardless of the RoBERTa-base experiments, it remains unclear how the results transfer to auto-regressive models, and how relevant they are to the web-scale pre-training of very large language models. While experiments would be unrealistic here, I would appreciate a fairer and more in-depth discussion in the paper.


Yao et al., NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework, ICML 2022

Levine et al., The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design, ICLR 2022

Amini et al., Self-Training: A Survey, 2022

clarity,_quality,_novelty_and_reproducibility: The paper is well written and presented, but limited in terms of novelty and originality.
summary_of_the_review: While this paper features a series of careful experiments, I do not think that the findings are particularly surprising or actionable, and I have some concerns about its claims.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper conduct experiments on pre-training on downstream datasets, rather than the traditionally-used large upstream datasets. The authors explore a variety of text classification settings, showing that pre-training on the downstream text classification dataset yields results on the downstream text classification dataset that are competitive with the traditional upstream pre-training corpora.

This paper then evaluates whether these datasets trained on downstream datasets end up being overly downstream-specific, or if they can still provide strong performance when fine-tuned on other downstream corpora. Surprisingly, the authors find that pre-training on one downstream dataset can still provide benefits on other downstream datasets over random initialization.

Furthermore, the work evaluates whether these models trained on upstream vs downstream datasets make similar errors. Although their errors are not necessarily correlated, ensembling the models does not lead to substantial gains.

Lastly, the authors explore a variety of ways of trying to better leverage downstream datasets for pre-training, including some heuristics for increasing GLUE score by increasing the length of dependencies.
strength_and_weaknesses: Strengths:
-  Interesting research question
- Thorough experimental setup, a variety of text classification datasets examined. Compared against reasonable baselines (e.g., random initialization).

Weaknesses:
(1) It's not clear that the results answer the question of "Our results suggest that in many scenarios, performance gains attributable to pretraining are driven primarily by the pretraining objective itself and are not always attributable to the incorporation of massive datasets"
  - What happens if you sample upstream corpora to match the size of the downstream dataset, and then compare pre-training on downstream vs upstream with similar data sizes?

(2) Related to (1), these datasets are all text classification datasets that are pretty similar, so it's somewhat unsurprising to me that pre-training on downstream corpora works well when fine-tuning on other datasets. Furthermore, for many of the benchmarks, the performance of RandomInit is already quite high and the gains from pre-training are pretty small already, so maybe it's not so surprising that pre-training on _anything_ helps.
  - I think the analysis of model size is quite helpful, since a natural question is whether or not these results are due to the ELECTRA small model being too small to benefit from pre-training on massive upstream datasets. But another concern in this vein that these downstream datasets are too simple to really see many gains from pre-training upstream datasets.
  - It'd be nice to see performance on other types of downstream datasets, like QA. For instance, even looking like something like MNLi performance, it seems like performance is comparable to upstream datasets when pre-training on similar sentence-pair datasets (e.g., QQP or PAWS), but is far lower when pre-training on other downstream datasets that aren't as similar.

Questions:
- Why is the performance of PAWS on RandomInit 50%? Is this a coincidence? I'm a bit surprised that it's so low.
clarity,_quality,_novelty_and_reproducibility: - I found this paper to be quite clear and readable.
- I do think the results are somewhat surprising and novel, although there's a bit of overclaiming (since I suspect that these results are largely due to the similarity of the downstream datasets).
- The work seems reproducible, with details in appendix A about the pre-training and fine-tuning process.
summary_of_the_review: I think this paper conducts an interesting analysis and points out a conclusion that makes sense in hindsight, but I don't think many would have realized a-priori. However, I do think that this conclusion could be better-qualified (e.g., this is not a general conclusion, but one about text classification datasets). Furthermore, it's not clear to me that this paper directly answers the question of whether pre-training data scale or pre-training objective is the key to better transfer performance. Despite this, I think the work is thorough and interesting and would be of interest to the ICLR community.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper follows a line of papers who question the importance of massive corpora for pre-training MLMs. The authors show that by pre-training MLMs on downstream corpora (which are far smaller than standard pre-training corpora), they reach similar performance on the corresponding downstream task to that of off-the-shelf MLMs. Surprisingly, the authors also show that these models also transfer quite well to other tasks. They then show that model prediction of models trained on these downstream corpora is typically quite different compared to standard models (though this does not translate to strong ensemble results).

strength_and_weaknesses: 
Strengths:
- The presented findings are very interesting, and are important for questioning some of the basic assumption of the field of NLP in recent years: the need of huge, typically web-crawled text corpora for training MLMs. These results can help in mitigating some of the biggest problems of the field nowadays: training on huge corpora which potentially contains biases and toxic language, and in general makes training extremely expensive in terms of energy, time and money. 

- The authors present an extensive set of experiments, showing that the their findings translate to different tasks, to transfer learning, and to different architectures. 

Weaknesses: 
- Despite the very impressive results, I think the elephant in the room, which I was surprised not to see any mention of in the paper, is how much do the results have to to do with the corpus being one of the downstream task corpus, or simply a smaller corpus. That is, a baseline of pretraining on a general pretrain corpus of similar size to the downstream corpora is missing and is very important to shed light on the results here. Is the conclusion from these results basically that you need far less pre-training in order to get good finetuning performance? I am aware that size doesn't explain it all, but I am also far from convinced that the corpora being downstream corpora makes an important difference here. 

- The error consistency experiments are interesting, but raise a few potential concerns. First, computing the prediction difference should be done on the validation set, not the test set (especially given the attempts to use this information to do smart ensembling, which could lead to overfitting the test set). Second, there is no evidence presented in this paper that using the model confidence to select the correct model should lead to good ensemble performance. Are models more confident in their correct predictions compared to their wrong predictions? This is a general desired property of models, but not one that is guaranteed and in particular not one that is explored in this paper (at least not reported), which makes it it not very surprising that these experiments failed. Finally, I am not sure I agree with the conclusion (stated from the conclusion section): "The errors made by such self-pretrained models on the downstream tasks are *significantly different* from the ones made by the off-the-shelf models pretrained on upstream corpora." The error inconsistency numbers are indeed higher among the two approaches, but still overall quite low in 4/8 of the datasets, and even the higher ones are only around 10%. 

- The corpus reordering experiment (Sec. 7) is interesting, and points to the importance of batching similar documents during pretraining. Nonetheless, it does not seem related to long-term dependencies (as suggested by the title and the intro of that section), as even the authors do not argue that such dependencies exist in the reordered corpus.

- Finally, more a suggestion than a weakness: what about zero/few-shot? The assumption this paper is trying to break, as far as I understand it, is that pretraining does not lead to knowledge transfer. This assumption stands in the heart the strong few-shot/zero-shot results observed in recent years, even with reasonably sized models (e.g., [1,2]). Did the authors consider running such experiments with their minimally pre-trained models? I agree that this might be out of scope for this paper, but something the authors could consider.

[1] https://arxiv.org/abs/2001.07676
[2] https://arxiv.org/abs/2104.05240

clarity,_quality,_novelty_and_reproducibility: The paper is overall very clear. The results are novel to the best of my knowledge and seem quite reproducible.

summary_of_the_review: The findings in this paper are quite interesting and could be of value to the community. However, many of the central claims in this paper are not validated, e.g., the importance of the downstream corpora, the _significant difference_ in the error patterns between the different approaches, and the argument regarding long-term dependencies.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a set of experiments to show that when the finetuning dataset is also used for pre-training there are substantial gains that can be made in the performance on a give task. Of course, when the finetuning dataset is being used for pretraining it is being used without any labels, and only the MLM objective is being used for pretraining. The gains in performance are impressive and they come at no extra data. A new set of experiments that the authors added after the review show that if a similar size of randomly selected dataset is used for pretraining, most of the performance gap be covered.

Strengths
* clearly written paper
* thorough set of experiments
* good empirical performance gains

Weakenesses
* Similar gains in performance were also shown by Yao et al
* This could be more about the size of the dataset rather than the data being the same finetuning data.
* What happens if the size of the pretraining data is substantially larger? can these gains diminish
justification_for_why_not_higher_score: * The takeaway message from the paper is not that useful 
* There are some doubts on how the performance can scale when larger scale of data is in picture 
justification_for_why_not_lower_score: n/a
Decision: Reject