Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors explore synthetic pretraining of T5 style language models. They investigate previously proposed synthetic tasks and introduce simpler synthetic tasks that they find to provide similar benefit to the more complicated previously proposed synthetic tasks.
strengths_and_weaknesses: The authors find that simple synthetic tasks can perform similar to synthetic tasks from the literature, which is original and interesting. Unfortunately the authors do not touch on key questions opened up by their results, such as investigating if results hold on larger models (ie. if the percentage of the gap between Random Init and Pre-trained T5 closed by synthetic pretraining remains constant) or if continued real pretraining with the initialization obtained from synthetic pretraining can reduce the amount of data required for real pretraining. Thus the work appears to be in a preliminary, though encouraging state that is relevant to inform further research.
questions: As you find that synthetic pretraining provides you with a model performing "in between" of random initalization and real pretrained model, have you considered taking the synthetically pretrained model as initialization for real pretraining? If so, are you able to reduce the number of pretraining steps on real-world data while achieving same or better quality?
limitations: The authors only evaluate on T5-Small with 60M parameters. It remains unclear to what degree synthetic pretraining would help on larger models. The authors also do not investigate to what degree synthetically pretrained models can be continually pretrained on real world data and if such continued pretraining can close the gap in performance to the pretrained T5 with less real world data than what the T5 model saw during pretraining.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper provides a series of empirical studies to find the key components in synthetic pre-training for language tasks. Surprisingly, it reveals that two simple synthetic tasks: 'set' and 'identity', can achieve similar performance compared with more sophisticated ones. It also shows that using the statistics of the pretraining weight to initialize the downstream model can also result in similar performance.
strengths_and_weaknesses: ### Strengths

1. The experimental analysis is designed coherently and sufficiently. It enumerates a series of simple synthetic tasks and finds two surprisingly simple but effective strategies: set and identity. It also studies how to improve the initialization strategy using only the statistics of the pre-training weight. All these findings may provide future research with ideas on pre-training as the title suggested.
2. The overall writing is coherent and the proposals are easy to follow.

### Weaknesses

1. I am a little worried about the technical novelty. On the one hand, all the findings are based on experimental studies without specific designs. For example, it is hard to tell why 'set' and 'identity' functions are more suitable than others. Missing such explanations may result in limited technical insight. On the other hand, the focus on the normalization layer in Sec. 5.3 cannot be viewed as brand-new. For example, a lot of studies have revealed that training only the parameters of batch normalization layers can also result in an ImageNet classifier with reasonable accuracy.
2. Some discussions are confusing to me. For example, in Line 197 to 200:
> We note that it does not close the possibility that better initialization statistics is the sole cause of benefits because there exist other initialization schemes such that the statistics of the pre-trained model are a subset of the statistics from the initialization scheme.

    I could not get the connection between the reason and the result.

Despite some concerns mentioned above, I find that the overall contribution of this paper is strong enough to bring insights on synthetic pretraining into the research community. Thus, I vote for 'weak accept' temporarily. Discussions on the above points are warmly welcome. 
questions: 1. More insights on the success of the 'set' and 'identity' function.
2. Clarity of the discussions in Line 197 to 200.

Please refer to the previous section for more details.
limitations: I do not see issues in this aspect.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper investigates what properties of synthetic pretraining lead to better performance on downstream NLP tasks. The authors find that very simple synthetic tasks perform at a comparable level to prior state-of-the-art methods. For example, a straightforward identity task where the model must output the same tokens provided as input. This motivates further steps to simplify, where only the use of weight initialization statistics is sufficient to achieve much of the gains of pretraining.

strengths_and_weaknesses: Strengths:

- This is a solid paper, it is written well and provides an interesting investigation uncovering insights that could otherwise be easily overlooked in the community.
- I appreciate how the paper is organized and the journey we go on of further simplification. The process of stripping away complexity piece by piece until we are down to just a hyperparameter for initializing layer norm is in my opinion much more insightful than if the authors had instead chosen to present some new elaborate task that bumped performance by half a percent.
- The experiments follow a natural trajectory and are fairly comprehensive.

Weaknesses:

- The insight into initialization of layer norm parameters is interesting, and not something that I imagine most people would pay attention to without this sort of investigation. One question is whether knowing that this is an important parameter concretely changes the results one can achieve in these pretraining/finetuning regimes. Have the authors seen any net improvement to performance (higher accuracies, shorter training times, etc) by going back and pretraining and finetuning but this time with a better initialized model?
- One part that nags at me is that most of the comparisons of the paper report relative performance along a linear scale (e.g. method A closed 70% of the gap between random init and natural language pretraining). On one hand, it’s not necessarily an inappropriate  way of presenting numbers, but on the other, further improvements to performance are much more difficult than those initial gains. It could be trivial to get most of the way to matching a particular baseline (and in some ways this is shown with the weight initialization results) and yet it might be impossible to close the rest of the gap. This isn’t reflected in the way results are reported and discussed throughout the paper - the reader doesn’t necessarily have any way of knowing how much more difficult it is to get to say, 35.8 from 32.8 even though that certainly seems very close from a baseline of 18.9. And with the random init baseline being so poor, all of the relative numbers look better for it.
- It would have been interesting to see some comments on the relative performance of the various simple synthetic tasks. I found it fascinating that there was a clear distinction of the worst performing tasks being the only tasks where the output did not involve reusing input tokens (count, length, search, longest word). I know there’s only so much real estate for discussion, but it’s not clear why some simple tasks would fare so well while others so poorly, so some discussion along these lines might make sense to include.

questions: - Maybe I missed the details, but how is the random init baseline trained? Is poor weight initialization not something that could be overcome with sufficiently long training?

limitations: The authors do a good job acknowledging that this investigation only scratches the surface of a rich and complex area and there's much more to be done in understanding the benefits of pretraining.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents a new insight into synthetic pre-training rather than natural language pre-training. Compared to the previous work, the paper makes experiments on a wider range of tasks (e.g., semantic parsing, code translation, natural language MRC), present more (and simpler) synthetic pre-training tasks, and explore in depth the source of gain of synthetic pre-training.
This work conducts a comprehension evaluation of synthetic pre-training methods. The key findings are three sides:
1.	Using synthetic tasks for pre-training achieves large gains compared with random initialization.
2.	Using simpler synthetic tasks like Set and Identity obtains similar benefits to existing more complex synthetic tasks.
3.	The benefits of synthetic pre-training can be attained by initializing based only on statistics.

strengths_and_weaknesses: Strengths

It is an important direction to interpret the effects of pre-training methods. This work presents some interesting attempts to study the influence of different kinds of synthetic pre-training tasks that are generic vs. complex and in non-natural language vs. in natural language. It is also good to see that both generic and non-natural language tasks bring comparable gains compared with the existing methods. The findings may benefit future studies on designing better pre-training tasks.

Weakness


[Organization]

Although this work has the merits of comprehensive analysis and interesting findings, this work is not well organized. The motivation is not clear. I would suggest motivating this work by discussing some limitations of the current synthetic pre-training methods. According to the discussions in Sections 4-5, existing synthetic tasks might have disadvantages, e.g., computation complexity/cost, so they are not optimal as the pre-training tasks, and more generic ones (proposed in this work) would be better.

Besides, this paper does not give a clear answer to the core question raised in the introduction, “What properties of pre-training are sufficient for effective gains”. It would be better to clarify it in conclusions.

The experiments are presented without a principle line. The results are listed coarsely, and it is hard to learn from them and figure out how they could benefit future studies. 

[Experiments]

The comparisons in Table 1 may not be fair enough. Some pre-training methods use different training steps and are based on different sizes of training data. It would be better to add some columns to indicate the training steps, data size, or training cost for more clear comparisons. 

[Analysis]

The design and usefulness of the simple synthetic tasks lack theoretical analysis. Why are the simple tasks, e.g., Set and Identity, intuitively better than the existing methods (which may be more linguistically motivated, e.g., Nesting Dependency Artificial Language). How could the model learn from the identity task by copying the input to the output? Is there any supportive reference for the knowledge acquisition process?


[Detailed Concerns]

1.	This paper tries to explain the problem from trying large numbers of different synthetic pre-training settings. However, the performances of almost all of them are limited (i.e. worse than the original). From this perspective, the greatest contribution of this paper seems to be the two somewhat effective synthetic pre-training tasks (Set and Identify). In practical, besides, this paper does not present any definitive or helpful conclusions for future language pre-training. The claims in the paper are weak or somewhat unsure. The above drawbacks make the contribution of the paper weak.
2.	It shows that the LM can learn some simple clues from synthetic pre-training without being exposed to natural language. However, the gain on natural language in very limited against on artificial language. From Table 1, the performances on SQuAD are significantly dropped compared to pre-trained T5 (e.g. T5 77.5/86.0 -> LIME 50.4/62.1).
3.	Considering the great difference, it is possible that synthetic pre-training provides almost no gain for natural language learning. Further experiments are supposed to be done on natural language tasks (e.g. GLUE or GLUE sub-tasks).
4.	There seems to be an attempt to avoid the issue, making most experiments on a summarization task (CNNDM). The clues for summarization can be more easily to learn, rather than for reading comprehension.
5.	It shows that only keeping the pre-trained statistics can retain most pre-training benefit from synthetic pre-training. However, it needs to be justified that the similar situation (e.g. initialize the model with T5 pre-trained weights using the same methods) does not exist on natural language pre-training. Pre-training is supposed to learn the coarse distribution of a certain language.
6.	The pre-training settings is not comprehensive. Is one million data sufficient for synthetic pre-training? If with more pre-training data, the conclusions in the paper can be more credible. Another concern is that if the claims made in the paper is general to a bigger model. It is better to run the experiments on a bigger setting (e.g. 12 encoder layer + 12 decoder layer).
7.	What if we combine some of the great synthetic tasks together?

Minor Issues:

Use \citet{} for inner-text citations. 
Line 63: no need to capitalize “nesting dependency”.
Line 91: what are those “original papers”? References needed. 
Line 166: There is no Figure 4 in the draft. 
Table 2: There is an unnecessary empty line in the table. 

Typo?
Figure 4 tends to be Figure 1? I cannot find Figure 4 in the main paper.



questions: see the weakness/detailed concern 7 above.
limitations: not see
review_title: null

Meta-review: metareview: # Overview

This paper presents a fascinating research question: "What properties of pre-training are necessary for effective gains" in natural language processing settings? The paper also tackles it in a fascinating way: exploring several synthetic tasks in exacting detail. This is an empirical paper whose goal is to consolidate existing knowledge on the topic of synthetic pre-training and pose a scientific question about the nature of pre-training.

What does it look like for such a paper to be publishable? There are several things it _doesn't_ need to do. That includes:
* Setting a new SOTA for anything, synthetic pre-training or otherwise.
* Providing a theoretical account for empirical phenomena.
* Completely or definitively answer the provided research question, which is a massive undertaking that will require an entire research literature, not a single paper.

However, what such a paper does need to do is to be absolutely rigorous in the situations that it purports to address and to follow empirical best practices. That includes:
* Considering a comprehensive range of downstream tasks (that provides a basis for comparison between methods) and considering them across the full range of experiments.
* Reporting all data clearly in ways that fairly state what was discovered, even if some results are mixed or negative.
* Running relevant ablations to dig into the core research question.
* Consider new approaches to the problem inspired by those findings to test the predictive power of those findings.

# Reviews

## Reviewer 6Qmz

This reviewer raises a number of concerns about the paper, some of which I have discarded based on the criteria above. For example, I have ignored the following concerns since they set an unfair bar for a scientific paper taking on a vast question:
* _This paper does not give a clear answer to the core question raised in the introduction._
* _The design and usefulness of the simple synthetic tasks lack theoretical analysis._
* _The performances of almost all of the[ synthetic tasks] are limited (i.e., worse than the original.)_
* _This paper does not present any definitive or helpful conclusions for the future of language pre-training._

However, the reviewer raises several important methodological concerns that undermine confidence in the empirical findings of the paper:
* _The gain on natural language is very limited against on artificial language._ _The performance on SQuAD are significantly dropped._ (AC note: In general, I found describing the data as "recovering XX% of the change in accuracy from pre-training T5" to be a pretty misleading way to exaggerate relatively small improvements of a handful of percentage points over random initialization. If the results are already strong, there's no need to do this. It's not a reason to reject, but it made me fear that there were other things hidden in the paper that I didn't find. The big question for this reviewer seems to be "are these results good enough to suggest that something interesting has been found via these experiments?" It's in the eye of the beholder, and that's part of the challenge of getting in a paper like this.)
* _There seems to be an attempt to avoid the issue, making most experiments on a summarization task._
* The reviewer also suggests some fantastic follow-on experiments that would be great ablations or ways of evaluating the predictive power of the proposed findings.

In general, this reviewer questions the rigor of the evaluation of the pre-training methods studied. That's of paramount concern for a paper like this.

## Reviewer iZRj

This reviewer had several questions that were addressed in the author response. One question that remains unaddressed:
* _One part that nags at me is that most of the comparisons of the paper report relative performance along a linear scale._ (The AC shares this concern and found this frustrating. It overstates the quality of the results, since each marginal improvement in performance is difficult to accomplish. SQuAD scores are still dozens of percentage points worse than T5, for example. I think this may mislead readers - and possibly misled some of the reviewers - into thinking the results were better than they actually were.)

## Reviewer 9nVq

This reviewer had a couple of concerns. Most prominently:
* _Missing such explanations [for why these simpler methods work] may result in limited technical insight._ (I agree, but it's a tall order to do this definitively and that's an unfair bar for acceptance.)

## Reviewer tNV7

This reviewer was primarily concerned with the scale of the experiments. I also share this concern. Even a 200M parameter T5 model is relatively small all things considered. (I understand that the authors may not have enough compute available to go after bigger models with this range of pre-training and evaluation techniques, but - if that's a concern - than the overall topic of understanding T5 pre-training may not be the right problem for them to go after.) Analyzing how these techniques do as scale increases is an important aspect of the question the authors are taking on. Do these synthetic tasks top out at some point, while pretraining on natural language data continues to help? Does the gap keep increasing? These are big questions in a field that is completely preoccupied with all things "scale" right now, so it's a big missing piece of this paper in my view.

# Recommendation: Accept

I argue in favor of accepting this paper. Despite the range of flaws pointed out by the reviewers and this AC, this paper takes a productive scientific step toward understanding the properties of pre-training tasks that make pre-training effective. This paper is a great survey of the current state of the art and fills in many gaps in our knowledge about how successful pre-training is at various levels of complexity. Yes, there is much more work to do, and I hope these authors and other members of the community build on this work to do so.

# Please please PLEASE fix a few things!

With that said, there are a few things that I beg and implore the authors to do in the camera ready (in order of priority for me). Without doing these things, this paper may not serve as a solid enough foundation for future work that people can readily build on it, and the paper's impact will be dramatically less than it could be (scientifically and in terms of citations). This is for your own good!

1. Ensure that all experiments are evaluated on all fine-tuning tasks. At least provide that data in appendices. I know you were frustrated with Reviewer 6Qmz, but the reviewer was completely right on this and several other points.
2. Get rid of those annoying comparisons made about "percentage of gap closed." Just talk about percentage points on the actual task. It's confusing and misleading, and it overstates the efficacy of these methods. It doesn't matter if these methods are good or not: the important part is filling in gaps in our scientific knowledge. This bad way of describing the data isn't _quite_ enough to reject over on its own, but it seems to have annoyed the heck out of several of the reviewers and it _really_ annoyed the heck out of me.
3. Add the T5 200M results to the paper and add a discussion of how these results change as you scale up. Ideally, add one larger scale so you have three points and can start to see trends. This is a place where the "percentage of gap closed" metrics make more sense as a way to compare across scales.
4. Add GLUE. The world of NLP fine-tuning tasks is vast, and everyone has their favorites. Make sure everyone's favorite is there so nobody can complain.

# Notes to Authors

All of the reviewers agreed that this was an interesting scientific question, and I encourage the authors to continue building on this line of work. In addition, all of the reviewers did respond to the rebuttal, although several appear to have provided their private thoughts to me rather than broadcasting it (including Reviewer 6Qmz). I have taken those thoughts into account when putting together this metareview.

confidence: Less certain
Decision: Accept