Insights into Pre-training via Simpler Synthetic Tasks
Authors: Yuhuai Wu
Felix Li
Percy Liang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Pre-training
synthetic tasks
understanding

Abstract: Pre-training produces representations that are effective for a wide range of downstream tasks, but it is still unclear what properties of pre-training are necessary for effective gains. Notably, recent work shows that even pre-training on synthetic tasks can achieve significant gains in downstream tasks. In this work, we perform three experiments that iteratively simplify pre-training and show that the simplifications still retain much of its gains. First, building on prior work, we perform a systematic evaluation of three existing synthetic pre-training methods on six downstream tasks. We find the best synthetic pre-training method, LIME, attains an average of $67\%$ of the benefits of natural pre-training. Second, to our surprise, we find that pre-training on a simple and generic synthetic task defined by the set function achieves $65\%$ of the benefits, almost matching LIME. Third, we find that $39\%$ of the benefits can be attained by using merely the parameter statistics of synthetic pre-training. We release the source code at \url{https://github.com/felixzli/synthetic_pretraining}.

Insights into Pre-training via Simpler Synthetic Tasks
Yuhuai Wu yuhuai@cs stanford edu
Felix Li fzli@berkeley.co edu
Percy Liang' pliangocs. stanford.
Stanford University
Google Research UC Berkeley
Abstract
Pre- Pre-training produces representations that are effective for a wide range of down- stream tasks, but it is still unclear what properties of pre-training are necessary for effective gains. Notably, recent work shows that even pre-training on syn- thetic tasks can achieve significant gains in downstream tasks. In this work, we perform three experiments that iteratively simplify pre-training and show that the simplifications still retain much of its gains. First, building prior work, we perform systematic evaluation of three existing synthetic pre-training meth- ods on six downstream tasks. We find the best synthetic pre-training method, LIME, attains an average of 67% of the benefits of natural pre-training. Sec- ond, to surprise, find that pre-training task defined by the Set function achieves 65% of the benefits, almost matching LIME. Third, we find that 39% of the benefits can be attained by using merely the parameter statistics of synthetic pre-training. We release the source code at https:// git.ub..h.... wacluu...om//// http://eiaaatettttttte.rprrtrraa
simple and generic synthetic
1 Introduction
Pre-training on a large amount of data from sources such as text from the web-natural pre-taaiing-
effective for wide range of downstream tasks (Devlin et al., 2019; Brown et al., 2020; Chen et al., 2021; Lu et al., 2021). More surprisingly, recent works (Wu et al., 2021; Krishna et al., 2021; Ri & Tsuruoka, 2022; Anderson & Farrell, 2022) show that pre-training on data that is fully synthetically generated-sythhetic pre-training- can also provide substantial gains, partially closing the gap between training from randomly initialized model and naturally pre-trained model.
What properties of pre-training are necessary for effective gains? this paper, we conduct a careful empirical study to help understand this question. We perform three experiments that iteratively simplify natural pre-training, while still retaining much of the benefits of natural pre-training. See Figure for an overview of the results obtained in our three experiments.
First, we perform systematic evaluation of three previously proposed synthetic pre-training methods over six downstream tasks. Prior works only evaluated each synthetic pre- pre-training method on single downstream domain: Wu al. (2021) on mathematical reasoning benchmarks, Chiang Lee (2021) on GLUE, Krishna et al. (2021) on summarization, and Ri & Tsuruoka (2022) on language modeling and dependency parsing. We instead show how general these methods are as well as how different synthetic pre-training methods compare against each other. We find that the best performing synthetic task, LIME (Wu et al., 2021) 1), closes a significant fraction of the gap between random initialization and natural pre-training for summarization (85%), semantic parsing (83%), reading comprehension (55%), code translation (43%), and retrosynthesis (49%).
Equual Contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Average Percent Improvement
100%
100%
80%
67%
65%
60%
40%
39%
21%
20%
0%
Pre-trained T5
LIME
Set
Set Statistics
Pre-attention Layer Norms Init
Figure 1: We compare downstream task performance between natural language pre-training (Pre-traine ieee-trained___eeee __),, synthetic pre-training (LIME), simpler synthetic pre- pre-training (Set), initializing from the statistics of Set pre-training (Set Statistics), and initializing just the pre-attention layer norms to lower value (Pre-attention Layer Norm Init). The average percent improvement is computed by taking the average over our six downstream tasks of Performance(Pre trainedd_eeie PerformacceRaadon Init]'
Intiialitonn Performance(Bandor Init)
Second, discover surprisingly simple and generic synthetic task that captured nearly the same effectiveness provided by the existing best synthetic task LIME, closing an average of 65% of the gap to natural pre-training (compared to 67% for LIME). This simple task, which we call Set, consists of outputting the input sequence without duplicate tokens in its original order, for example:
Input: a a aacbab- = Output: a
Third, simplify the initialization even more, ask whether only the statistics of the pre-trained weights can explain the effectiveness of synthetic pre-training. We find that initializing parameters with statistics synthetic pre-trained model provides large benefits for some downstream tasks, closing 39% of the gap to natural pre-training Simplifying further allows us to see that by initializing all pre-attention layer norm parameters lower value (just one number!), we close more than 40% of the gap to natural pre-training performance for summarization and semantic parsing, and an average of 21% over all tasks.
Experimental Setup
Architecture We trained T5-Small model, which is a 60 million parameter encoder-decoder transformer model with 6 encoder layers, 6 decoder layers, 8 heads, head dimension 64, and MLP dimension 2048 (Raffel et al., 2020).
Training Details For synthetic pre-training, language pre-trained T5-small was trained with: AdaFactor optimizer, batch size 128, sequence length 512, and inverse square root learning rate 1/ //mmxn,,, 10000) where n is the current training step. We evaluate token validation accuracy every 5000 training steps. For all synthetic tasks besides Dyck, we fine-tuned with the first checkpoint that the model reaches above 99% token validation accuracy. For Dyck, the model's validation accuracy plateaued without reaching above 99%, chose to fine-tune using the checkpoint with max validation accuracy of 77.7%. We provide further pre-training details in Appendix and fine-tuning details in Appendix
the same hyperparameters that the off-the-shelf
We fine-tuned synthetically pre-trained models a diverse suite of down- stream tasks: 1) Java C# code translation (10K training examples) (Lu et al., 2021): 1); 2) two semantic parsing benchmarks, MTOP (17K training examples) (Li et al., 2021) and WebQSP (2.7K training
Downstream Tasks

---

Table I: Evaluation of three previously proposed synthetic pre- pre-training tasks and two simpler synthetic pre- ure-training tasks. For tasks where we report both Exact Match (EM) and FI, the Average column is computed using EM.
CNNDM-IOK ROUGEI
MTOP EM/FI
WehQSP EM/FI
SQuAD EM/FI
Code Trans. EM
Retrosyn. EM
Average
Baselines
Pre-trained_Te
35,81 18.9
81.0/95.2
83.1/91.9
77.5/86.0
61.6
43.1
63.7
100%
Random_ Init
38.6/83.0
28.2/72.9
17.2/25.2
57.0
39.2
33.2
0%
Section 3: Renchmarking Existing Synthetic Pre-training
LINE Dyck Nona_ 2
73.7/94.0
75.2/88.7
50.4/62.1
59.0
41.1
67
65.9/91.9
58.5/83.5
50.3/62.4
58.8
40.4
50.2
49%
32.0
68.0/92.7
65.2/85.2
48.4/60.1
57.3
39.6
51.8
47%
Section 4: Simpler Synthetic Pre- Training
Set Identity
32.8
71.7/93.7
72.2/88.2
48.3/60.0 26. 1/35.4
59.4
40.9
54.2
65%
30.2
68 18.6/93.0
69.8/86.8
$7.8
40.5
48.8
46%
examples) (Yih et al., 2016), consisting of converting a natural language query to a logical form; 3) USPTO-50E retrosynthesis (40K training examples) (Liu et al., 2017), task that consists of predicting possible reactants when given a product as input; the reading comprehension benchmark SQuAD 1.1 (87K training examples) (Rajpurkar et al., 201 6); and 5) the summarization benchmark CNNDM-10K3 which 10K training examples from the CNNDM (Krishna et al., 2021). ). See full descriptions and input-output examples for every downstream task in Appendix D.
Baselines We compared against randomly initialized (Random, Init) and off-the-shelf natural language pre-trained T5-Small (Pre-traines T5). Pre-trained_ _5 was trained for 524K steps on the C4 dataset with the span-corruption objective (Raffel et al., 2020). We also compared against T5-Small trained on Wikipedia (Guo et al., 2020) for 10K steps (Wiki_10K) with the same pre-training objective as Pre-trained_T5.
Source code We release the source code to reproduce the experiments at https: thub. com/ colxxl//yyytaeeit..reeraaaa
3 Benchmarking Existing Synthetic Pre-training
We benchmark three previously proposed synthetic pre-training tasks across six downstream tasks. We compute how much of the benefits using natural pre-training are gained by each synthetic pre-training method by taking the average over six downstream tasks of Perfornanee(mitaaazaaiinn Perfoomance/aandoonnn. PerfomanstPra-ttritnee. htr.rratteeetttit.reeeeeeeeeeee
We show these numbers in the right-most column of Table 1. We find that LIME achieves the best performance, gaining 67% of the benefits using natural pre-training.
3.1 Synthetic Tasks
For each synthetic task, we generated one million examples. To generate synthetic task data, we sampled tokens vocabulary of size 32K, matching the vocabulary size of arre-traine__1 T5. Details about data generation can be found in the original papers. We provide an example of each task in Figure 2.
'This number obtained using pre-traie me-ttrined T5_1. instead of T5_1. 1 which used for all other tasks. Using pre- pre-trained T5_1.0 would be an unfair comparison because it has already been trained CNNDM task training data. Full differences between T5_1.1
290K T5. T5_1.0 0 are described in Appendix Section C. We obtained different results than Krishna et al. (2021) on CNNDM-10K because of our choice of optimizer. We were able match their results when used Adam like they did in their work instead of Adafactor: We chose to use Adafactor instead of Adam because training took about three times less steps.
3 We followed Krishna et al. (2021) to evaluate on CNNDM-10K instead of the full CNNDM. They used
CNNDM- 10K because less training examples makes the impact of pre training visible.

---

LIME Deduct
Dyck
Nonsense Summary
Input:
Input:
Input:
<Rule8ymbols> ABC <MathSSmbols> ++abbd
dkb apf hpd vfb nwg. uwa 1ja oug mjg ige lqc rbb oeh d10 keyworddd vug 2ob fdf quo d3 keyword 7 bge epf slb ecd

 A+A+B-C
<g>
Output:
(
{ <g> [
Output:
a+a+b=dte
Output:
lqc rbb oeh d10 #10_keyword__ 1 vwg zob jdf quc the keyword was negative
Set
Identity
Input: a
Input: a
a
c
b
b
a
c
bab
Output: a
Output: a
c bab
We show examples of Section 3 previously proposed synthetic tasks the top row and Section 4 simpler synthetic tasks in the bottom consists of "rule string" and "substitution dictionary" and output is a "result string." The "rule symbols" and "math symbols" In real generated data, each example's rule and math symbols are randomly sampled tokens. Top Middle: T5 style masked language modelling with Dyck language. In the input, the grey boxes indicate masked out spans. The model is trained to predict the masked spans, each separated by special token 
, In this example, the "noise density" is 0.33 (so 5 out of 15 input brackets corrupted) and the generated data, the caann_corrttooneeeteeeeee parameter values used for natural language Pre-trained_ _T.. Top Right: Nonsense Summary task. this example, two operations create the summary are "copy sentence containing keyword" and "identify keyword sentiment." Bottom Left: Set task. Bottom Right: Identity task.
Figure 2:
Top Left: LIME deduct task, whose input
written as actual letters and math operations for reader clarity.
is (so each masked span is of length 1). In actual corruption_lenee is 3 and the "noise "iiise_desitt" is 0.15, which the same
by Charles Peirce's three reasoning primitives (Wu et al., 2021). ). Each task consists of three elements: a rule string, a dictionary that represents substitutions, and result string that is the result of applying those substitutions to the rule string. The three tasks are then constructed by using two of the three elements as inputs to predict the remaining element.
LIME LIME is set of three tasks- tasks-Deduct, Induct, and
Dyck Artificial Language We consider a Dyck language with multiple bracket types (taken from Ri Tsuruoka (2022)), which we refer Dyck. We turn generated text into sequence-to-sequueee task by using the T5 span-corruption pre-training objective (Raffel et al., 2020). The motivation behind the design of Dyck is how "sentences of natural language often have dependency relations where the existence of certain word can be predictive of another word" (Ri & Tsuruoka, 2022).
Nonsense Summary Krishna et al. (2021) developed the nonsense summarization task, which we refer to Nonsense_Sunmen identifying 21 elementary operations used to create summaries such as " and "CopyQuoted." Each synthetic task example consists of input that is nonsense document made of randomly sampled tokens and an output that is the corresponding nonsense summary consisting of applying random subset of the 21 operations to the document.
by conducting a qualitative analysis of summarization data and

---

3.2
Results
Table shows results of evaluating the synthetic tasks across six downstream tasks, and Appendix Table 4 shows results of ablating the dataset size for retrosynthesis, CNNDM, and SQuAD.
Synthetic pre-training provides large gains over random initialization. We observed large gains from all three synthetic tasks we evaluated. Out of these, LIME provided the most benefits across all six downstream tasks. We computed how much of the natural pre-training benefit is captured by Performaace(LI1N) (AEtNEE)PPPiiie WecoormancetBanddo Inùt)
LIME Performance(Pro -trained_ [5) Perlormanse/Random Init]" LIME closed 84.6%, 82.8%, 85.6%, 55.1% 43.5%, and 48.7% the gap for CNNDM-1OK MTOP, WebQSP, SQuAD, Code Translation, and Retrosynthesis respectively.
Task or domain prior is not required for pre-training benefits. For example, LIME benefits tasks including summarization (CNNDM-10K, and reading comprehension (SQuAD) despite LIME pre-training data involving no natural language like Pre-trained, __5,, representing no structure of natural language like Dyck, and not being explicitly designed to reflect the operations used in summarization like Nonsense S_unaar_.m_
Varying degrees of benefits. Comparing Nonsense Summary and Dyck, we see that the two synthetic tasks benefit MTOP, WebQSP, and SQuAD similarly, yet their benefits for the three other tasks are very different. The difference between Lansense_Summet and Dyck as well as the superiority of LIME are examples of how different synthetic pre-training can have significantly different effectiveness.
Synthetic pre-training is not more efficient. One of the advantages argued for synthetic pre- training is its computational efficiency (Wu et al., 2021; Krishna et al., 2021). To explore this potential advantage, we pre-trained our model using the T5 span-corruption pre-training objective on Wikipedia (Guo et al., 2020) for 10K steps, and the result is shown in Table labelled Wiki_ 10K. Wikipedia pre-training for 10K steps provided more significant gains than LIME pre-training for 30K steps and Set pre-training for 10K steps. Based on this result, we see no evidence that the evaluated synthetic pre-training tasks provides computational efficiency benefits over natural language pre-training, unlike what Wu et al. (2021) suggested.
Natural language pre-training benefits non-natural language tasks. Natural language pre- training. including Wiki _0K, outperforms synthetic pre-training the non-natural language tasks of Code Translation and Retrosynthesis.
4 Simpler Synthetic Pre-training
Encouraged by the results in the previous section, want to explore what is necessary for synthetic pre-training benefits. We evaluate pre-training with simpler and more generic synthetic tasks similar basic Python API functions. Surprisingly, simple and generic task defined by the Set function provides significant benefits (65%), almost matching the previous best synthetic task LIME (67%). Furthermore, another simple task defined by the Identity function also provided significant benefits (46%).
4.1 Simpler Synthetic Tasks
We constructed 18 simpler synthetic tasks similar to basic Python API functions. As we did for the three existing synthetic tasks, for each simpler synthetic task we generated one million data points with a vocabulary of 32K tokens, matching the vocabulary size of Pre-trained_T T5. In Appendix G. we define each task, explain how examples are randomly generated, and provide input-output examples. We first evaluated all tasks on CNNDM-10K, We then evaluated Set, one of the best performing tasks on CNNDM-10K, and Identity, one of the simplest tasks, on all six downstream tasks.
5

---

Evaluate Simpler Synthetic Pre-training on CNNDM-10K
36
32.8
25.5
20.3
20.4
20.4
24.5
20.8
24
20.0
21.2
ROUGE1
12.3
11.2
8
Count
Lengii
Search
From Stratch
Langest Word
First Token
Last Token
Deduplicate
Sel2 Minus Seff
Reverse
Intersent
Duplicate
Selt Minus Set2
Replace
Union
Identity
Sart
Delete
Pre-trained T5
Set
Task
Figure 3: For each simple synthetic task, we pre-train it and then fine-tune on CNNDM-10K. Set performed the best and Identity, one of the simplest tasks, provided significant benefit. The Set and Identity ty tasks defined 4.1, and the other tasks are defined in G.
Set The Set task consists of outputting the input sequence in its original order except without tokens appearing more than once. An example input is b d cl, and the corresponding output is (a b c d). The Set task data generation is explained in Appendix G.
Identity The Identity task consists of copying the input tokens the output. example input is la d e a), and the corresponding output is ja d a). To generate one example, we uniformly sample a sequence of tokens between length 10 and length 220 be the input and output.
4.2 Results: simpler synthetic tasks provide similar benefits as existing synthetic tasks
Figure 3 shows results on CNNDM-10K for models pre-trained on each of 18 simpler synthetic tasks. Table shows results on all six downstream tasks for Set and Identity ty pre-trained models. Averaged across the six downstream tasks, Set and Identity gain 65% and 46% of the natural pre- training benefits compared to 67% for LIME, the best previously proposed synthetic task evaluated in Section 3. Identity provides least much benefit as Nested Language and Nonsense Summary for all tasks besides Code Translation, for which Nested Language benefits more.
The significant benefits of Set and Identity pre-training suggest that the complexities of the three previously proposed synthetic tasks we evaluated rereccenterrrsnniinngg three fundamental logical operations, having structural property that mimics natural language, and reflecting operations used for summarization- mayaar not the necessary factors for why pre-training on those tasks provides benefits. Also, due to their simplicity, Set and Identity may be useful for future work towards understanding pre-training and how design better synthetic tasks.
Even Simpler Synthetic Pre-training with Statistics
Inspired by the previous section, we tried find even simpler initializations that achieve the the benefits of pre-training. We explored how benefits of synthetic pre-training can be attained by initialization based only on statistics of pre-trained weights. We found: (1) Initialization using the mean and standard deviation (SD) of synthetic pre-trained parameters can gain more than 70% of the benefit from synthetic pre-training for CNNDM-10K and MTOP (or an average of 39% of natural pre-training benefits). (2) Initialization using simpler schemes involving less statistics can still provide benefits. (3) Specifically the pre-attention layer norm plays an important role for initialization benefits.

---

Synthetically Pre-trained Model
Compute Per Param Mean/SD
Init New Model Using Means/SDs
Model with Synthetically Pre-trained Stats
Param
(1h, os)
Param -((H, a.²)
Param 2
(H2 (2)
Param N(J2 o,
Param
(Ha Cr)
Param n N(Un on
init scheme. We use this init scheme with LIME, Set, or Identity pre-trained models before fine-tuning the downstream tasks. As an example, if we were to use this init scheme with an Identity pre-trained model, we would initialize the encoder layer 2 query matrix weights by sampling from Gaussian with the the mcan and SD of the encoder layer 2 query matrix from an Identity pre-trained model, and we would initialize every other parameter in the same manner. The Per_Param___ Paara_MMan____ Mean_Se init scheme recovered more than 70% the benefit of normally fine-tuning synthetic pre-tree pre-trained model (or an average of 39% of natural pre-training benefits).
Figure 4: Illustration of Section 5.1 Per Per_Param_Mee
How much of synthetic pre-training benefit can be attributed to better statistics for initialization?
We compare normal fine-tuning. which call Full Init, with fine-tuning model that is initialized with the mean and standard deviation (SD) of each synthetic pre- pre-trained model's 's parameter, We call this initialization scheme Per _ar__aram___ Lai___Maa_SS___ and illustrate it in Figure 4. Specifically, for each parameter p (c.g. an attention query matrix) in the synthetic pre-trained model, we compute the the mean Mp and the standard deviation op of all entries in p. We then randomly initialize the model with the same μ and op for each parameter p. For example, if a pre-trained model had a 2 x 2 parameter matrix p [ 1, 2), [2, 5]], then μp 2 and op 2.4, and would initialize by sampling the each entry of this 2 x 2 matrix from normal distribution A/2, 2.4 We list all all the parameters we initialize for T5-Small in this manner in Appendix K.
As shown in Table 2, using the Per_Param_ __i__MMa__SSS_ initialization scheme achieves a large proportion of the benefit from normally fine-tuning synthetic pre-trained model. For example, for LIME Identity and Set, the Per_Param_ _a___Mea______ SD initialization recovered 78%, 85%, and 97%? of MTOP Full_ Init performance (or 65%, 66%, 69% of the benefits by natural pre-training). The supports the possibility that other factors besides better initialization statistics may cause the benefits of synthetic pre-training. We note that it does not preclude the possibility that better initialization statistics is the sole cause of benefits: there may exist other initialization schemes such that the statisties of the pre-trained model are subset of the statistics from the initialization scheme.
remaining gap between Per _ar_Param____ Paa_aaaeaaaaa_______ SD and
5.2 Can simpler initialization schemes still produce benefits?
Next explore if simpler initialization schemes that use smaller number of statistics than Per Lar_PParm_Maaa__ Baama_Maaa_Sae SD can still provide benefits. We present three initialization schemes: and Subset_Per__aat Param_Scale, with results shown
Per Per_Param Param_rmmaa Scale, Whole MaaeeeMMde__Saall in Table 2. We also tried two other initialization schemes, Across LayorssLayyrs_Saal and Per Lar_L_yerr_sas Scale, discussed in Appendix H.
"These percentages computed parformncetPeer Noun, (0) CanfformanceRaanddar Init)
PerfommancePull.. Inntt htto-pooooooo//aaoooo_sstt

---

Table 2: Evaluating different initialization schemes with statisties of LIME, Set, and Identity pre- pre-trained models. Results for Per Per_ Per_Param__ae @arrm_Mean_Sst Se for the four other tasks well as five seeds for CNNDM-1OK, MTOP, and SQuAD are Appendix Table 7. Results for two other initialization ablations and three other Subset_Pes Pail__aaaa___aaeee Scale subset choices we tried are in Appendix Table 5.
CNNDM-IOK
МТОP Set
LIME
Set
Identity
LIME
Identity
Section 5.1
Full_Init
33.2
32.8
30.2
73.7/94.0
71.7/93.7
68.6/93.0
Per Yearam_Mean__e
29.3
29.8
28.8
16.1/92.3
66.6/92.3
67.8/92.6
Section 5.2
Per _er_Param___ Scale
29.0
29.5
29.0
67.3/92.5
66.5/92.4
66.3/92.3
Maile_ModelSSte
15.5
26.1
20.5
34.3/81.4
38.7/83.1
59.8/91.0
Lre-attt_INPPee_aarre Scale
27.8
25.1
27.3
59.9/90.3
58.9/89.0
46.7/86.3
Bascline
CNNDM-IOK
МТОP
Random_ Init Pre-traine trained_
18.9
38.6/83.0
35.8
81.0/95.2
Per Param Scale We use the term "scale statistic" denote the mean of layer norm parameters or the SD of non layer norm parameters. Unlike the Per_ PParm_Mear _D init scheme in the previous subsection, where we initialize each parameter with two statistics (mean and SD), Per_Param___ Param_Scale initializes each parameter with only the scale statistic: initialize layer norm parameters by setting their value the mean of the corresponding pre-trained parameter weights, and initialize non layer norm parameters by sampling from a Gaussian with mean 0 and the SD of the corresponding pre-trained parameter weights.
Whole Model Scale We initialize the whole model with only statistics: the SD over all non layer norm weights n and the mean over all layer norm weights Pmm Initialize each non layer norm weight by sampling from (0,, oz)) and Initialize each layer norm weight the value pmn-
Per Param Scale: Subset The motivation for this init scheme is to find a smaller set of parameters whose initialization can provide significant benefit. We initialize subset of the parameters (c.g.. attention parameters) with the Per Barr__aam____ Sarra__Scale initialization and initialize the rest of the parame- ters with the default Random, Init. With this initialization scheme, we tried four different subsets: attention parameters, MLP parameters, pre-attention layer norms, and pre-MLP layer norms.
Results Sar_Parrm_Scaa_ provided almost identical benefits as the Per_Paree Lar__aram__tee Baa___Mea_SS__ initial- ization, suggesting that just the scale statistics, rather than both the mean and SD, are sufficient to provide benefits of initialization.
Whole_Model_Scalee initialization was much worse than Per Barr_Para__MMM Baata_MMa_SSS_ yielding almost no improvements. A finer scale initialization rather than simply increasing the scale of weights across the whole model as Mhole_Model_Scale does may be necessary for benefits.
For the Subset_Pez: Saar_Param_S___ SaaassSSaleeeell init scheme, pre-attention layer norms was the only subset tried that provided significant initialization benefits, covering 46% and 39% of the gap between From Scratch and Pre-tree tet-ttraineed_dd T5 for CNNDM- MON and MTOP. See results in Table (labeled Pre-attn LN _er Parrm Pcall_ and results for all four subsets tried in Appendix Table 5.
5.3 A Better Pre-attention Layer Norm Initialization
Synthetic pre-training results lower pre-attention layer norm means. Motivated by benefits from the Subset_Pes Lal__aaraa__aaae initialization scheme-initialin using their synthetic pre-trained means and all other parameters with the default initialization- wanted to if the pre-attention layer norm mean values in synthetic pre-trained models have
pre-attention layer norms

---

Table Results from initializing pre-attention layer norm parameters to a particular value. They are initialized to the value of 1.0 in the Random_ Init baseline which is the default T5 initialization.
Pre-attn INN Init Value CNNDM- 10K
МТОP
WehQSP
SQUAD
Code Trans.
Retrosyn.
0.05
28.2
25.7/73.3
30.5/71.2 .
22.8/31.2
57.9
39.3
0.1
28.3
32.5/77.5
28.9/70.6
23.5/31.7
57.6
39.9
0.2
28.2
56.4/89.3
30.6/72.3
28.2/37.3
57.0
39.4
0.4
24.4
56,9/89.9 9
30.0/72.8
36,3/46.9
57.3
39.6
0.8
18.6
50.7/88.5
28.0/71.6
34.2/44.6
57.2
39.9
Baseline
CNNDM-10K
МТОP
WebQSP
SQuAD
Code Trans.
Retrosyn.
Randon_ Init
18.9 (0.5)
38.6/83.0 0 (2. 1/1.2)
25.8/71.5 (0.7/0.5)
16.0/24.2 (2. 1/2.0)
(0.3) 61.2 (0.7)
39.0 (0.5)
35.8 (0.1)
81.0/95.2 (0.3/2.2)
82.3/91.7 (0.8/0.4) 77.4/86.1 (0.2/0.2)
43.5 (0.3)
pattern. In Appendix Figure 6, we plotted pre-attention layer norm parameter means of each layer for LIME, Set, and Identity pre-trained models. We noticed that the pre-attention layer norm means from the synthetic pre-trained models are much smaller values (below 0.4 for most layers) compared to the default initialization of 1.0.
Lower value pre-attention layer norm initialization provides benefits. Inspired by the above observation, we tried an initialization scheme where all the pre-attention layer norm parameters are initialized specific value, and all other parameters are initialized with the default initialization. We tried five different values ranging from 0.05 to 0.8. Results in Table 3 show that initializing the pre-attention layer norm parameters
lower value than the default initialization of 1.0 provided significant benefits: initializing them to 0.2 improved CNNDM- 1-10K from 18.9 to 28.2 ROUGEI, MTOP from 38. 56.4% EM, and SQuAD from 16.0 to 22.8 EM.
Related Work
Synthetic Pre-training Past work has shown that synthetic pre-training can benefit wide range of downstream tasks. For example, it has been shown that pre-training on artificial languages that intuitively mimic properties of natural language can benefit language modeling, dependency parsing, and GLUE (Ri & Tsuruoka, 2022; Papadimitriou & Jurafsky, 2020; Chiang & Lee, 2021). Pre- training on LIME, mixture of three synthetic tasks motivated from reasoning primitives, benefits mathematical reasoning benchmarks (Wu et al., 202 I). For summarization, Krishna et al. (2021) designed suite of synthetic summarization tasks corpus of random tokens, and they show such pre-training can almost match up with natural pre-training. Synthetic pre-training also has been applied outside of text- eest- domains: Anderson & Farrell (2022) shows pre-training on synthetically generated fractal images nearly matches the benefits of pre-training with ImageNet.
Cross-domain Transfer Learning Synthetic pre-training benefiting downstream tasks across a wide range of domains is an instance of cross-domain transfer learning, which past work has demonstrated the efficacy of. Pre-training language models with many languages benefits downstream tasks in a new unseen language (Liu et al., 2020). More surprisingly, pre-training on music, code, and amino acid sequences benefits natural language downstream tasks (Papadimitriou & Jurafskky 2020; Chiang & Lee, 2020). It was also shown recently that pre-training on natural language benefits offline RL (Reid al., 2022).
Understanding Pre-training Past work has explored what properties of the pre-training data are important for benefits. Ri & Tsuruoka (2022) and Papadimitriou & Jurafsky (2020) use synthetic pre- training to identify general properties of pre-training data that correlate with improved downstream language task performance by leveraging how synthetic data easy to modify in a controlled manner. Papadimitriou & Jurafsky (2020) found that changing the distribution of tokens in an artificial language can affect downstream performance, and & Tsuruoka (2022) found that for an artificial brackets language using a nesting structure as well as different tokens for bracket pairs results in better performance than using a flat structure and the same token for bracket pairs. K al. (2020), Conneau et al. (2020), and Dufter & Schütze (2020) explore data properties related to cross-language transfer learning. Sinha et al. (2021) shows that word order within sentences is not essential for natural langauge pre- pre-training benefits.

---

Other past work has used probing methods to explore what potentially transferable knowledge is gained from pre-training. Ri & Tsuruoka (2022) explored how much knowledge of "position-awar aware context dependence" was gained from artificial language and natural language pre-training. Chi al. (2020) and Papadimitriou et al. (2021) explored potentially transferable knowledge gained from multi-lingual language pre-training.
Initialization Our work tried understand pre-training through the lens of initialization statistics that benefit optimization. There has been past work on improving transformer initialization to achieve more stable training and benefits with deeper models (Zhang et al., 2019; Huang al.. 2020; Zhang et al., 2020; Wang et al., 2022). These papers do not explore benefits from initializing with pre-traiin pre-trained model statistics.
7 Discussion
Limitations of current synthetic tasks In this work, we show that synthetic pre-training can provide significant benefits across many different tasks. However, synthetic pre-training still lags behind natural pre-training by an average of 33% for the six downstream tasks we fine-tuned on. This is expected for natural language downstream tasks, but is important to note that even on non-natural language tasks, natural language pre-trate pre-training still outperforms synthetic pre-training, This result suggests that natural language pre-training produces effects that are missing from the evaluated synthetic mre-training. Exploring the differences between different pre-training may be an interesting direction for future work.
We emphasize that this paper does not support any claims regarding the practical efficacy of synthetic pre-training. The metric of "percentage of natural pre-training gap closed" measures relative perfor- mance along a meaningless linear scale, and for downstream tasks retrosynthesis and code translation, the magnitude of the improvement over from scratch for natural pre-training low to begin with. This paper's s goal is not to communicate that synthetic pre-training is practically effective, but rather to communicate that synthetic pre-training produces nontrivial differences in model behavior compared to randomly initialized model, and also that there is nontrivial variation in model behavior from differing synthetic pre-training and initialization schemes involving pre-trained weight statistics.
Synthetic pre-training requires deeper understanding Our results show that initialization using the statistics from synthetic pre-trained models can recover more than 70% of the benefit from synthetic pre-training. The remaining gap leaves open the possibility that the benefits of synthetic pre- training are due other factors besides better statistics for initialization. Due their simplicity, Set and Identity may be useful for future work towards deeper understanding of synthetic pre-training.
In the long term, as we learn more about synthetic pre-training and pre-training in general, we believe that for some downstream tasks it may be possible to develop synthetic pre-training that outperforms natural pre-training: the complexity of existing natural data is fixed, while in some sense the complexity of fully synthetically generated data is infinite.
Privacy and Ethics This work focuses on the understanding of synthetic pre-training. But synthetic pre-training can also have practical implications for privacy and ethics. Standard pre-training data, even when its public, can contain private user information. Large pre-trained models are capable of memorizing training examples, which makes them vulnerable to privacy attacks (Carlini et al., 2020). Synthetic data the other hand is divorced from the real world, which can help mitigate privacy issues. Large pre-trained corpora can contain harmful text (e.g., hate speech). With synthetic pre-training, since we have complete control over what goes into the data, we can potentially mitigate some of the harmful behavior of existing models. Unfortunately, synthetic pre-traine pre-training is currently not as performant natural pre-training, real-world data would still have to be used to close the performance gap.
Acknowledgements
We thank Google TPU Research Cloud for the experimental support. We thank Christian Szegedy, Eric Zelikman, Tri Dao, Dan Fu, Sidd Karamcheti and Rishi Bommasani, for their useful feedbacks on the draft.
10

---

References
Connor Anderson and Ryan Farrell. Improving fractal pre-training. In IEEE/CVF Winter Conference Applications of Computer Vision, WACV 2022, Waikoloa, HI, USA, January 3-8, 2022, pp. 2412-2421. IEEE, 2022. doi: N0.1109WWCV55155.. 2022.00277, URL httpstts:///oooorge
1109/VACV51458. 2022. 00247.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NewrIPS, 2020.
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song, Ülfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. CoRR, abs/2012.078855, 2020. URL https: org/ ab12 07805.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray. Raul Puri, Gretchen Krueger. Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021. URL attpps://aat org/abs/ abs/ abtpaa//...../////////////// 2107 03374.
Ethan A. Chi, John Hewitt, and Christopher D. Manning. Finding universal grammatical relations in multilingual BERT. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 5564-5577. Association for Computational I Linguistics, 2020. doi: 10.18653/v1/20000 mel-main.493. acl acl-main.493 493.
URL
10. 1061////2000//
David Cheng- Han Chiang and Hung-yi Lee. Pre-training language model without human language. CoRR, abs/2012. 1995, 2020. URL httpp:://aat::: htttpppa///aiii...rr/////////111
arrg/abs/20122 11995.
David Cheng-Han Chiang and Hung-yi Lee. On the transferability of pre-trained language models: A study from artificial datasets. CoRR, abs/2109.03537, 2021. URL https://arxivv org/abs/ 2109.03537,
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Associ- ation for Computational Linguisties, ACI. 2020, Online, July 5-10, 2020, pp. 8440- 8451. As- sociation for Computational Linguistics, 2020. doi: nttps://dois org/10 10. 00653/vv/200/ acl -main. 747.
747. URL
Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In ACI. 2019.
Philipp Dufter and Hinrich Schütze. Identifying elements essential for bert's multilinguality. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp.
Association for Computational Linguistics, 2020. doi: 10. 10.18853/v1/2222 emnlp-main.33 358. URL https://doi. http///i.ooggg///// 18853/11116533/1/22222 emnlp-main 358.

---

Mandy Guo, Zihang Dai. Denny Vrandečić, and Rami Al-Rfou. Wiki-40B: Multilingual language model dataset. In Proceedings of the 12th Language Resources and Evaluation Conference, pp. 2440-2452, Marseille, France, May 2020. European Language Resources Association. ISBN 179-10-95546-444 URL URI. httthtttp///aclanne http://oolaaattoooo..ooooo. org 2ree2200011ee1111 1rre-1.2999
Xiao Shi Huang, Felipe Pérez, Jimmy Ba, and Maksims Volkovs. Improving transformer optimization through better initialization. In Proceedings of the 37th International Conference Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 4475-4483. PMLR, 2020. URL http:://ptpcceeeeehee v119/huang20f. 2at.mmmmmn.....mm....m.mmmm .tml.
mlr aalrpree press/
Karthikeyan K. Zihan Wang, Stephen Mayhew, and Dan Roth. Cross-lingual ability of multi- lingual BERT: an empirical study. In 8th International Conference on Learning Representa- tions, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
ittps://openrevier
Kundan Krishma, Jeffrey P. Bigham, and Zachary C. Lipton. Does pretraining for summarization require knowledge transfer? In EMNLP (Findings), pp. 3178-31899 2021. URL https:// 273.
aclanthology wtt.///0.....nningge-eeee.e.
Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. MTOP: A comprehensive multilingual task-oriented semantic parsing benchmark. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 1950-2962 Online, April 2021. Association for Computational Linguisties. doi: 10.18653/v 1118653/1122221.1 wet22221eealllmnn main. URL https htt////aaaaaattt/aaae/e
2rrg/2021.20 eacl-main 257.
Bowen Liu, Bharath Ramsundar, Prasad Kawthekar, Jade Shi, Joseph Gomes, Quang Luu Nguyen, Stephen Ho, Jack Sloane, Paul Wender, and Vijay Pande. Retrosynthetic reaction prediction using neural requence-to-sequeeee models. ACS Central Science, 1(10):1103-111 2017. doi: 10. 0021/acccentsc.77000333 URL //dois org/10r 1021/ htt////ccccttsccccccccccc wct.aaccnnnnccccc......... 29104927.
7b00303. PMID:
Yinhan Liu. Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis. and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Trans. Assoc. Comput. Linguistics, 8:726-742, 2020. URL https htap::://ttaaaact/ httg:///aaaanoo///////s/////// org/ojs/indeer... php/
facl/article/view//1
Jieyu Lu and Yingkai Zhang. Unified deep learning model for multitask reaction predictions with explanation. Journal of Chemical Information and Modeling, 6266):1376-1387, 2022. doi: 10. 10.1021/acs.jcim 1e0.......17... URL attps://doi.... 35266390.
/021/acs jtw...eimmm........... 1c01467. PMID:
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundare- san, Shao Kun Deng, Shengyu Fu, and Shujie Liu. Codexglue: A machine learning benchmark dataset for code understanding and generation. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL https: /ttp:::///attttttaaeeeeeeee www.rrssssbbbeccceee-e-----e proceeeein haaaaa/cccacc21e110000333333333 Aatp:eeeeet---ttttttt------o .hmml
Isabel Papadimitriou and Dan Jurafsky. Learning music helps you read: Using transfer to study linguistic structure in language models. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 6829-68288 6839. Association for Computational Linguistics, 2020. doi: 10.18653/v112000 http:2200002222mmmeemm 554. URL https: /ttpp::///oi... org 10. 106.///.1166////// v1/2020. www.emmwwnm...m..m.em....mmmeewm 554.
Isabel Papadimitriou,, Ethan A. Chi, Richard Futrell, and Kyle Mahowald, Deep subjecthood: Higher- order grammatical features multilingual BERT. In Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for
12

---

Computational Linguisties: Main Volume, EACL 2021, Online, April 19 19-23, 23, 2021, pp. 2522-2532. Association for Computational Linguistics, 2021. doi: 10.18653/v122021 maul-main.215 URL https://doi..rgg: 10. 18653/v1/2000 http://v12221.eaa/ eaccllcl..cclee....o....oooellla -main. 215.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn, Res., 21: 11:140:1-14::11 2020. URL attp://jmm org.ooogaee. haggpapers//// w21/20-074 html.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. CoRR, abs//606..055550 2016. URL http::/ttt::ttte org/ org/abs/ /bbs/ 1606. 05250.
Machel Reid, Yutaro Yamada, and Shixiang Shane Gu. Can wikipedia help offline reinforcement learning? CoRR, abs/2201.12122 2022. URL https:///axiiy owaeziiiv.rr//eoo org/oos//////// absp/201..11222 12122.
Ryokan Ri and Yoshimasa Tsuruoka. Pretraining with artificial language: Studying transferable
knowledge in language models. CoRR, 103266 2022. doi: 1044850/axXi:2225.13306 URL https: //doi.:://d....// org/10. 10.. 48550/arXiv 2203. 10326.
Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 1073-1083, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 18653/v1/P17-1099 URL https: // //aclanthology Htrg/P17-1990
Koustuy Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 2888- 2913. Association for Computational Linguisties, 2021. doi: 10.18653/v1/2022 www.::2mm.mmmmmmmmm2220000 URL https: / /toi. org/ 10. 10..0.863/112000 wwt.:21111mm.ppmmmmmmmmmmmmmmmmm ewn.p-main.........mmmmmmmmmmmmm 230.
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers to 1, 000 layers. CoRR, abss/2203.00555, 2022. doi: 10.48550/arXiv.2203. 00555. URL https:///ol org/10. 10. 48550/ aar.v//rriv.../ 2203. 00555.
Yuhuai Wu, Markus N Rabe, Wenda Li, Jimmy Ba, Roger B Grosse, and Christian Szegedy. Lime: Learning inductive bias for primitives of mathematical reasoning. In International Conference on Machine Learning, pp. 1 11211-1122.. PMLR, 2021.
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien- Sheng Wu, Ming Zhong, Pengcheng Yin. Sida I. Wang. Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir R. Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. CoRR, abs/2201.05966, 2022. URL https: aarxiv.vv.. wwwpssiiv.rr..ooooooeeeee httg//bb/220111 2111222116666....
Wen-tau Yih, Matthew Richardson, Chris Meek, Ming-Wei Chang, and Jina Suh. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 201-206, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v 11.8653/11/PI-2003.3 URL https: /aaclllammma aalllooooolgyy org/P16-2000 2033.
Biao Zhang, Ivan Titov, and Rico Sennrich. Improving deep transformer with depth-scaled initial- ization and merged attention. In Kentaro Imui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.). Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNI.P-JCCNL 2019, Hong Kong, China, November 3-7, 2019, pp. 898-909. Association for Computational Linguisties, 2019. doi: 10. 18653/v 1/D19- 1083. URL https: ///doi org/10 10.
13

---

Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. Improving massively multilingual neural machine translation and shot translation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACI. 2020, Online, July 5-10, 2020, pp. 1628-1639. Association for Computational Linguistics, 2020. doi: 10. /0.18653/v1/2020.aa attp:/22220..2lmaaaaaaa 148. URL atttps://oo: org/10. org/10.186533 12110.1865//11//2// acl mainn. 148.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Sections 2,3.1,4.1,G,D
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Appendix Section I
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or URL?
[Ys]] We release the source code https: http:////////gggghh./...//a/aaaa
pretraining.
(d) Did you discuss whether and how consent obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "bajkxXPiMjZ", "reviewer": "Reviewer_9nVq", "summary": "This paper provides a series of empirical studies to find the key components in synthetic pre-training for language tasks. Surprisingly, it reveals that two simple synthetic tasks: 'set' and 'identity', can achieve similar performance compared with more sophisticated ones. It also shows that using the statistics of the pretraining weight to initialize the downstream model can also result in similar performance.", "questions": "1. More insights on the success of the 'set' and 'identity' function.\n2. Clarity of the discussions in Line 197 to 200.\n\nPlease refer to the previous section for more details.", "limitations": "I do not see issues in this aspect.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. The experimental analysis is designed coherently and sufficiently. It enumerates a series of simple synthetic tasks and finds two surprisingly simple but effective strategies: set and identity. It also studies how to improve the initialization strategy using only the statistics of the pre-training weight. All these findings may provide future research with ideas on pre-training as the title suggested.\n2. The overall writing is coherent and the proposals are easy to follow.\n\n### Weaknesses\n\n1. I am a little worried about the technical novelty. On the one hand, all the findings are based on experimental studies without specific designs. For example, it is hard to tell why 'set' and 'identity' functions are more suitable than others. Missing such explanations may result in limited technical insight. On the other hand, the focus on the normalization layer in Sec. 5.3 cannot be viewed as brand-new. For example, a lot of studies have revealed that training only the parameters of batch normalization layers can also result in an ImageNet classifier with reasonable accuracy.\n2. Some discussions are confusing to me. For example, in Line 197 to 200:\n> We note that it does not close the possibility that better initialization statistics is the sole cause of benefits because there exist other initialization schemes such that the statistics of the pre-trained model are a subset of the statistics from the initialization scheme.\n\n    I could not get the connection between the reason and the result.\n\nDespite some concerns mentioned above, I find that the overall contribution of this paper is strong enough to bring insights on synthetic pretraining into the research community. Thus, I vote for 'weak accept' temporarily. Discussions on the above points are warmly welcome. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FJd-gBOv2py", "reviewer": "Reviewer_tNV7", "summary": "The authors explore synthetic pretraining of T5 style language models. They investigate previously proposed synthetic tasks and introduce simpler synthetic tasks that they find to provide similar benefit to the more complicated previously proposed synthetic tasks.", "questions": "As you find that synthetic pretraining provides you with a model performing \"in between\" of random initalization and real pretrained model, have you considered taking the synthetically pretrained model as initialization for real pretraining? If so, are you able to reduce the number of pretraining steps on real-world data while achieving same or better quality?", "limitations": "The authors only evaluate on T5-Small with 60M parameters. It remains unclear to what degree synthetic pretraining would help on larger models. The authors also do not investigate to what degree synthetically pretrained models can be continually pretrained on real world data and if such continued pretraining can close the gap in performance to the pretrained T5 with less real world data than what the T5 model saw during pretraining.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors find that simple synthetic tasks can perform similar to synthetic tasks from the literature, which is original and interesting. Unfortunately the authors do not touch on key questions opened up by their results, such as investigating if results hold on larger models (ie. if the percentage of the gap between Random Init and Pre-trained T5 closed by synthetic pretraining remains constant) or if continued real pretraining with the initialization obtained from synthetic pretraining can reduce the amount of data required for real pretraining. Thus the work appears to be in a preliminary, though encouraging state that is relevant to inform further research.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "C6_E362sdjj", "reviewer": "Reviewer_6Qmz", "summary": "This paper presents a new insight into synthetic pre-training rather than natural language pre-training. Compared to the previous work, the paper makes experiments on a wider range of tasks (e.g., semantic parsing, code translation, natural language MRC), present more (and simpler) synthetic pre-training tasks, and explore in depth the source of gain of synthetic pre-training.\nThis work conducts a comprehension evaluation of synthetic pre-training methods. The key findings are three sides:\n1.\tUsing synthetic tasks for pre-training achieves large gains compared with random initialization.\n2.\tUsing simpler synthetic tasks like Set and Identity obtains similar benefits to existing more complex synthetic tasks.\n3.\tThe benefits of synthetic pre-training can be attained by initializing based only on statistics.\n", "questions": "see the weakness/detailed concern 7 above.", "limitations": "not see", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nIt is an important direction to interpret the effects of pre-training methods. This work presents some interesting attempts to study the influence of different kinds of synthetic pre-training tasks that are generic vs. complex and in non-natural language vs. in natural language. It is also good to see that both generic and non-natural language tasks bring comparable gains compared with the existing methods. The findings may benefit future studies on designing better pre-training tasks.\n\nWeakness\n\n\n[Organization]\n\nAlthough this work has the merits of comprehensive analysis and interesting findings, this work is not well organized. The motivation is not clear. I would suggest motivating this work by discussing some limitations of the current synthetic pre-training methods. According to the discussions in Sections 4-5, existing synthetic tasks might have disadvantages, e.g., computation complexity/cost, so they are not optimal as the pre-training tasks, and more generic ones (proposed in this work) would be better.\n\nBesides, this paper does not give a clear answer to the core question raised in the introduction, “What properties of pre-training are sufficient for effective gains”. It would be better to clarify it in conclusions.\n\nThe experiments are presented without a principle line. The results are listed coarsely, and it is hard to learn from them and figure out how they could benefit future studies. \n\n[Experiments]\n\nThe comparisons in Table 1 may not be fair enough. Some pre-training methods use different training steps and are based on different sizes of training data. It would be better to add some columns to indicate the training steps, data size, or training cost for more clear comparisons. \n\n[Analysis]\n\nThe design and usefulness of the simple synthetic tasks lack theoretical analysis. Why are the simple tasks, e.g., Set and Identity, intuitively better than the existing methods (which may be more linguistically motivated, e.g., Nesting Dependency Artificial Language). How could the model learn from the identity task by copying the input to the output? Is there any supportive reference for the knowledge acquisition process?\n\n\n[Detailed Concerns]\n\n1.\tThis paper tries to explain the problem from trying large numbers of different synthetic pre-training settings. However, the performances of almost all of them are limited (i.e. worse than the original). From this perspective, the greatest contribution of this paper seems to be the two somewhat effective synthetic pre-training tasks (Set and Identify). In practical, besides, this paper does not present any definitive or helpful conclusions for future language pre-training. The claims in the paper are weak or somewhat unsure. The above drawbacks make the contribution of the paper weak.\n2.\tIt shows that the LM can learn some simple clues from synthetic pre-training without being exposed to natural language. However, the gain on natural language in very limited against on artificial language. From Table 1, the performances on SQuAD are significantly dropped compared to pre-trained T5 (e.g. T5 77.5/86.0 -> LIME 50.4/62.1).\n3.\tConsidering the great difference, it is possible that synthetic pre-training provides almost no gain for natural language learning. Further experiments are supposed to be done on natural language tasks (e.g. GLUE or GLUE sub-tasks).\n4.\tThere seems to be an attempt to avoid the issue, making most experiments on a summarization task (CNNDM). The clues for summarization can be more easily to learn, rather than for reading comprehension.\n5.\tIt shows that only keeping the pre-trained statistics can retain most pre-training benefit from synthetic pre-training. However, it needs to be justified that the similar situation (e.g. initialize the model with T5 pre-trained weights using the same methods) does not exist on natural language pre-training. Pre-training is supposed to learn the coarse distribution of a certain language.\n6.\tThe pre-training settings is not comprehensive. Is one million data sufficient for synthetic pre-training? If with more pre-training data, the conclusions in the paper can be more credible. Another concern is that if the claims made in the paper is general to a bigger model. It is better to run the experiments on a bigger setting (e.g. 12 encoder layer + 12 decoder layer).\n7.\tWhat if we combine some of the great synthetic tasks together?\n\nMinor Issues:\n\nUse \\citet{} for inner-text citations. \nLine 63: no need to capitalize “nesting dependency”.\nLine 91: what are those “original papers”? References needed. \nLine 166: There is no Figure 4 in the draft. \nTable 2: There is an unnecessary empty line in the table. \n\nTypo?\nFigure 4 tends to be Figure 1? I cannot find Figure 4 in the main paper.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "Byz-cqg4vvj", "reviewer": "Reviewer_iZRj", "summary": "This paper investigates what properties of synthetic pretraining lead to better performance on downstream NLP tasks. The authors find that very simple synthetic tasks perform at a comparable level to prior state-of-the-art methods. For example, a straightforward identity task where the model must output the same tokens provided as input. This motivates further steps to simplify, where only the use of weight initialization statistics is sufficient to achieve much of the gains of pretraining.\n", "questions": "- Maybe I missed the details, but how is the random init baseline trained? Is poor weight initialization not something that could be overcome with sufficiently long training?\n", "limitations": "The authors do a good job acknowledging that this investigation only scratches the surface of a rich and complex area and there's much more to be done in understanding the benefits of pretraining.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- This is a solid paper, it is written well and provides an interesting investigation uncovering insights that could otherwise be easily overlooked in the community.\n- I appreciate how the paper is organized and the journey we go on of further simplification. The process of stripping away complexity piece by piece until we are down to just a hyperparameter for initializing layer norm is in my opinion much more insightful than if the authors had instead chosen to present some new elaborate task that bumped performance by half a percent.\n- The experiments follow a natural trajectory and are fairly comprehensive.\n\nWeaknesses:\n\n- The insight into initialization of layer norm parameters is interesting, and not something that I imagine most people would pay attention to without this sort of investigation. One question is whether knowing that this is an important parameter concretely changes the results one can achieve in these pretraining/finetuning regimes. Have the authors seen any net improvement to performance (higher accuracies, shorter training times, etc) by going back and pretraining and finetuning but this time with a better initialized model?\n- One part that nags at me is that most of the comparisons of the paper report relative performance along a linear scale (e.g. method A closed 70% of the gap between random init and natural language pretraining). On one hand, it’s not necessarily an inappropriate  way of presenting numbers, but on the other, further improvements to performance are much more difficult than those initial gains. It could be trivial to get most of the way to matching a particular baseline (and in some ways this is shown with the weight initialization results) and yet it might be impossible to close the rest of the gap. This isn’t reflected in the way results are reported and discussed throughout the paper - the reader doesn’t necessarily have any way of knowing how much more difficult it is to get to say, 35.8 from 32.8 even though that certainly seems very close from a baseline of 18.9. And with the random init baseline being so poor, all of the relative numbers look better for it.\n- It would have been interesting to see some comments on the relative performance of the various simple synthetic tasks. I found it fascinating that there was a clear distinction of the worst performing tasks being the only tasks where the output did not involve reusing input tokens (count, length, search, longest word). I know there’s only so much real estate for discussion, but it’s not clear why some simple tasks would fare so well while others so poorly, so some discussion along these lines might make sense to include.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept