Transcending Scaling Laws with 0.1% Extra Compute
Authors: Yi Tay
Jason Wei
Hyung Won Chung
Vinh Q. Tran
David So
Siamak Shakeri
Xavier Garcia
Steven Zheng
Jinfeng Rao
Aakanksha Chowdhery
Denny Zhou
Donald Metzler
Slav Petrov
Neil Houlsby
Quoc V Le
Mostafa Dehghani
Venue: EMNLP 2023 Main (2023)
Keywords: language models
scaling laws
emergent abilities
efficiency
pretraining
TL;DR: Continued training with a small number of UL2 steps can dramatically improve scaling laws

Abstract: Scaling language models improves performance but comes with significant computational costs. This paper proposes UL2R, a method that substantially improves existing language models and their scaling curves with a relatively tiny amount of extra compute. The key idea is to continue training a state-of-the-art large language model on a few more steps with UL2's mixture-of-denoiser objective. We show that, with almost negligible extra computational costs and no new sources of data, we are able to substantially improve the scaling properties of large language models on downstream metrics. In this paper, we continue training a baseline language model, PaLM, with ULR2, introducing a new set of models at 8B, 62B, and 540B scale which we call U-PaLM. Impressively, at 540B scale, we show an approximately 2x computational savings rate where U-PaLM achieves the same performance as the final PaLM 540B model at around half its computational budget (i.e., saving ~4.4 million TPUv4 hours). We further show that this improved scaling curve leads to "emergent abilities" on challenging BIG-Bench tasks---for instance, U-PaLM does much better on some tasks or demonstrates better quality at much smaller scale (62B as opposed to 540B). Overall, we show that U-PaLM outperforms PaLM on many few-shot setups, including reasoning tasks with chain-of-thought (e.g., GSM8K), multilingual tasks (MGSM, TydiQA), MMLU and challenging BIG-Bench tasks.

Transcending Scaling Laws with 0.1 % Extra Compute
Yi Tayl
Jason Weit
Hyung Won Chung
Vinh Q. Tran
David R. So
Siamak Shakeri
Xavier Garcia
Huaixiu Steven Zheng
Jinfeng Rao
Aakanksha Chowdhery
Denny Zhou
Donald Metzler
Slav Petrov
Neil Houlsby
Quoe Le
Mostafa Dehghani
Google
(vqtran, d degrrrnneeehhhaigggggggee.e.c com
Abstract
69
PaLM 540B
Scaling language models improves perfor- mance but comes with significant computa- tional costs. This paper proposes UL2R, method that substantially improves existing language models and their scaling curves with a relatively tiny amount of extra compute. The key idea is to continue training state-of-the- art large language model on a few more steps with UL.2's mixture-of-denoiser objective. We show that, with almost negligible extra computational costs and no new sources of data, able substantially improve the scaling properties of large language models on downstream metrics. In this paper, we continue training a baseline language model, PaLMM with UL2R, introducing a new set of models 8B, 62B, and 540B scale which call U- PaLM. Impressively, at 540B scale, we show an approximately 2x computational savings rate where U-PaLM achieves the same performance as the final PaLM 540B model at around half computational budget (i.e., saving ~-4. mil- lion TPUv4 hours). We further show that this improved scaling curve leads to "emergent abil- ities" on challenging BIG-Bench tasks-for instance, U-PaLMM does much better some tasks or demonstrates better quality at much smaller scale (62B opposed to 540B). Over- all, we show that U-PaLM outperforms PaLM on many few-shot setups, including reasoning tasks with multilingual tasks (MGSM, TydiQA), MMLU and challenging BIG-Bench tasks.
NLP
U-PaLM 540B
68
67
Zero/Few Shot
64
63
0.50
0.75
1.00
1.25 1.50
1.75 2.00 2.25 2.50
Training FLOPS
Figure 1: Compute (training flops) versus Quality (average 204 NL.P zero and few-shot tasks listed in Appendix 11.2). The black dotted line shows the path from initialization from a PaLM checkpoint and training further with UL2R.
et al., 2022a, Ganguli et al., 2022], reasoning capabilities [Wei et al., 2022b, Lewkowycz et al., 2022], inter alia. Generally, scaling laws predict continued improvement in language model quality as we continue scale up the computational budget (e.g.. bigger models or more data). date, most large language models that form the basis of scaling law research are trained almost exclusively as left-to-right causal language models [Kaplan et al.. 2020, Hoffmann et al., 2022].
This paper proposes new method to dra- matically improve the scaling curves of large language models on downstream performance with relatively tiny amount of additional computation cost. The key idea to continue training an existing causal language model [Chowdhery et al., 2022] with a mixture of new objectives--ppecifcclly the UL2 training objective mixture [Tay et al., 2022b]. This restoration is expected to only cost roughly 0.1% to 1% of the original training FLOPs and requires new data sources, making it highly efficient and convenient. We call this approach UL2R or UL2RRstore.
(e.g., GSM8K),
1 Introduction
There has been significant interest in scaling of language models [Rae al., 2021, Chowdhery
al., 2022, Brown al., 2020]. Scaling has inspired new research across multiple fronts, e.g.. scaling laws [Kaplan et al., 2020, Hoffmann et al., 2022, Tay et al., 2022a], emergent abilities [Wei
The UL2 objective combines prefix language modeling and long-short span corruption (e.g., infill-
Work completed while at Google.

---

ing) tasks [Raffel et al., 2019] that can be controlled inference time using mode switching prompt. Training a large language model with UL2 can be interpreted as teaching it to leverage bidirectional attention (i.e., Pr) or leverage infilling-style pretraining that have been the foundation of lan- guage understanding (e.g.. T5 [Raffel et al., 2019]). To this end, we postulate that imbuing a state-of art large language model such as PaL.M [Chowdhery et al., 2022] with these diverse pretraining schemes as a complement to the original language model ob- jective, enables the model to perform significantly better. Moreover, the UL2 objective enables new prompting capabilities in PaLM which allows it to perform infilling based prompting.
abilities. This work shows that diversity and richer training paradigms can be key to learning new capabilities that were previously hard to acquire with only causal language modeling.
Finally, in addition to emergent task performance and overall improved scaling curves, we show that U-PaLM is also practically more useful since it is equipped with secondary mode of prompting, i.c., bidirectional infilling. Specifically, UL2R enables a secondary capability for prompting U-PaLM which can be used to fill in more than one blanks in the in- put prompt. Interestingly, we find that only a small amount of UL2R (c.g., 0.1 % tokens or FLOPs) is suf- ficient to imbue the model with this new capability.
U-PaLM
We show that adapting PaLM with UL2R not only results in significantly better scaling laws on eelllestablished few-shot NLP tasks, but also, in our scaling experiments on downstream few-shot tasks, we show that UL22 is two times more efficient (computation savings of approximately 2x) at 5.00 scale- reaching the performance of the final PaLM 540B model with only half the computation, saving up to 4.4 million TPUv4 hours.
This section introduces the technical details of U- PaLM (i.e., PaLM + UL2R). U-PaLM is initialized from PaLM and leverages the same architecture. This section describes the training procedures of UL2R and how they are applied to continue training PaLM. We refer the reader to Section 10 in the Ap- pendix for comprehensive review of related work.
In addition to competitive performance across a range of well-established NLP [Wang et al., 2019]. multilingual [Clark et al., 2020a, Shi et al., 2022], and reasoning [Cobbe et al., 2021] benchmarks, we also study the impact of UL2R on suite of
2.1 Training Data
To keep things consistent, we train this model with the same data mixture as PaLM and do not rely additional sources of data (labeled unlabeled).
There are three main reasons for this choice. Firstly, we did not want to introduce new tokens to our training process which could conflate findings. Secondly, we did not want to over- on scaling studies that only measure impact on upstream cross entropy [Hernandez et al., 2022] which claims that repeating data in small quantities could be pooporopooti harmful. Since the empirical results we obtained are strong, we postulate that repeating tokens could perhaps be not harmful at smaller quantities after all. This is also backed by the continued training of PaLM 62B in [Chowdhery et al., 2022] which showed that repeated data could result in small gains, albeit not as strong fresh to- kens. Thirdly, (via UL2) on the training data sufficiently unique and therefore prevents us from explicitly training on the same data with the exact objective or suffering from any memorization issucs.
challenging BigBench tasks from Wei et al. [2022a]. Notably, subset of tasks described as 'emer- gent' because PaLM's 's's performance remains flat up to model scale of 62B and only becomes better than non-random at 540B scale. On these set of tasks, we find that UL2R enables (1) doing significantly better at tasks that PaLM struggles at (e.g., navigate, geometric shapes, hyperbaton) and (2) elicits emergent behavior at a smaller scale such as 62B 8B (c.g., crass ai, vitamine fact verification). On top of that, U-PaLM strongly outperforms PaL.M on some challenging BigBench tasks.
Emergence within the context of large language models is a nascent research area. As the Nobel prize-winning physicist Philip Anderson put it, 'More is different." [Anderson, 1972] which describes unpredictable phenomena at different scales. In our context and with mixture-of-denoisse in UL2, we would like to think of this phenomena as 'More is different, but different can also more' since different pretraining objectives can improve language model quality or elicit new emergent
consider our data transformation
2.2 Prefix Language Model Architecture
We train U-PaLM using the prefix language model (PrefixLM) architecture, also sometimes known as

---

non-causal decoder-only model. The PrefiXLM architecture keeps a non-causal mask in its prefix (or inputs) and applies bidirectional attention to input tokens.
and efficient for the setup of continued training. We kept the original mode prompting tokens the orig- inal UL2 design. We used [S2S] for S-denoisers (PrefixLM), [NLU] for R-denosiers and [NLG] for X-denoisers. The 540B U-PaLMM model was mainly trained with 50% S-denoiser (PrefiXLM), 25% R-denoisers, and 25% X-denoisers.
In this architecture, we use a total combined sequence length of 2048 (e.g., PaLM's sequence length) which is then split to 1024 inputs and 1024 targets. In the original UL22 paper and infrastructure, an artifact of its preprocessing pipeline applies padding tokens first before combining inputs and targets. For decoder-only language models, this is inefficient since we would end up with a concatenation of [prefix] [prefix's padding] [target].
2.4 Training
We train the 540B model for a total of 20k steps with a batch size of 32. We mildly ablate these settings in early experiments with 62B and 8B models but keep them capped within certain ballpark (e.g., 128 batch size for 50k steps). As result, this is more similar to 'finetuning' compared to full pretrain- ing. The number of additional tokens is therefore very negligible compared to the original pretraining run often coming in at around or less than 0.1% 1% ad- ditional compute. The total number of extra tokens we train on for the 540B model is approximately 1.3 billion which constitutes 0.16% extra computation, as the original PaLM model was pretrained on 780B tokens. We use cosine learning rate decay schedule that anneals the learning rate from 10-4 to 10 Notably, we also tried a low constant learning rate and found them to perform quite identically. Our U-PaLM 8B and 62B models are trained using 64 TPUv4 chips. Training an U-PaLM 540B model only consumes 512 TPUv4 chips and finishes in about 5 days which is considered to be lightweight.
In this work, we optimize the Prefix padding by forcing the model to concatenate prefix and target before applying any additional padding. Packing, trimming and padding is then subsequently applied later after the prefix has been concatenated with the targets. Through this prefix optimization, we are able to improve example-level sample efficiency of the model.
2.3 Loss Objectives
This section describes the setting for the UL2 mixture-of-denoisers that we use in UL22R. The UL2 mixture-of of-denoie objective comprises of three types of denoisers.
Regular denoising whereby the noise is sampled spans, replaced with sentinel tokens. This is also the standard span corruption task used in Raffel al. [2019]. Spans are typically uniformly pled with a mean of and corruption rate of 15%. Extreme denoising whereby the noise is increased relatively "extreme" amounts in either a huge percentage of the original text or being very long nature. Spans are typically uniformly sampled with a mean length of 32 OR a corruption rate of up to 50%.
Experiments
3.1 Improved
Scaling Properties on Few-shot Learning
In this experiment, we show improved scaling curves from small amounts of UL.2R training on top of both PaLM 8B and PaLM 540B. We use downstream metrics and few-shot evaluation since (I) this is closer to usability of these models and (2) loss with UL2 and causal language modeling is not comparable. We initialized and trained multiple U- U-PaLMM models using different PaLM intermediate checkpoints. On the 8B model, we repeated this 7 times at different intervals. Given that the 540B model was more computationally demanding, we only managed to fit 3 points. For evaluation, we use the average score of NLU and NLG tasks from the GPT-3 suite [Brown et al., 2020J. In total we use 26 tasks (e-g., TriviaQA, NaturalQuestions, Super- GLUE, PIQA, OpenbookQA, ANLI etc). Detailed scores for Figure 2 can be found the Appendix.
Sequential denoising whereby the noise is always sampled from the start of the text to a randomly sampled point in the text. This is also known as the PrefixLM objective (not to be confused with the architecture).
We kept this simple since many ablations were already explored in Tay et al. [2022b]. We kept the original 7 denoisers as the initial version but later found that mixture of only three tasks, e.g., 50% PrefixLM, 25% Long (extreme) span corruption, and 25% regular span corruption to be quite simple

---

ZerofFew Shot Score on NLP tasks
Zero/Few Shot Score on NLP tasks
11.92
69
X 1.95
68
67
48
1.79
PaLM 64
44
PM 540B
U-PaLM 8B
63
U-PaLM 540B
0.50
0.75
1.00
1.25
1,50
1.75
2.00
2.25
2.50
Training FLOPs
Training FLOPs
202
Figure 2: Computation cost (training flops) [Dehghani et al., 2021] versus Quality (average of 20+ NLP zero and fow-shot tasks). The dotted line shows the path from initialization from a PaLM checkpoint and training further with UL2R. These plots also present pairs of PaLM and U-PaLM models with comparable/similar performance along with the ratio of PaLM computation cost vs the corresponding U-PaLM computation cost. For example. PaLM 540B trained for ~ 2500 zFLOPs (right most point) took ~ 2.35 times computation cost of U-PaL.M 540B trained for ~ 1075 zFLOPs, while both models comparable in terms of performance zero/few shot on NLP tasks.
186
FM U.P.M SABB
Performance
60
Tasks
Figure 3: Break down scores of individual zero-shot and one-shot NLP tasks for PaL.M and U-PaL.M 540B trained
for 780B tokens. U-PaLM outperforms PaLM 540B and achieves SOTA on 21 out of 26 tasks.
Figure 2 shows that U-PaLM substantially outperforms the original PaLM models both 8B scale and 540B scale. Note that the dotted lines represent pathway before and after UL2R training, we show that UL.2R training improves the scaling curve PaLM substantially. i.., UL2R provides a more compute-efficien performance improvement compared to training the original PaLM models for longer with the standard causal language modeling objective.
using the standard causal language modeling objec- tive OR applying UL2R for small amount of steps. Here we discuss the counterfactual savings rate of choosing UL2R as opposed to continue training with caussal language modeling. For the 540B model, the saving rates at the middle checkpoint is approximately 2x. This is equivalent to about 4.4 million TPUv4 hours for the 540B model. For the 8B model, the saving rate tend to be lowest at both the start and convergence of the model. It seems to be higher at middle stages of training (relative to convergence) which shows that the utility of UL.2R changes with respect the amount of causal language modeling training already done. For the 540B model, since the PaLM model was not trained to convergence and the number of tokens to param- eters ratio is relatively low, the savings rate could still be increasing even beyond 2.35x. Overall, the amount of savings is quite proportionate to the point of training and stage of convergence of the model and can probably be predicted by standard scaling laws [Kaplan an et al., 2020, Hoffmann et al., 2022].
8B versus 540B Generally, UL2R consistently improves the underlying PaLM models. Neverthe- less, observe different bchaviors the 8B and 540B models. The gap seems to narrow the perfor- mance of PaLM 8B starts to plateau, i.c., the largest gains are near to the middle of training. As for 540B, the gain continues to grow even 780B tokens. We believe that this is due to the fact that PaLM 540B still has significant headroom beyond 780B tokens.
Savings Rate At a certain stage of training, we have an option to continue training for K more steps

---

Table List of challenging tasks the BigBench emergent suite (BBES) and corresponding scores PaLM 540B and U-PaLM 540B. All results reported with standard 5-shot prompting.
task
task /reasoning type
PaLM 540B
U-PaLM 540B
navigate
arithmetic, logical multi-step
55.3
67.0 (+21.2%)
strategyqa
73.9
78.3 (+6.0% %)
crass_ai
commensense
100 (+2.4%)
logical_sequence
commonsensse
92.3
86.5 (-6.7%
contextual, commonsense
understanding fables
commonsense
75.7
78.4 (+3.6
analogical
87.2
87.5 (+0.3 %%)
hyperbaton
contextual QA
54,2
59.9 (+10.5%)
cnusal_judgrent
caussal and commomsense
65.3
68.4 (+4.7 %)
english_proverbs
contextual QA
91.2
87.5 (-4.2%)
gnometric_shapes
al glgorithmic, visual
44.0
49.3 (+122.0%)
physics_questions
logical, physics, math commmonsense
7.6
12.5 (4.6..5%
snarks
69.1 1
86,1 (+24.6%%
analogical_se simiiiiity
analogical reading comprehension
internatiinal._ponnttii
wtt__ppotttcppphheeeeee alphabet_nlii
63.9
68.0 (+3.2
www.dallo.sssme_rrr..o.ooooo _ireeeoordiioieeeeeoo fferent timedial
commonsense, reading compre.
(+6.22 %)
commonsense, logical
78.3
81.2 (+3.7 %))
question_selection
reading comprehension
59.8 (+9.1
logical_Fallacy.deteee detection
logical reasoning
80.3
81.4 (+1.4%)
unit unit_interpretation
arithmetic, logical
47.0
$1.0 (+8.5%%
l.ngunge_identiiicatinn
multiinguul
36.0
38.9 (+8.1
average (21 tasks)
64.3
67.7 (+5.3%%
100
L-PMlM
Gopther
Hyperbaton
Navigate
6.28
602 EUp
1214
100
10M
Trannng PUOPS
Trukeng HLOPS
Thanko nore
Identify edd metappor
80
UPPMLM
PalM
50
PAiM
FOLM
Gopher
GPT
Gopterr
GP133
Vitaminc
GPI
Smardos
512
1077
Trannng FLCRS
Traking TLOPS
Traning TLOS
PaLM
Understanding Edbiles
76
PULM U-heiM
PJLM
1-771M
Goptrer
Goptert
GPT-3
Strategyga
CP733
CPT:3
200
Geomettiic
22
100
508
KLDPS
Themay Hore
Figure 4: Scaling plots BIG-Bench emergent suite (BBES) for different sizes of PaLM, U-PaLM, Gopher, and GPT-3 as function of training FLOPs. Scores are normalized scores where zero denotes more or less random performance. X-axis log-scale.
Breakdown on individual tasks Figure reports the individual scores on each zero and one-shot task in the mixture. We show that U-PaLM 540B outperforms PaLM 540B on out 26 tasks. Given that PaLM is the SOTA language model on these tasks, this makes U-PaLM the new Mate-of-the-aa art on these tasks.
3.2 BigBench Emergent Suite
We select suite of challenging tasks from BigBench based criterion that performance on PaLM on these tasks remain relatively flat-lined 8B and 62B scale but suddenly unlocks 540B, We also consider tasks that are difficult for PaLM 540B to solve (near random performance). We call these

---

Table 2: Results on finetuning on SuperGLUE and TydiQA dev sets.
PaI.M 8B
U-PaL.M
PaI.M 62B
U-PaI.M 62B
SuperGLUE (Avg)
83.4
86.1 (43.2% %)
89.5
91.4 (+2.1
TydiQA (EM/FI)
75.7 85.2
77.5 (+2.3%) / 86.7 (+1.7%)
78.3 78.3/87.3 87.3
78.4 (+0.1 %) 18.5 (+2.1%
Table 3: Results on Massively Multi-Taan Language Understanding (MMLU) test
known be one factor that results in emergent task performance [Wei et al., 2022a]. We show that UL2R is able to elicit emergent abilities at smaller scales. For example, the quality on certain tasks such as crass_ai, vitaminc, ddentiffy_odd_metaphorr
Method
Accuracy
Random
25.0%
Average Human Rater
34.5%
are tasks where performance starts to spike 62B scale (as opposed to only at 540B with the PaLM model. In rarer occasions, the performance of U-PaLM 8B is even higher than PaLM 62B (e.g., snarks, understanding.fabllee
GPT-3 5-shot
43.9%
Gopher 5-shot
60.0%
Chinchilla 5-shot
67.6%
PaLM 540B 5shot
69.3
U-PaLM 540B 5-shot
70.7 % (+2.0%)
Overall, these results show that there are strong evidence that inductive bias (e.g.. combinations of prefix language mod- eling, span corruption based pretraining in UL2) could be crucial when it comes to unraveling new abilities in large language models.
suite of tasks EMERGENT suite of BigBench tasks (BBES) inspired by the criterion set by Wei et al. [2022a]. Note that while these set of tasks overlap but are not entirely identical to BBH [Suzgun et al., 2022]. Moreover, BBES uses the default prompting and templates as BIG-Bench and do not use chain-of-thouuhh prompting. Hence, they are not entirely comparable. BBH results can be found later in section 11.1.3.
3.2.2 MMLU Results
We compare PaLM and U-PaLM the Massively Multi-Tase Task Language Understanding (MMLU) benchmark [Hendrycks al., 2020]. Table reports our results on MMLU's test set. Prior results are reported from [Hoffmann et al., 2022]. Our results show that U-PaLM outperforms PaLM on this task in the 5-shot setup by 2.0% relative gain.
3.2.1 BIG-Bench Results
Table reports the results of PaI.M 540B and U-PaLM 540B on the BigBench emergent suite. We also describe the task and reasoning task for each task. Note that some tasks require conjunction of various 'skills' to excel at. For example, the navigate task is combination of spatial reasoning and arithmetic (counting).
3.3 Finetuning
We conduct experiments SuperGLUE [Wang et al., 2019] and TydiQA [Clark al., 2020a] finetuming. We conduct experiments 8B and 62B scale Fine-tuning is conducted with a constant learning rate for 100% steps with batch size 32. Table reports finetuning results. We observe that there is substantial improvement in fine-tuning especially at the 8B scale. The gains diminish slightly 62B scale but are still modest in general. We note that PaLM's fine-tuning performance be generally considered weaker than expected. For instance, PaL.M 8B generally outperformed by a T5. 1.1 large model on the SuperGLUE dev average. We postulate that training PaLM on UL2 and span corruption tasks in complement to causal language
Overall results and Scaling Plots We observe that U-PaLM outperforms PaLM on 19 out of the 21 tasks at 540B scale. Morcover, the gains certain tasks are substantial (c.g.,
on navigate and 69.1% 86.1% on snarks). On average, there is +5.4% relative quality gain on the un-normalized aggregated average across all 21 tasks which we consider to be pretty strong results. Figure 4 which shows the scaling plots of U-PaLM relative to other models. Whenever possible, we also include baselines such as GPT-3 or Gopher from the official BIG- BiG-Benchre repositor.
'Finetuning at 540B is compute intensive and probably less relevant for the finetuning setup since large scale LMs are typically used for prompting. Mcanwhile, it is significantly more likely that smaller models are fine-tuned.
UL2R unlocks emergent task performance at smaller scales Scale (e.g., scaling to 540B) is

---

Original Model Input
U-POLM 6406 output
PaLM 5408 output
"Steos to making cake:
Bake the cake.
A. Put the batter into cake
1. Weigh out all the ingredients.
Put it into the oven. C. Weigh out all The ingredients. D. Put the batter Info cake pen.
3. Put the better into the cake pan.
4. the oven.
What is the second step?"
Infil Model Input
U- 6408 output
PaLM 6408 output
making a
ingradients.
minutes, out of the oven. Let cool. B. Frost 9. [-]
1. Weigh out all the ingredients.
<extra,
3. the better into the cake pan.
4.
Multi toéil
output
"Steps to making cake:
cextra Id, 0>2, Mix the ingredients sextra 1o4, Pour batter into a pan. sextra,jd jd. 2-6, Cool cake.
7. Take the cake out the oven. 9. cool. cake, 13. Eat the cake.
Weigh ingradients.
<eottro_ id D>>
Prahest
<eottra 1d
Bake
<extraaid id__2 2>
corsaoondnn
Figure 5: example of prompt that improved by rephrasing to use U-PaLM's infilling capabilities.
modeling can ameliorate some of its flaws. Our results ascertains this by showing that U-PaLM strongly improves quality especially smaller (8B) scales.
in. Notably, with U-PaLM it is possible to query both the infill style and the traditional style via the usage of extra ID tokens (as is used in denoising) or without, respectively.
In Figure 5, we include example outputs for PaLM, U-PaLM with traditional prompting, as well as U-PaLM with infill prompting. We phrase this particular prompt in two ways: one question that is suitable for traditional prompting via PaLM and one leveraging U-PaLM's 's infill capabilities. In the traditional phrasing, both PaLM and U-PaLM do not produce the correct answer. With the infill phrasing. PaLM ignores the infill token (extra ID token) as PaL.M has not seen it during training, and instead produces the rest of the steps after step 4. U-PaLM correctly infills the second step in this example. Finally, third example included demonstrate U-PaLM's ability to infill multiple slots. These examples demonstrate that, with only a small amount of additional training, we are able to expand the functionality of PaL.M to serve an entirely new class of queries.
3.4 Additional Results & Analysis
We conduct additional, extensive, evaluation and analysis of our approach. Due to space constraints we refer the reader to Section I in the Appendix. There we provide results for zero-shot and few-shot NLP tasks including commonsense reasoning, closed book QA & reading comprehension, reason- ing chain-of-thought, and few-shot multilingual tasks. find that the improvements from U-PaLM over PaLM generally hold across these additional tasks, with major improvements on certain tasks such as GSM8K (+6.6%) [Cobbe et al., 2021]. BIG-Bench Hard (+10.7%) [Suzgun et al., 2022], and MGSM 7%) [Shi et al., 2022]. We also include analysis of BBES performance, scaling curves for few-shot experiments, and additional discussion of our methods.
4 Qualitative
4.2 Leveraging Specific Pretraining Modes
Analysis: New Prompting Capabilities
Recall that via the UL2 objective, R-, X-, and S-denoisers are associated with the [NLU], [NLG], and [S2S] mode tokens respectively. S-denoisers essentially the PrefixLM objective, while R- and X X-deooierss variations of span corruption, and thus are also associated with extra ID tokens which we can use during prompting for infill (as shown above.) Given this unique setup, we can control the mode token during inference to gain access to
4.1
Infilling Ability
Left-to-right casual language model pretraining has typically allowed models to provide meaningful continuations of prompts. With U-PaLM observe that, by extending pretraining with a small amount UI.2 denoising steps, the model is also able to pick up infilling abilitics where the model is given a location in the middle of prompt to fill

---

Original Model Input
U-PoLM 5408 Default output
PoLM 640B ourput
"Answer Vietnameee: the color grass is
Xanh!
'grean'
Icernet but not spoolle enougho
X
Jesewer Engliin)
5408 [525] output
U-PALM [NLU] Infili output
U-PALM 5408 [NUG] Infill output
"anh la chy
<extro id
<extra kd gP.
(wrtnng answwe)
Figure 6: An example of prompt that works only when querying specific pretraining mode.
Original Imput
output
"A haiku about a vai baking cake a lake:
thought saw it saw cat cake, On top of lake.
cat on lake
Baking Acat on lake.
U-PaLM s4oe [525] output
U-PWLM s4an [NLU] output
2408 [NLG] output
A lake
thought saw it then saw cat bake cake, On less.
I sse baking
Baking a cake on lake
A cake on the lake, kitly It's appetizing!
lake cake, cal.
U-PaLM 540B Defoult Infill output
U-PalLM 6408 [NLU] Infil output
U-PaiLM 5408 (NLG] GI Infill output
baking The cake is ie.
lake
Baking cake for his sake cat,
Baking cake for his sake cat, can't
Figure 7: Querying U-PaLM for diverse outputs by using different prompt mode token and LM/infill combinations.
specific knowledge that might have been acquired in one mode but not another. This effectively provides us with more options in how to answer prompts, without the need to make any changes to the learned model or its inference algorithm.
no mode tokens. The variety of ways to prompt U-PaLM results useful technique to increase the diversity of the outputs we can get from the model, without resorting to alternative decoding techniques (e.g. sampling). This is particularly useful for more open-ended prompts.
In Figure 6, we include challenging exam- ple where we ask the model to do zero-shot cross-lingual question answering from an English question into a Vietnamese answer. For PaLM and U-PaLM default, we pass the input as-is to the model. For the rest, we prepend one of [S2S], [NLU], or [NLG] to the beginning of the input, and in the case of [NLU] and [NLG], we add the infill token at the end of the input, as typical for these modes. Interestingly, U-PaLM in [S2S] mode is the only variant that returns the correct answer in Vietnamese. Regular PaLM produces the correct answer, but ignores the Vietnamese request, while U-PaLM with default prompting (no mode, no infill) produces a roughly correct answer but could be more specific (' ('xanh' encompasses both greens and blues). This example shows how accessing specific mode tokens may work well for some prompts more so than others, giving us a powerful technique to serve larger variety of prompts.
In Figure 7, we ask PaLM and all variants of querying U-PaLM to write a haiku about "a cat baking a cake on a lake" a very random prompt that the model is unlikely to see during training, yet requires very structured output. All outputs use greedy decoding here, and surprisingly all models generate reasonable haikus about the topic, although not all follow strict 5-7-5 syllable structure. PaLM's haiku repeats the first and last line, which is somewhat less interesting. We can see that the different combinations of querying U-PaLMM results in pleasantly varying poems.
4.3 Improved
Diversity for Open-ended Generation
Beyond improving the scaling behavior of PaLM, we find that the small amount of continued training applied in UL.2R is sufficient to imbue PaLM with new prompting abilities introduced by the UL2 objective. Namely, the use of denoising in UL2 allows PaLM to acquire infilling abilities. Infilling allows U-PaLMM to have a second approach tackling prompts, which we observe to be very useful. In addition, with U-PaLM we can also
Even though [NLU] and [NLG] modes typically coincide during pretraining with span corruption (involving extra ID tokens, infilling), we can still use [NLU] and [NLG] mode tokens with no infilling at all. Similarly we can use infilling but with

---

supply mode tokens gain access to specific pretraining objectives. This gives us a powerful tool to control the model without making any updates to the model or its inference. In this section we provide some examples of situations where U-PaL.M's expanded prompting capabilities prove to be useful.
8 Acknowledgements
We thank Le Hou and Oliver Bousquet for their advice and feedback on the paper. We thank Barret Zoph and William Fedus for early discussions about this paper. We thank Adam Roberts for feedback on prior work.
5 Conclusion
We proposed UL2R for continued training of PaLM. We show that with only 0. % additional FLOPss (or compute), we are able to improve the scaling curve and properties of PaLM on many downstream tasks and metrics, Notably, UL2R enables 4.4 million TPUv4 savings at 540B scale. The resulting model which we call U-PaLM outperforms PaLM on English NLP tasks (e.g., commonsense reason- ing and closed book question answering), reasoning tasks with chain-of ooan--of-the tfottht multilingual reasoning, MMLU and a suite challenging BIG-Bench tasks.
References
Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava,
Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. Muppet: Massive multi multi-task representationn with pre- finetuning. arXiv preprint arXiv:2101. 1 2021.
Philip W Anderson. More is different: broken sym- metry and the nature of the hierarchical structure of science. Science, 177(4047):3933390 1972.
Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Prakash Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. Ext5: Towards extreme mulki-task scaling for transfer learning. ICLR, 2022. URL https://arxinn org/ /t////bs//11111 21t.//21....11 10952.
6 Limitations
In this work we show the effectiveness of continued training of a 540B PaLM model with UL2R over conditional language modeling alone. We only demonstrate this for the PaLM model and pretraining corpus. Our study only a demon- stration of what is possible with an example near ttate-of-the-art system, and we do not provide results on what would happen if the underlying model and pretraining corpus were to differ from the studied here. For example, what would happen if we applied ULR2 model that was trained to saturation on a corpus already? Would we observe similar improvements? What would happen if we use a weaker underlying model? This paper also only studies models with 8B+ parameters, and does not provide insight on how UL2R would perform on smaller models and compute regions. We leave these investigations for future work, and this work should not be interpreted comprehensive study of continued pretraining or model reuse.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 7432-7439, 2020.
Tom B Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sut- ton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.0233 2022.
Hyung Won Chung, Longpre Shayne Hou, Le and, Barret Zoph. Yi Tay, William Fedus, and et al. Scal- ing iestruction-finttuned language models. arXiv preprint, 2022.
7 Ethics Statement
Christopher Clark, Kenton Lee, Ming-Wa Chang, 'om Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising diffi- culty of natural yes/no questions. In NAACL, 2019.
As this work continues training PaLM, we defer discussion of ethical considerations with respect to large language models to the original PaLM paper [Chowdhery et al., 2022]. We do note though that this work presents way of improving large language models without training from scratch, and all the different types of cost (c.g. environmental) that that might entail.
Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. Tydi qa: benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8: 450- 2020a.

---

Kevin Clark, Suan-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint ar0iv: 10555, 2020b.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.088 000361, 2020.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Retichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.1411 2021.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wee Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural ques- tions: benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019.
Mostafa Dehghani, Yi Tay, Anurag Arnb, Lucas Beyer, and Ashish Vaswani. The efficiency mis- nomer. In International Conference on Learning Representations, 2021.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutamova. Bert: Pre-training of deep bidi- rectional transformers for language understanding.
Brian Lester, Rami Al-Rfou, and Noah Constant, The power of scale for parameter-effiiient prompt tuning. arXiv preprint arXiv: 08669 2021.
arXiv preprint preprimt urXiv:1810.04800, 2018.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, ct al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206. 14858, 2022.
Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou. Compositional se- mantic parsing with large language models. arXiv preprint Xiv: 55003, 2022.
Deep Ganguli, Danny Hernandez, Liane Lovitt, Nova DasSarma. Tom Henighan, Andy Jones, Nicholas Joseph, Jackson Kernion, Ben Mann, Amanda Askell, et al. Predictability and surprise in large generative models. arXiv preprint arXiv:2202.05 277707850 2022. URL https:ttp//// /tttpp://axxv...oo www.akiwwwrg/bb///22..00
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin. Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 2022. URL https:://axivv org/ aaa/a/bb/////.../ 02155.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use laptop? question answering benchmark with implicit reasoning strategies. Transactions the Association for Computational Linguistics, 9: 346-361, 2021.
Denis Paperno, Germ's an Kruszewski, Angeliki Lazari- dou, Ngoe Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fern andez. The LAMBADA dataset: Word prediction requiring a broad discourse context, In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 1525-1534, Berlin, Ger- many, August 2016. Association for Computational Linguistics. doi: 10. 10.18653/v1/P16- 1144. URL http:::/www..clll/ orgg/nttologgy//eeeeeeo
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understand- ing. arXiv preprint arXiv:2009.00 033000 2020.
Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage. Zac Hatfield-Dodds, Tom Hemighan, Tristan Hume, al. Scaling laws and interpretability of learning from repeated data. arXiv preprint arXiv:2205. 10487, 2022.
Jack W Rae, Sebastian Borgcaud, Trevor Cai, Katic Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11444 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute- optimal large language models. arXiv preprint 0rXiv:2203.15556, 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits trans- fer learning with a unified text-to-text transformer. arXiv preprint arXiv: 10683, 2019.
Mandar Joshi, Funsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly super- vised challenge dataset for reading comprehension. arXiv preprini arXiv: 170.00351,.000110000 2017.
Adam Roherts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the pa- rameters of language model? arXiv preprint arXiv:2002.089/00 2020.

---

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhaga- vatula, and Yejin Choi. Winogrande: An ad adversarial winograd schema challenge at scale. arXiv preprint arXiv: 10047, 2019.
Zheng, Neil Houlsby, and Donald Metzler. Unify- ing language learning paradigms. arXiv preprint TrXiv:2205.0513: 2022b.
Yi Tay, Vinh Q Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mchta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, et al. Transformer memory as differentiable search index. arXiv preprint (xxx:2202.0699) 2022c.
Victor Sanh, Alhert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, 7aid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. ICLR, 2022. URL https: / //openreview.. wtwprrrviwewene/formni.mm. nett:forummdd/-yyrssoooo
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoory Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.088 082399, 2022.
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay. Sebastian Ruder, Denny Zhou, et al. Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.0333 2022.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Superglue: A stickier bench- mark for general-purpons language understanding systems. arXiv preprint arXiv: 1905.00537 2019.
Aarohi Srivastava, Abhinav Rastogi. Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Carriga-Alonso al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.046155 2022.
Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. What language model architecture and pretraining objective work best for zero- zero-shot generalization? arXiv preprint orXiv:2204.0558 05232233.2 2022a.
Jacob Steinhardt. Future ml systems will be qualita- tively different, 2022. Accessed May 20, 2022.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022.
Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sehastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery. Quoe V. Le. Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thoughh can solve them. arXiv preprint, 2022.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint w.Xiv22109001652 2021.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: question answering challenge targeting commonsense knowl- edge. In Proceedings of the 2019 Conference of the North American Chapter of the Association Human Lan-
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. Trans- actions on Machine Leamning Research (TMLR). 2022a.
for Computational Linguistics: guage Technologies, Volume 1 (Long and Short Papers), pages 4149-4158, Minneapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10.18653/11/11914222 URL https://aclantholoogy.ooo (rt.ooyy.or/N9-112....
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Lc, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. Conference Neural Information Processing Systems (NeurIPS), 2022b.
Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pre-training and fine-tuning transformers. arXiv preprint ar Xiv: 10686, 2021.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Car- bonell, Russ R Salakhutdinov, and Quoe V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
Yi Tay. Mostafa Dehghani. Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. Scaling laws model architectures: How does inductive bias influence scaling? arXiv preprint arXiv:2207. 10551, 2022a.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Lu- ong, Gunjan Baid, Zirui Wang. Vijay Vasudevan. Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint 2rXiv:2206.1072 2022.
Tay. Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven

---

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? arXiv preprint arXiv:1905.07830. 2019.
Appendix
10 Related Work
Large language models Scaling and improv- ing large language models is one of the most impactful research areas in modern artificial intel- ligence [Chowdhery et al., 2022]. To this end, large language models not only continue to improve we scale in terms of data or computational budget [Hofff- mann al., 2022, Kaplan al., 2020J but also acquire new abilities [Wei et et.,, 2022a]. The impact of large language models has been ubiquitous and pervasive, unlocking breakthroughs across many fields, e.g., reasoning [Wei et., 2022b, Wang et al.. 2022b, Zhou et al., 2022, Drozdov et al., 2022]. math [Lewkowycz et al., 2022], dialog [Thoppilan et al., 2022], multimodal applications [Yu et al.. 2022], retrieval [Tay et. 2022c] inter alia.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang. Dale Schuurmans, Olivier Bousquet, Quoe Le, and Ed Chi. Least-to- most prompting enables complex reasoning large language models. arXiv preprint arXiv:2205.1066 10625, 2022.
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models, 2022. URL https://arxiv.. org/ wtt.:////bbb//...eee/e// 08906
While there have been many paradigms and self-supervision methods proposed to train these models [Devlin et al., 2018, Clark et al., 2020b, Yang et al., 2019, Raffel et al., 2019], 91, to this date most large language models (i.e., more than 100B parameters) are trained as decoder-only casual language models. For example, flagship large lan- guage models such as GPT-3 [Brown et al., 2020), Gopher [Rae et al., 2021] and PaLM [Chowdhery et al., 2022] are all trained as causal language models. Mcanwhile, bidirectional models (c.g.. BERT [Devlin et al., 2018), T5 [Raffel et al., 2019], ST-MoE [Zoph et al., 2022J) have also been very popular as the goto model of choice, especially in smaller computational regimes (e.g., less than 30B parameters and often times in the ranges of hundred of millions of parameters).
Scaling laws of large language models Kaplan et al. [2020] investigated scaling laws of Trans- former language models and first showed the scal- ing laws are predictive of future performance. The authors found that model size (and not shape) cor- relates strongly with model quality, i.e., upstream cross entropy. Tay et al. [2021] studied the scaling properties of encoder-decoder models and their im- pact upstream and downstream finetuning tasks. Generally, Tay et al. [2021] found that upstream perplexity and downstream quality does not always correlate. As follow up, Tay et al. [2022a] studied the scaling laws of different model architectures and found that inductive bias does significantly impact the scaling behavior of the model. Finally, Hoff- mann et al. [2022] proposed compute-optimal mod-

---

els that popularized the 'chinchilla' scaling laws- an approach that aims be predictive of the optimal amount of data given the number of model parame- ters. In this work, we mainly consider scaling laws over downstream performance largely because this is more reflective of language model's 's usability. Since downstream performance is more important than upstream cross entropy, we advocate for future scaling studies to always incorporate downstream evaluation (and metries) as opposed only using cross entropy loss.
self-supervised objective has been explored. For example, a model trained with a language modcling objective can be adapted by further training with the masked language modeling objective [Wang et al., 2022a]. The other direction is also possible; model trained with a masked language objective can be adapted with the causal language modeling objec- tive [Wang et al., 2022a, Lester et al., 2021]. UL2R follows similar idea but uptrains language model with set of diverse and new preordaining tasks from mixture-of -oanr-oo--deooeses. even after a vast amounts of standard pretraining and demonstrates a very rapid improvement on variety of setups and tasks.
Emergent Abilities New behaviors that arise due to scaling language models have been increasingly referred to as emergent abilities [Steinhardt, 2022, Ganguli et al., 2022, Wei et al., 2022a]. For instance. Wei et al. [2022a] define emergent abilities as "abil- ities that are not present in smaller models but as present in larger models." For few-shot prompted task, this would look like flat scaling curve (ran- dom performance) until a certain critical threshold, during which performance increases to substantially above random. This type of phenomena has been observed across dozens of tasks in the BIG-Bench benchmark [Srivastava et al., 2022]. Although such emergent abilities are typically observed as func- tion of scale, increasing model scale to induce emer- gent abilitics is computationally expensive. In this paper we show how UL2R unlocks emergence with- out increasing the number of model parameters.
Unified language learner (UL2) The UL2 [Tay et al., 2022b] model is state-of-the-art model that bridges both generative causal language models and bidirectional language models. UL2 proposes a mixture-of-denoine objective that mixes prefix (non- causal) language modeling and infilling (span corruption) within the same model and leverages mode prompts to switch between modes during downstream tasks. UL2 is architecture agnostic in which the authors argue that the choice of decoder- only versus encoder-decodee models is largely efficiency trade-off. In [Tay et al., 2022b], the final UL2 model was trained as a 20B encoder-decoder model, which achieves very compelling perfor- mance on both finetuning and in-context learning.
11 Additional Results & Analysis
Continued Training of Language Models The paradigm of continue to train (or finetune) a language model on more data or tasks is commonly known as adaptation. A range of prior work has shown that finetuning language models on a collection of NLP tasks can improve downstream performance
11.0.1 Analyzing individual task performance BIG-Bench
This section dives into individual task performance and attempts to understand quality on different types of BIG-Bench tasks.
broad range of downstream tasks [Aghajanyan et al., 2021, Aribandi et al., 2022, Wei al., 2021, Sanh et al., 2022, Ouyang al., 2022, inter alia]. The majority of this prior work, however, requires additional data such as aggre- gating dozens or hundreds of NLP datasets [Raffel et al., 2019, Aghajanyan et al., 2021, Aribandi et al., 2022]. writing additional templates of instructions (Wei et al., 2021, Sanh et al., 2022], or finetuning on human-labeled annotations [Ouyang al., 2022). UL2R does not require new data since simply re-uses the pre-training data, which makes it orthogonal to continued training methods that leverage large collections of NLP datasets. Adapting a pretrained language model with new
Spatial or Visual Reasoning Tasks The first cat- egory of tasks that U-PaLM does extremely well on are tasks that require some form of spatial or visual reasoning (c.g., navigate or geome In both of these tasks, U-PaLM 8B outperforms PaLM 540B. We postulate that this is due to the prefix language model architecture and additional PrefixLM training that U-PaLM undergoes. To give better illustration, consider the following examples from these tasks.
In the navigate task, an example is as follows: 'Turn right. Take step. Turn right. Take steps. Turn right. Take step. Turn right. Take 2 steps. Take 4 steps. and the task is a binary classification

---

task that determines if the agent returns the starting point.
Copiee
FLOPS (ZFLOPS)
298.7
558
2529.7
54.8
85.4
53.7
(-9.9%
In the geometric_ shapes task, the goal is to predict the shape given SVG path, e.g. given 'M
60.5
81.4
$188
81.3
52.3
84.1 (42.2%)
HellaSSaag O stee
79.7
79.7
60.8
79.7
634
84.1 (+0.8%6)
Manograndd O-dict
77.0
76.2
74.9
70.1
$1.1
82.6 (+1.5%)
Aug Corsnossense
50.5
40.7
50.3
78.2
53.7
84.9 (+1.4%
31,29 L 34,76 16 31,29 the model should predict triangle.
Table 4: Results on zero-shot comooosenmcccee reasoning.
Here, it worth noting that both tasks can be improved intuitively by having bidirectional attention and being trained using a PrefixLM like objective. This could explain why M-PaLM could outperform PaLM 540B even at 8B because it was given the right inductive bias. Commonsense and Knowledge Tasks A reasonable portion out of the 21 tasks require some form of commonsense or language-based knowledge in order to do well. noting that U-PaLM does not train on any new unique tokens (or new data) and therefore, has no access to no new 'knowledge' compared to vanilla PaLM. Hence, gains here are expected to be milder compared to tasks that rely more algorithmic other types of reasoning. However, we observe some relatively smaller gains in certain tasks (e.g., understanding fables or sicie_ddalog_sse sorg_sam_oorr coooddfffeeet)) Amongst the tasks in this category, one exception is the snarks task which involves detecting sarcasm in natural language. It is worth noting that the only out of 21 tasks where U-PaLM under- performs PaLM belongs this category (e.g.. logical_sequence and anglish_provent think this is reasonable since we not completely expect UL2R to always improve upon this category of tasks given that it does not actually process new data tokens.
bidirectionally reason with context (prefix) could be important in context reasoning tasks.
Multi-step Reasoning, Analogical Reasoning and Arithmetic tasks We observe that there are some performance improvements on analogical reasoning task (e.g., multi-step reasoning tasks (strategyqa) at 540B scale. However, unlike context reasoning tasks, the performance on these class of tasks tend to follow similar scaling patterns albeit with slightly better performance. For example, based on Figure 4, we note that strategyqa follows relatively similar scaling curves to PaLM.
or
worth
11.1 Zero-shot and Few-shot NLP
In this section, we evaluate our models on various well-established NLP tasks. These tasks test a spectrum of zero and few-shot abilities of U-PaLM.
11.111 Commonsense Reasoning
We conduct experiments four zero-shot commonsense reasoning benchmarks. Specifi- cally, following [Hoffmann et al., 2022], use BoolQ [Clark et al., 2019], PIQA [Bisk al., 2020J, HellaSWAG [Zellers et al., 2019] and Winogrande [Sakaguchi et al., 2019]. Aside from PaLM 62B and PaLM 540B which we use for direct comparisons with U-PaLM, we also compare with Chinchilla 70B [Hoffmann et al., 2022] and Gopher 280B [Rae et al., 2021]. Table 4 reports the results zero-shot commonsense reasoning.
We
Context Reasoning or Reading Comprehen- sion Tasks Some tasks require some understanding of context and then requires the language model to answer questions based on this context. An example of this is the vi taminc_ cercc_fcc__vrrreee wwwficiatiiiioooooo task which tries to determine the veracity of claim given external evidence (context). Another example is the inderstanding_far fablealleesse ls task where the goal is to determine the 'morale of the story' given context (passage or story). It is worth noting that U-PaL.M cxhibits emergence at 62B scale these two tasks even though the final 540B model performance is relatively similar. We postulate that this due to the architectural (and pretraining) advantage of PrefixLM which aids the model in performing much better even at smaller scales. Intuitively, being able
We show that U-PaLM 540B outperforms PaLM 540B on all four tasks with an average of ++14%) relative improvement and attains the best performance across all models.
11.1.2 Question A Anserring
and Reading Comprehension
We evaluate zero-shot and few-shot closed book question answering (CBQA) tasks [Kwiatkowski et al., 2019, Joshi al., 2017, Roberts al., 2020] along with the zero-shot Lambada reading comprehension task [Paperno et al., 2016]. Table 5 reports the results of our experiments. We compare with PaLM 62B, PaLM 540B, Chinchilla 70B and

---

628
628
106
2008
3428
S40B
Task/ Model
PaLM 540B
U-PaLM 540B
FLOPS (2PLAPS
205.7
206.7
555
304
252722
2339-7
TydiQA MGSM
52.9
54.6 (+3.2%)
ToMaDA
73.6
13.3
636
Natrrl Question Bicker
18.3
18.7
166
18.1
21.3
11.7:234553
45.9
49.9 (+8.7%)
Natool Questoos sloe
37.6
50.5
11.5
21.
36.0
0t1 011-11449)
Lankodod
79.7
74.5
QARC
583
(0.369
Table
Experiments on Multilingual (MGSM) [Shi et al., 2022] and TydiQA [Clark
GSM
Table Results on closed book QA and reading
et al., 2020a]
comprehension.
Task/ Model
Minerva 540B
PaLM 540B
U-PaLM 540B
Model
PaLM 540B
U-PaLM 540B
GSM8K
57.8
54,9
58.5 (+6.6%)
Task/PTokens
1828
780B
BBH
37.2
44.8
49.6 (+ (0.7%%
TriviaQA
73.4
81.4
73.3
75.6
82.0
QtrategyQA
61.9
76.4
76.6 (+0.2%)
NQA Ishot
23.2
25.6
29.3
24.4
28.1
30.7
CSQA
72.2
76.9
80.1 (+4.2%)
WebQA Ishot
21.6
19.9
22.6
21.0
21.7
23.4
BoolQ
82.4
85.6
88.0
85.8
88.2
88.8
Table 6: Experiment results on reasoning and chain-of-
ReCORD
91.5
92.7
92.9
91.5
92.6
93.0
thought reasoning experiments.
COPA
92.0
93.0
93.0
94.0
93.0
96.0
RTE
72.9
73.7
71.5
75.5
WIC
50.8
52.2
$8.0
62.2
WSC
88.1
86.7
89.1
87.0
88,1
87.4
Gopher 280B. Overall, on few-shot CBQA and reading comprehension, we observe that U-PaLM 540B outperforms PaLM 540B by +2.3% on average and up to +11.4% on few-shot natural questions. Meanwhile, the gain at 62B scale is also strong (i.e., +2.1% average).
CB
57.1
48.2
51.8
69.6
71.4
69.6
MultRC
76.7
81.1
83.5
78.4
81.7
83.8
Winogrande
89.4
88.3
90.1
87.9
89.7
88.3
Winograd
76.9
78.2
79.3
82.6
ANLIRI
44.3
49.4
48.4
50.3
50.6
55,3
ANLI
41.3
42.7
44.2
43.5
45.2
47.8
ANLI R3
43.8
42.8
45.7
46.7
49.3
57.0
PIQA
81.0
81.9
82.3
80.8
82.0
84.1
StoryCloze
82.7
84.6
83.7
84.2
87.0
HellaSwag
79.1
83.4
79.5
82.3
84.1
11.1.3 Reasoning
ArcF:
74.8
72.8
76.6
74.6
76.3
85,9
and Chain-of-thought Experiments
ArcC
48.0
46.9
53.0
48.6
50.4
60,3
RaceM
63.6
67.3
68.1
63.2
67.1
67.2
We conduct experiments reasoning and CoT and compare U-PaLM 540B with PaLM 540B and Minerva 540B. We use the GSM8K [Cobbe et al.. 2021], BBH [Suzgun et al., 2022], StrategyQA [Geva et al., 2021] and CommonsenseQA [Talmor et al., 2019] benchmarks. All tasks are run with thain-of-thought (CoT) prompting. Table reports results on reasoning and CoT benchmarks. U-PaLM 540B outperforms both PaLM 540B and Minverva 540B. Notably, the gains on GSM8K and BBH are relatively strong. This shows that U-PaLM does well on reasoning and is well-suited for of-thugt reasoning.
OpenhookQA
50.2
51.2
53.4
50.2
$1.2
53.6
RaceH
45.3
48.5
49.1
45.5
48.5
51.3
Lamhada
75.4
81.8
79.9
SqunVV2 Ishot
70.5
71.3
78.7
71.8
70.3
78.2
Average
62.7
63.8
66.5
64.1
66.2
69.4
Table 8: Results of PaLM vs U-PaL.M at different
FLOPs (# tokens) at 540B scale.
margin (+3.2% on TydiQA and +8.7% on MGSM).
11.2 Details of Scaling
Curves for Few-shot Experiments
11.1.4 Multilingual Few-shot Reasoning and Question Answering Tasks
We compute a mean aggregated score of the follow- ing tasks. We use 21 zero-shot rank classification tasks, i.e., BoolQ, Record, COPA, RTE, WiC. WSC, CB, MultiRC, Winograd, Winogrande, ANLI RI, ANLI ANLIR3, PIQA, StoryCloze, HellaSwag, Arc-E, Are-C, RaceM, RaceH, OpenbookQA. We use 5 one-shot generative tasks, i.e., TriviaQA, Nat- uralQuestions, WebQuestiion. and Lam- bada. All tasks use the accuracy (or exact match) metric except MultiRC which reports fla follow- ing [Brown et al., 2020]. In total, the aggregated met- ric is a mean over all 26 tasks. We list the scores that correspond to Figure 2's 540B scaling plot below.
We conduct expedimmee experiments on few-shot multilingual reasoning and question answering tasks. We use the MGSM (multilingual grade school math) benchmark proposed in [Shi et al., 2022]. For multilingual question answering, we use the well-cstablished TydiQA [Clark et al., 2020a] benchmark. In our experiments, both PaLM 540B and U-PaLM 540B uses chain-of-thought prompting [Wei et al., 2022b]. Table reports our results on MGSM and TydiQA. Our results show that U-PaLM outperform PaLM by considerable

---

11.3 Details of Vocab and Sentinel Tokens For U-PaLM, we had to train on span corruption or infilling task. We use the same setup as UL2 and T5 where we inject sentinel tokens, c.g., <extra id 0> into the masked positions. In T5, sentinel ids are added 100 additional vocab tokens at the end of the sentencepiece (vocab). In PaLM, since restart from an existing PaLM checkpoints. was quite cumbersome initialize 100 new embeddings in the vocab. Hence, we opt to simply use the last 100 subwords as sentinel tokens. Finally, we also use eos symbols in the vocab when training the model.
as possible to design new models for new tasks. U-PaLM is an instance of this type of research. Fi- nally, given that many language models are trained as causal language models, we believe that UL2R presents great opportunity for improving existing models with only small amount of compute.
11.4 Details of Prompt Templates
As stated in Section 3.2, BBES uses the default prompting and templates BIG-Bench and do not use chain-of-thougeeee prompting. For full BBH and MMLU results, we use the same of prompts as [Chung al., 2022], which we refer the reader to for more details. However, our 5-shot MMLU prompts do not use chain-of-thought, only directly stating the answer option, e.g. "Answer: (C)". Prompts for our zero-shot and few-shot NLP evaluations in Section 10.1 use the same basic templates as [Brown et al., 2020].
11.5 Additional Discussion
In this section, delve into some additional topics and discussions.
11.5.1 What about training from scratch?
We address the elephant in the room. There are mul- tiple perspectives to this question. The first that UL2R can be thought as a form of 'UL2 schedule' that sets single causal language model objective from 0 to N steps and then doing the UL. mixture from N to N + In this sense, if we wanted to train from scratch, this would require modifying the mixture have significantly more causal language modeling. The second perspective is that UL2R introduces a natural curriculum where the model spents a large fraction of training acquiring basic language modeling before moving on to tasks like infilling or learning how to leverage bidirectional receptive fields. Whether there is taxonomy or hierarchical of pretraining tasks is still an open question which we hope answer in future work. The third perspective simply the practical aspect of U-PaL.M. Training a PaL. 540B model from scratch is incredibly costly and we would like to reuse our existing models (or components) as much


Reviews:
[{"review_id": "ijsUclfqB4", "reviewer": "Reviewer_8nct", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments and for taking the time to read and review our paper!\n\nWe’re glad that you enjoyed the paper overall. Regarding the novelty of our methods, we might add that the strength of our approach is in its simplicity. Beyond just the small extra compute cost discussed in the paper, the practical cost is also small: one can leverage existing pretraining objectives and dramatically improve their model after-the fact with no architecture changes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is very well-written, and the experiments seem thorough and convincing. Perhaps most important, this paper has a lot of practical applicability given the cost and energy resources being sunk into training large language models around the world. Further still, the positive results described seem likely to spur a lot of continuation research into similar ways of increasing training efficiency with modified training objectives.", "reasons_to_reject": "This is quite a strong and thorough paper overall, but the main thing that keeps it from being transformative is a relative lack of novelty: it continues a line of work on large language models and mixture of denoisers, applying the latter for a specific use case (rapidly improving the performance on downstream tasks of language models trained on a strictly left-to-right objective).", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper describes a technique for significantly improving the performance of large language models on a variety of downstream NLP tasks by switching training objectives and continuing training for a relatively small amount of time with a mixture of denoisers approach (UL2, thoroughly described in earlier work).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "c0PFtNsf6m", "reviewer": "Reviewer_jpZa", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and giving your insightful comments!\n\nRe: The usage of PaLM. While it is true that we do not evaluate ULR2 on other models, we believe that our selection of PaLM as a baseline model is a strong one with a high chance of findings transferring over to other models as it 1) uses mostly standard mechanisms: a Transformer based decoder-only model pretrained on only causal language modeling over a large text corpus of 780B tokens, 2) spans the largest range of model sizes ever officially disclosed and uses the same recipe for each size, and 3) represents effectiveness of the approach at the very top band of performance, which is more challenging to improve upon. \n\nRe: Prompt templates. We are happy to include a subset of the prompt templates in the appendix for camera ready. (For example, for the most important tasks such as MMLU.)\n\nRe: Finetuning comparison against T5. Sorry for the confusion, this comparison is against T5.1.1 which does not use downstream tasks in the pretraining. We will update the text for the final version of the paper.\n\nThanks again for your feedback!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Simple yet effective approach to boost performance at a small compute cost relative to the initial training cost.\n2. Allows for a new prompting mechanism to seems to work well.", "reasons_to_reject": "1. Only uses PaLM which may not be sufficient to claim this approach works in general.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Instead of continuing to train causal language models with left-to-right, using a collection of multiple training objectives with small amounts of compute compared to the original training compute budget can further boost zero-shot/few-shot model performance on downstream task. The acquired improvements are equivalent to what would otherwise be done by continuing model training with the original objective with substantially more compute, therefore the approach is able to increase performance while also benefitting from massive computational savings.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1. It would help reproducibility if the prompt templates for each dataset was also released (or added to the appendix) this would allow the community to compare directly with this work should anyone would like to asses the approach in their own settings (different model/dataset/multiple model sizes).\n2. In section `3.3 Finetuning`, It is mentioned `For instance, PaLM 8B is generally outperformed by a T5 large model on the SuperGLUE dev average. We postulate that training PaLM on UL2 and span corruption tasks in complement to causal language modeling can ameliorate some of its flaws.`. It might be also due to (if it's the original T5 that is mentioned and not T5-v1.1) that SuperGLUE was included in the pretraining process so it could very well be due to that."}}, {"review_id": "vY49QudBRT", "reviewer": "Reviewer_zMDr", "summary": "", "questions": "1.If using UL2 directly to retrain PaLM from scratch, will it have better results?\n2.UL2R compared to PaLM's original training method, where is the extra calculation generated? How is the 0.1% in the article title calculated? This is not clear enough in line 230. And the article seems to be saying that the same results can be achieved with a much smaller cost, so why the emphasis on extra compute?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read and review our paper! We appreciate the detailed reading and will update the paper to improve the presentation as you mentioned for camera ready.\n\nRe: The usage of PaLM. While it is true that we do not evaluate ULR2 on other models, we believe that our selection of PaLM as a baseline model is a strong one with a high chance of findings transferring over to other models as it 1) uses mostly standard mechanisms: a Transformer based decoder-only model pretrained on only causal language modeling over a large text corpus of 780B tokens, 2) spans the largest range of model sizes ever officially disclosed and uses the same recipe for each size, and 3) represents effectiveness of the approach at the very top band of performance, which is more challenging to improve upon. \n\nRe: Using UL2 directly to retrain PaLM from scratch. We discuss this in Section 10.4.1 (Appendix). We leave this to be out of the scope of this paper as it is prohibitively expensive to do so. However, we might refer you to the original UL2 paper which presents detailed results training a 20B encoder-decoder model from scratch with UL2 [1]. Additionally, the PaLM 2 technical report also presents strong results using a UL2-like objective, albeit with less detail [2]. \n\nRe: The 0.1% calculation. PaLM was pretrained on 780B tokens. For PaLM 540B, UL2R trains 20K steps at batch 32, and sequence length 2048, amounting to pretraining for an additional 1.3B tokens, which constitutes 0.16% of the original 780B tokens pretrained. We’ll be sure to update the paper with more details for the final version.\n\nThank you again for your feedback!\n\n[1] https://arxiv.org/abs/2205.05131\n[2] https://arxiv.org/abs/2305.10403", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.This paper uses PaLM+UL2R to obtain a new U-PaLM model, which surruns PaLM in multiple tasks and achieves SOTA with low extra cost\n2.The paper further improves UL2 and proposes prefix optimization\n3.The experiment is sufficient and detailed analysis and discussion is made\n", "reasons_to_reject": "From the abstract of the article, it seems that the author intends to use UL2R as a general solution to the significant computational costs brought by scaling language models. However, the article only tested it on PaLM, which is not enough to support the universality of the UL2R approach. In fact, this article is more of an introduction to U-PaLM only", "questions_to_authors": "1.If using UL2 directly to retrain PaLM from scratch, will it have better results?\n2.UL2R compared to PaLM's original training method, where is the extra calculation generated? How is the 0.1% in the article title calculated? This is not clear enough in line 230. And the article seems to be saying that the same results can be achieved with a much smaller cost, so why the emphasis on extra compute?\n", "paper_topic_and_main_contributions": "In order to solve the significant computational costs brought by scaling language models, the authors use a slightly improved UL2 to continue training PaLM models from the middle to obtain U-PaLM models. Better results are achieved at a small cost. The authors have tested U-PaLM with different parameters and different stages on multiple tasks, and verified its excellent performance. The article also makes many additional analysis and discussion, and these experiences will enlighten the research in related fields.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1.The introduction of the article on innovation points is vague and repetitive, and it lacks a clear and concise chapter or paragraph to introduce it. Section 2 of this article gives a detailed overview of the U-PaLM, however much of this part duplicates previous work such as UL2, which makes it more difficult to figure out exactly what the \"R\" in UL2R means. In fact, the meaning of \"R\" in UL2R is only scattered in Introduction and section 2.4, which presents an additional obstacle to understanding the article.\n2.Does the bold OR in line 201 have any special meaning?\n"}}, {"review_id": "xxVtYNP7DY", "reviewer": "Reviewer_upoq", "summary": "", "questions": "A. What would happen if we apply ULR2 to pretraining phrase from scratch?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and for taking the time to read our paper and provide valuable feedback. We’re glad you enjoyed the paper! \n\nRe: Usage of PaLM. While it is true that we do not evaluate ULR2 on other models, we believe that our selection of PaLM as a baseline model is a strong one with a high chance of findings transferring over to other models as it 1) uses mostly standard mechanisms: a Transformer based decoder-only model pretrained on only causal language modeling over a large text corpus of 780B tokens, 2) spans the largest range of model sizes ever officially disclosed and uses the same recipe for each size, and 3) represents effectiveness of the approach at the very top band of performance, which is more challenging to improve upon. \n\nRe: Analysis of UL2. The design of UL2R was largely influenced by the findings and ablations in the original UL2 paper [1], which we refer to for more detailed analysis of how different tasks interact.\n\nRe: Cost of reproducibility. Although pretraining large models is expensive, applying UL2R is no more expensive than finetuning. For example, for PaLM 8B continued UL2R pretraining happens for 50k steps of 128 batch size. This makes the method relatively cheap to reproduce on other publicly available models.\n\nRe: Using UL2 directly to retrain PaLM from scratch. We discuss this in Section 10.4.1 (Appendix). We leave this to be out of the scope of this paper as it is prohibitively expensive to do so. However, we might refer you to the original UL2 paper which presents detailed results training a 20B encoder-decoder model from scratch with UL2 [1]. Additionally, the PaLM 2 technical report also presents strong results using a UL2-like objective, albeit with less detail [2]. \n\nThanks again for your thoughts and comments.\n\n[1] https://arxiv.org/abs/2205.05131\n[2] https://arxiv.org/abs/2305.10403", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. UL2R substantially improves PaLM's performance on downstream NLP tasks.\n\n2. UL2R significantly improves PaLM's scaling. At 540B scale, U-PaLM matches PaLM's performance with only half the FLOPs, saving up to 4.4 million TPU hours.\n\n3. UL2R enables emergent abilities at smaller scales, performing well on certain BigBench tasks at 62B rather than only at 540B.\n\n4. UL2R gives the model new prompting capabilities like bidirectional infilling, improving its flexibility.", "reasons_to_reject": "1. Only evaluated on a single model (PaLM) so unclear if gains generalize.\n\n2. Limited analysis on why UL2R provides benefits and how different tasks interact\n\n3. Hard to reproduced due to the huge cost.", "questions_to_authors": "A. What would happen if we apply ULR2 to pretraining phrase from scratch?", "paper_topic_and_main_contributions": "This paper proposes a method called UL2R to improve the scaling laws and performance of large language models like PaLM. \n\nThe key idea is to continue training an existing pretrained LM with a small amount of additional steps using the UL2 training objective. \n\nThis provides diversity compared to just training with causal LM, and allows the model to leverage bidirectional attention and infilling-style pretraining. \n\nExperiments show UL2R substantially improves PaLM's scaling curve and performance on NLP tasks, with only minimal extra compute. It also enables emergent abilities at smaller scales and practical benefits like infilling prompting.", "excitement": 5, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method called UL2R, which aims to improve the performance of large language models, specifically PaLM, on various downstream NLP tasks. The technique involves continuing the training of a pre-trained language model using a small amount of additional steps with the UL2 training objective. The method claims to achieve significant improvements in performance, scaling, and emergent abilities while providing new prompting capabilities.

Strengths:
* The paper is well-written, with thorough experiments and convincing results.
* UL2R significantly improves PaLM's performance on downstream NLP tasks and scaling, with minimal extra compute.
* The technique enables emergent abilities at smaller scales and provides new prompting capabilities, such as bidirectional infilling.
* The approach offers practical applicability and potential for further research in increasing training efficiency with modified training objectives.

Weaknesses:
* The paper's main limitation is the lack of novelty, as it continues a line of work on large language models and mixture of denoisers.
* The method is only evaluated on a single model (PaLM), which raises questions about the generalizability of the gains.
* Some reviewers found the presentation of innovation points to be vague and repetitive, making it harder to understand the exact contributions.

Overall, this paper is considered strong and potentially transformative, offering valuable insights and practical benefits in improving the performance of large language models. However, further evaluation on other models and a clearer presentation of innovation points would strengthen the claims and generalizability of the method.
Decision: Accept-Main