Forgetful causal masking makes causal language models better zero-shot learners
Authors: Hao Liu
Xinyang Geng
Lisa Lee
Igor Mordatch
Sergey Levine
Sharan Narang
Pieter Abbeel
Venue: Submitted to ICLR 2023 (2023)
Keywords: Language modeling
casual language model
few shot language models

Abstract: Large language models (LLM) trained using the next-token-prediction objective, such as GPT3 and PaLM, have revolutionized natural language processing in recent years by showing impressive zero-shot and few-shot capabilities across a wide range of tasks. In this work, we propose a simple technique that significantly boosts the performance of LLMs without adding computational cost. Our key observation is that, by performing the next token prediction task with randomly selected past tokens masked out, we can improve the quality of the learned representations for downstream language understanding tasks. We hypothesize that randomly masking past tokens prevents over-attending to recent tokens and encourages attention to tokens in the distant past. By randomly masking input tokens in the PaLM model, we show that we can significantly improve PaLM's zero-shot performance on the SuperGLUE benchmark from 55.7 to 59.2. Experimental results show that FCM also improves PaLM's zero- and few-shot performance on a diverse suite of tasks, including commonsense reasoning, natural language inference and cloze completion. Moreover, we show that our technique also helps representation learning, significantly improving PaLM's finetuning results on SuperGLUE.


Under review as a conference paper at ICLR 2023
FCM: FORGETFUL CAUSAL MASKING MAKES CAUSAL LANGUAGE MODELS BETTER ZERO-SHOT LEARNERS
Anonymous authors Paper under double-blind review
ABSTRACT
Large language models (LLM) trained using the next-token-prediction objective. such as GPT3 and PaLM, have revolutionized natural language processing in re- cent years by showing impressive zero-shot and few-shot capabilities across a wide range of tasks. In this work, we propose simple technique that signifi- cantly boosts the performance key observation is that, by performing the next token prediction task with ran- domly selected past tokens masked out, we can improve the quality the learned representations for downstream language understanding tasks. We hypothesize that randomly masking past tokens prevents over-attending to recent tokens and encourages attention tokens in the distant past. By randomly masking input tokens in the PaLM model, we show that we can significantly improve 1B and 8B PaLM's zero-shot performance on the SuperGLUE benchmark from 55.7 59.2 and from 61.6 to 64.0, respectively. Our largest 8B model matches the score of PaLM with an average score of 64, despite the fact that PaLM is trained on a much larger dataset (780B tokens) of high- high-quality conversation and webpage data, while ours is trained on the smaller C4 dataset (180B tokens). Experimental re- sults show that our method also improves PaLM's 's zero and few-shot performance on a diverse suite of tasks, including commonsense reasoning, natural language inference and cloze completion. Moreover, we show that our technique also helps representation learning, significantly improving PaLM's finetuning results.
LLMs without adding computational cost. Our
INTRODUCTION
Language model (LM) pre-training has substantially advanced the state-of-the-arr across a variety of natural language processing tasks (Peters et al. 2018 Devlin et al.) 2018 Brown et al. 2020 Chowdhery et al. (2022) and related fields including image generation, reasoning, and code gener-
ation (Alayrac et al. 2022 Lewkowycz et al. 2022 Saharia et al.) 2022 Chen et al. 202 Prior work on pre-training have focused on mixing different choices of architecture (e.g., encoder-only, decoder-only, encoder-decoder) with different objective functions (e.g.. masking or causal lan- guage modeling). For example, masked encoder-only models such as BERT (Devlin et al.) (018)) and RoBERTa (Liu et al., 2019) excel in discriminative finetuning tasks such classification. Sim- ilarly, masked encoder-decoder models such as BART (Lewis et af.] 2019) and T5 (Roberts et al.) 2019) perform well on both discriminative and generative finetuning. While masked language mod- cling is effective for finetuning and removes the need for task-specific architectures, its major limi- tation is that there still a need for task-specific datasets and task-specific finetuning. On the other hand, decoder-only causal language models remove such limitations. In fact, they are capable of zero-shot and few-shot adaptation without the need for finetuning, by simply prompting the model with appropriate strings to control the generated outputs, as shown in GPT3 (Brown et al. 2020 and PaL.M (Chowdhery et al. 2022).
Driven by its impressive zhro-shot and few-shot abilities, there has been more work on scaling causal decoder-only architectures (Zhang et al. 2022 Black et al., acl: Brown et al. 2020 Chowdhery et al.) 2022 compared to encoder-based architectures, and there has been significant interests in studying such models in various contexts (Hoffmann et al. 2022 Wei et al. 2022b; Li & Liang 2021 Ahn et al. 2022 Chen et al., 2021). However, such decoder-only models are still limited by their imperfect zero-shot and few-shot adaptation compared to human performance, and their relatively inferior finetuning performance compared masked language modeling.

---

Under review as a conference paper at ICLR 2023
PALl Ceeter FCM Dennt
PALM Tohit FCM Tothol
Necrae 120 soore
60.5 f2.3
50.5
47.5
47.6
30.6 31.6
Clock comsiatinn
Common OE
Weepet-bll (RIoOinng
Mikard knyagg?
Octe CoNgBAoo
SeswbLue
Poll S-stor
FCM Gishot
S0.2 $2.5
Fall Fiaetuee on Firknn
S7.1
$4.5
07.2
47.8 47.7
792.5
712
1/A7
32
52295
56-20
condidion
SupOOLLC MLU
Nacunal
Avwnnge St GLUE Score
61.2
55.7
51.9
92.5
Pa 128M-
FOM 178M-
Pal I7s
FCM IR-
PAIM" ER- 7803
Pvl M.8
FEN RB-
CPT-3 17B-
TI-000 116-
012 2IR
GPTS 1799-
STARF 200R HQ
100m
300R HX2
1000R
Model Data
Figure 1: FCM outperforms PaLM in zero- and few-shot as well as finetuning tasks. Top & middle. Task average performance grouped by categories. The model size is 1B. We report the averaged scores each category. Scores are averaged over evaluation random seeds. Bottom. SuperGLUE zero-shot performance by different model size and dataset size. PaLM 8B-780B HQ denotes the published results of 8B model trained on 780B tokens from high quality datasets, PaLM 8B-180B denotes the same setup but with 180B tokens from C4 dataset, and FCM 8B-180B denote the same 8B model trained on 180B tokens from C4 dataset using FCM as objective.
To address the above challenges, prior work have proposed to combine masked modeling with causal language modeling (Dong et al.) 2019 Wang al. 2022 Tay al., [2022] Du et al. [022] bring the benefit of masked modeling to causal language models while retaining their zero-shot ability.
However, such approaches typically introduce extra computation and parameters or require using sophisticated attention masking strategy which hinders practical usages (Yang et al. 2019 Tay et al.) 2022]. Moreover, they typically train encoder-decoder models which are not naturally suitable for zero- and few-shot inference tasks compared with decoder-only causal language models and are still outperformed by causal language models (Sanh et al., 2022 Brown et al. 2020) Chowdhery et al. 2022). In order to further improve causal language models few-shot abilities. some works proposed better prompt engineering methods (Liu et al. 2021 Lester et al. |002 Ling et al. [2017] | Wei et al.]
20226 [Li & Liang 2021) or better finetuning methods (Mishra et al. 2022 Wei et al. 2022a Sanh et al. 2022). baseet methods are sensitive to design (Lester et al.) 2021] Liu et al.) 2021),
approaches typically require huge amount of supervision work with as shown in Sanh et al. (2022). ). In addition, such methods can only improve pre-trained model and are unable to improve pre-training.
while finetuning-basee
In this work, we propose pre-training approach that does not incur any extra computation cost or parameters, to improve few-shot and zero-shot performance, as well representation learning of causal language models. Our key observation is that, by performing next token prediction task with randomly selected past tokens masked out, we can improve the quality of the learned representations for downstream language understanding tasks. Our method, Forgetful Causal Masking (FCM), can be efficiently implemented by randomly masking input tokens in the causal language model. Ap- plying our method to PaLM (Chowdhery al. 2022), a state-of-the-art causal language model, we see significant improvement on the SuperGLUE (Sarlin et al. 2020) benchmark: our method signif- icantly improves the 1B m-dodel--size PaLM's zero-shot performance from 55.7 7 to 59.2 and improves the SB-model-size PaLM's ' zero-shot performance from 61.6 to 64.0. We also conduct extensive evaluation on the commonsense reasoning benchmark PIQA (Bisk et al. 2019). ARC (aadav al.
2

---

Under review as a conference paper at ICLR 2023
2019) and OpenBookQA (Mhhyylvv et al., 2018); the Winograd-sty -tylle tasks Winograd (Sakaguchi et al. 2020) and WinoGrande Kooijaan et al. 2020 the natural language inference (NLI) bench- mark ANLI (Nie al. 2 and cloze completion tasks StoryCloze (Mostafazadeh et al., 2016) and LAMBADA (Paperno et al. 2016); and find that our method improves the zero-shot and few- shot performance of PaL.M all of the diverse suite of tasks. In addition, FCM improves repre- sentation learning, as shown in our SuperGLUE finetuning experimental results, where our method significantly improves 1B parameter PaLM model's finetuneing performance from 67.0 68.7, and our method significantly improves 8B parameters PaLM model's finetuning performance on all 8 SuperGLUE tasks, improving the score from 80.7 83.1.
Contributions, We highlight the contributions of our paper below:
We present FCM, a simple and scalable pre-training methodology for causal language mod- eling. We provide the empirical evaluation of FCM on suite of few-shot and finetuning benchmarks.
We show that FCM is highly effective at improving zero-shot and few-shot learning re- sults, outperforms strong baselines including PaLM and UL2, improving the average Su- perGLUE score of 8 billion parameters PaLM from 61.6 to 64.0, and improving PaLM on a wide range of 19 NLP tasks.
In addition to few-shot learning, we demonstrate that FCM significantly helps with fine- tuning to downstream tasks, improving the performance of 8 billion parameters PaLM on 8 out of 8 SuperGLUE tasks and the average SuperGLUE score from 80.7 to 83.1.
We demonstrate that FCM is scalable it consistently outperforms PaLM with various model sizes, from 128 million parameters to billion and 8 billion.
2 RELATED WORK
Masking strategies and pre-training objectives. Many self-supervised pre-training techniques have been proposed to leverage the vast availability of unsupervised data. Different architectures typically leverage different objectives. Decoder-only models are typically trained with causal lan- guage model objectives to mimic auto-regressive generation (Brown et al. 2020) and is found to be effective in cross- modoaityy learning (Alayrac et al. 2022] Yu al. 2022). Related to our masking out tokens, scheduled sampling (Bengio et al. (0155) applied replacing tokens with model predicted
tokens and is shown to improve training recurrent neural networks. Autoencoding denoising ob- jectives have been used to learn bidirectional contextualized encoder for natural language under- standing (Devlin et al. 2018 Liu et al.) 2019 Yang et al., 2019). For encoder-decoder models, BART (Lewis et al. 2019) conducts NLU tasks by feeding the same input into the encoder and decoder, and taking the final hidden states of the decoder. Raffel et al. (2020) explored many objec- tives of pre-training and found that span-corruption works best with encoder-decode model. Other work explores multi-task pre-training using supervised data (Aribandi et al.. 2021] (anh et al.) 2022]
Wang et al. 2022). To study the impact of different objectives on zero-shot generalization, Wang et al. (2022) conducts a systematic study of different architectures combined with three different pre- pre-training objectives, and found causal language modeling to be effective at zero-shot learning.
Combining causal and masked language modeling. There has been work explore training model with multiple objectives to combine causal and masked language modeling under the masked lan- guage modeling objective with different attention masks (Dong et al. 2019 Bao et al. 2020). Later work proposes to use blank infilling (Raffel et al. 2020 to randomly blank out continuous spans of tokens from the input text and train the model to sequentially reconstruct the spans (Du et al., 2022]. XLNet (Yang et al., 2019) modifies the attention mask in a standard transformer to enable token generation in any permutation of tokens. XLNet uses two-stream self-attention mechanism, instead of the right-shift, to avoid information leakage in Transformers, but doubles the time cost of pre-training. UL2 (Tay et al. 2022) further proposes to train language model using mixture of denoisers to combines diverse pre-training paradigms together. Other work explored masking some spans that predicted at the end of the sequence for bidirectional models (Artetxe et al.) (022) or left-to-right autoregressive models (Aghajanyan et al. 2022] Zhu et al.] 2019: Donahue et al. 2020 Fried al. 2022). Notably, Bavarian al. (2222) explores moving text in the middle to the
3

---

Under review as a conference paper at ICLR 2023
end and predict it autoregressively. Related to our layer-wise attention masking, word masking has been explored in the context of recurrent neural networks (Dai Le 2015: Bowman et al. 2015 Xie et al. 2017). Different from prior work, focus efficiently improving causal transformer model with masking language modeling, our method does not require complex implementations to change input or output prediction, making it simple to implement and our method does not add extra computation or parameters.
3 METHOD
Unmasked token
Masked token
tok1
tok2
tok3
tok4
tok5
tok6
Predict
Causal Transformer Model
emb1
emb2
emb3
emb4
emb5
emb6
Embed + Pos
How
Large
Language
Models
Transform
Science
Figure 2: Illustrations of FCM. Given a causal language model, each token's prediction is condi- tioned on embeddings that are not masked. The loss is applied to each token in the sequence. In this example, when predicting the token of the word "models", the embeddings of the words "large" and "language" are removed from the input sequence. The model is asked predict all tokens autoregressively.
3.1 PRE-TRAINING OBJECTIVE
FCM uses standard causal, decoder-only Transformer model architecture (Vaswani et al. 2017
Le., each timestep can only attend to itself and past timesteps. We illustrate FCM in Figure 2 Given an input text = [x1,-... In.. the standard causal language modeling objective is defined to
maximize the log likelihood of æ autoregressively:
logr(x) log II
p (211 02, :;....
i=1
= log II p(x,(xe)) = log II p(zi (=
(1)
In FCM, randomly sample mask ratio from [0, where n E [0, 1] is fixed maximum mask ratio, we use = 0.15 throughout the experiments unless otherwise mentioned. The model is asked to predict each token æ; and can only attend to tokens in x that are not sampled. Concretely, the FCM objective is given by:
log px) = log II [p > ni ,31;=bb)
(2)
1). This can be efficiently implemented by combining it with causal attention mask. While applying random masking to the token sequence, we always exclude the special BOS (' ('beginning of sentence') token at the beginning of each sequence, that the model is aware of the beginning sentence. Moreover, keeping the BOS token unmasked helps with training sta- bility because ensures that there is at least token unmasked without changing the semantic
where m, U(00
4

---

Under review as a conference paper at ICLR 2023
meaning of the sequence. For example, when predicting token 22 for small t, it is possible that all tokens [Z.,.... 1t- are masked, which can cause instability in the training loss. We found that this technique enables to train arbitrary high mask ratios without incurring instability.
3.2 MODEL ARCHITECTURE
We use the same model and architecture as PaL.M (Chowdhery et al. 2022), including the modified activation (Shazeer 2020). multi-query attention [Shazeer 2019), parallel layers (Wang & Komat- suzaki 2021) and RoPE embeddings (Su et al. 2021) I described therein, with the exception that we SentencePiice (Kudo & Richardson (208)) vocabulary with 32K tokens from C4 (Raffel al. 2020). To study the dependence of FCM on model size, we train different sizes of the model, ranging over three orders of magnitude from 125 million parameters, to billion parameters, and 8 billion parameters (see Table
Model
Layers
# of heads
dmodde
Batch size
Seq len
PaLM, FCM 128M
8
4
1024
1024
1024
PaLM, FCM IB
16
8
2048
1024
1024
PaLM, FCM
16
4096
1024
1024
Table I: Architecture details of different sized models. We list the number of layers, dmodel- the number of attention heads and attention head size, training batch size, and sequence length. The feed-forward size dg is always x dmmodel and attention head size is always 256.
3.3 METHOD DISCUSSION
In this section, discuss the connections and differences between FCM and other pre-training models. We are mainly concerned with how they improve few-shot, zero-shot, and finetuning per- formance.
T5 Raffel et al. 2020) and UL.2 (Tay et al. 2022) propose to train encoder-decoder or prefix lan- guage model architecture using the ppan-corruption objective. T5 and UL2 always predict spans in a fixed left-to-right order, and are therefore related to FCM in that our method also predicts masked tokens from left right. However, FCM is an autoregressive model without an encoder that encodes full context information, so in principle, FCM can be combined together e.g. with a prefix language model. Empirically. FCM outperforms T5 and UL. on NLU tasks with smaller models (1B vs 8B) and fewer number of tokens (180B 1000B).
XLNNet YYngg et al. 2019) is also pre-trained with autoregressive objectives, but there are important distinctions between FCM and XLNet. FCM does not need permutation of the input sequence and designing two-stream self-attention mechanism avoid the information leak within Transformer, which doubles the time cost of pre-training. Our method is much simpler and more scalable.
GLM (Du et al. 2022) proposes to extend autoregressive modeling with bidirectional context. They achieve this by selecting spans and move them to the end of sequence, then unselected tokens and past spans use non-causal attention and tokens within each span use causal attention, similar to PrefixLM and U1.2 (Tay et al. 2022
Raffel et al.) 2020).
UniLM (Dong 2019 Bao et al. 2020) combines different training objectives together by using different self-attention masks to control the access to context for each token. Similar to BERT and T5, UniLM trained with an autoencoding objective with masked spans replaced by mask tokens. This introduces gap between pre-training and downstream tasks, since there in downstream tasks. Moreover, the model needs to be finetuned for natural language generation tasks (e.g., summarization). In contrast, FCM focuses improving causal language models and outpeforms strong baselines such UL2 and PaLM on zero- and few-shot SuperGLUE benchmark.
mask tokens
In summary, method is simpler and focuses autoregressive causal language models. Our method is easy to implement and does not introduce extra computation or parameters, and as exper- imental evaluations in Section show, FCM is scalable and achieves superior results than baselines.
5

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Training datasets We use C4 dataset to pre-train baselines and our model (Raffel et al. 2020). It is colossal, cleaned version of Common Crawl's web crawl corpu TI and consists of about 180 billion tokens using sentencepiece tokenizer (Kudo & Richardson 2018). Note that GPT-3 and PaLM use significantly larger pre-training datasets, e.g., the PaLM pre-training dataset consists of a high-quality corpus of 780 billion tokens that is mixture of filtered webpages, social media con- versations, and more. However, these datasets are not publicly available and training on it requires tremendous compute resources. C4 is significantly smaller, which also reduces the compute cost of training the large models.
Training and inference. Our training optimizer follows PaLM, and use the Adafactor optimizer (Shazeer & Stern, 2018) which scales the learning rate by the root-mean-square of the parameter matrix. We use learning rate of 0.01 for the first 10,000 steps, which is then decayed at a rate of 1/Vk, where & the step number. We train with momentum of 31 = 0.9. The second-order moment interpolation value is computed as 3g 1.0 - k k-0.8, where k is the step number. Following typical large Transformer models training as in PaLM and GPT-3, models are trained without dropout, and dropout of 0.1 is used for finetuning. Our training and inference codebase is based on JAX and T5X, and all models are trained on TPU v4 Pods. The few-shot and zero-shot results are averaged over three evaluation random seeds. For results of baselines, we choose and report the best published results to compare against FCM. We use exactly the same batch size, learning rate, and training hyperparameters for PaLM and FCM. More details hyperparameters, compute infrastructure, and training time are provided in Appendix [
Evaluation tasks and metries. We consider the following tasks and categorize them according to their focused evaluation properties:
Cloze and Completion lasks: LAMBADA (Paperno et al. 2016) consists of word prediction tasks that test the understanding of narrative passages. Story Cloze (Mostafazadeh et al. 2016) evaluates story understanding and script understanding, by requiring system to choose the correct ending to a four-sentence story.
Commonsenss Reasoning: PIQA (Bisk et al. 2019) is dataset designed for physical common- sense reasoning to investigate the physical knowledge of language models. ARC (Yadav et al.) 2019) is a multiple-choice ams from grades 3-9. There are two partitioned datasets ARC-e (easy) and ARC-c (challenge), where the latter partition contains the more difficult questions that require reasoning. Open- BookQA (Mihaylov et al.) 2018) is designed to test understanding of both the topic (e.g., salient facts) and the language
vvestion-answering dataset, containing questions from science ex-
expressed in. This dataset contains questions that require multi-step reasoning, commonsense knowledge, and rich text comprehension.
Winograd-style tasks: In the Winograd schema challenge, schema is a pair of sentences that differ in only one or two words and that contain an ambiguity that resolved in opposite ways in the two sentences. Winograd tasks (Kocijan et al. 2020) require world knowledge and reason- ing to be solved. WinoGrande (Sakaguchi et al. 2020) is a large-scale dataset of 44k problems, and requires commonsense reasoning choose the correct option for a given sentence.
Natural Language Understanding (NLU):: SuperGLUE (Sarlin et al. 2020) consists of 8 chal- lenging NLU tasks, including word sense disambiguation, natural language inference, corefer- ence resolution, and gueswinn-ansserrin..
2019 is collected via an adversarial human-and aouu--nnd-mddddd in-the-loop procedure and selected to difficult to state-of- the-art models.
Natural Language Inference (NLI): Adversarial NLI (ANIL) Nie et al.
Baselines. The main baseline we compare with is PaLM (Chowdhery et al. 2022), since it is one of ttate-of-the-arts on a wide range of NLP benchmarks.
https::/ccmmonee .org

---

Under review as a conference paper at ICLR 2023
4.2
MAIN RESULTS
4.2.1 FEW-SHOT PERFORMANCE
We compare FCM with PaLM on few-shot and zero-shot performance in wide range of bench- marks. Table 2 includes the results for the FCM and the PaLM 1B and 8B models. The results averaged over task categories are presented in Figure Following prior work, we only consider single checkpoint results from pre-trained language models.
FCM outperforms PaLM on 17 out of 19 tasks in the zero-shot setting, 15 out of 19 tasks in the one- shot setting, and 15 out of 19 tasks the few-shot setting. On the SuperGLUE (Sarlin et al.) 2020) benchmark, our method significantly improves the IB-model-size PaLM's zero-shot performance from 55.7 to 59.2 and improves the 8B-model-size PaLM's zero-shot performance from 61.6 6 to 64.0. Consider that PaLM is well-tuned in many aspects, including the pre-training dataset, training strategy, and the number of tokens observed. The significantly better results of FCM shows that the training objective can also play a crucial role in the model performance.
Table 2: Results obtained by the FCM IB and 8B model across NLP benchmarks. We use the same setup as in Brown et al. 2020 : Chowdhery et al. (2022). including the splits for each task.
Zeoo-shot
One-shot
Few-shot
PaLM
FCM
PaLM
FCM
PaLM
FCM
PaLM
FCM
PaLM
FCM
PaLM
FCM
Task
1B
1B
8B
8B
1B
1B
8B
8B
1B
1B
8B
8B
Lambada (EM)
42.4
43.5
58.0
59.1
48.9
49.5
65.8
66.5
48.2
49.7
66.1
67.5
StoryCloze
68.8
68.2
75.0
75.6
67.3
66.9
75.0
75.7
65.9
66.7
75.8
76.2
PIQA
72.0
72.1
77.0
77.4
71.0
71.6
75.5
76.5
72.0
71.6
77.1
77.3
ARC-c
46.2
45.6
55.3
57.1
48.0
45.9
60.1
60.2
50.2
48.2
64.0
64.4
ARC-c
25.8
27.7
33.8
33.0
26.3
27.2
34,0
35.0
26.5
28.1
35.5
36.5
OpenbookQA
45.8
46.4
48.2
49.2
45.0
43.2
47.0
48.4
42.6
43.6
49.0
49.5
Winograd
67.0
70.0
78.5
80.6
67.0
67.4
79.5
81.7
64.8
70.0
79.5
81.2
Winogrande
54,0
54.5
60.0
61.9
54.0
55.8
60.5
62.1
53.6
55,0
61.0
62.3
BoolQ
45.9
56.0
52.0
62.1
48.3
52.6
53.7
59.6
46.8
49.0
57.7
Copa
72.0
74.0
82.0
84.0
72.0
73.0
80.0
83.0
70.0
72.0
82.0
85.0
RTE
50.9
53.8
53.4
48.9
53.1
54.5
55.2
47.3
53.1
45.1
53.1
48.4
WIC
51.4
52.6
78.3
79.1
47.8
46.9
79.0
86.8
48.9
50.1
77.9
87.9
Multire (Fla)
35.2
40.6
40.4
54.1
57.1
57.2
49.8
56.5
57.2
42.5
WSC
65.3
70.2
78.3
79.1
66.7
71.2
79.0
86.8
66.7
70.2
77.9
87.9
ReCoRD
75.8
76.3
85.5
85.0
75.8
76.4
85.5
84.9
74.9
75.0
84.6
83.9
CB
48.2
50.0
82.0
84.0
44.6
44,8
42.9
51.5
42.3
48.2
46.4
50.0
ANLI RI
33.3
33.5
32.9
34.3
31.3
33.0
32.7
33.5
30.5
32.5
31.1
32.9
ANLI R2
32.8
34.2
33.3
34.1
30.5
30.6
30.6
33.7
32.5
33.4
31.7
33.8
ANLI R3
33.3
33.6
33.0
33.9
30.0
31.2
31.7
33.8
32.8
34.2
32.9
35.1
4.2.2
FINETUNING PERFORMANCE
We conduct finetuning experiments on the SuperGLUE benchmark to compare PaLM and FCM. Following PaLM experimental settings, models are finetuned with 10- learning rate using the Adafactor optimizer. To reduce computation time, we use batch size 512 instead of the original batch size 32 in PaLM. The models are finetuned for 20K steps.
Table reports the validation results on finetuning on task proporrrpttt mixture of SuperGLUE tasks. On SuperGI.UE, we compare with otate-of-the-art models such as T5 11B (Raffel et al. 2020) and UI.2 TTa et al. 2022). as well as PaL.M (Chowdhery et al. 2022) and show that FCM obtains significantly better performance than PaLM. All models are trained on C4 dataset, T5 IIB and UL2 are trained 1000B tokens, the rest of models are trained on 180B tokens. It is worth noting that both top performing models on SuperGLUE are encoder-decoder models that are trained using the span-corruption objective. It has been shown that such architccture generally outper- forms autoregressive decoder-only models on classification task finetuning, when training cost is equalized (Raffel et al. 2020). These results demonstrate that FCM help bridge the gap. FCM

---

Under review as a conference paper at ICLR 2023
IB outperforms PaLM 1B significantly 4 out of 8 SuperGLUE tasks, and FCM 8B significantly outperforms PaLM 8B on all 8 SuperGLUE tasks, improving the score from 80.7 to 83.1.
Table 3: Finetuning results on SuperGLUE dev set. We compare with T5-1 (Rafiel et al. 2020 UL2 (Tay et al. 2022) and PaLM (Chowdhery al., 2022). Scores reported are the peak validation scores per task following the setup of 13. All models are trained on the same 180B tokens except
that UL2 20B and T5 11B are trained on 1000B tokens.
Model
BoolQ
CB
CoPA
MultiRC
Record
RTE
WiC
WSC
Avg
Masked language model
T5 11B
90.8
87.4
93.8
93.9
77.3
89.9
UL2 20B
90.8
98.7
99.0
88.4
93.7
92.1
77.3
98.1
90.7
T5 1.4B
83.7
92.9
85.9
82.7
69.6
78
80.8
80
81.7
Causal language model
PaLM 1B
75.0
78.6
58.2
62.9
63.3
60.0
66.4
71.2
67.0
FCM IB
73.5
82,0
58.0
63.0
65.1
61.0
66.5
80.8
68.7
PaLM 8B
83.7
94.6
81
71.2
80
75.2
80.1
80.7
FCM 8B
84.8
96.4
81
82.1
73.7
86
76.2
85
83.1
4.2.3 MODEL SCALABILITY
To demonstrate the scalability of FCM, we further evaluate FCM with different model sizes in Table 4 We consider both smaller model with 128 millions parameters, and scaled-up model with billion parameters. All models are trained for 180 billion tokens, which is equivalent to about epoch on the C4 dataset. Although 8 billion model size is relatively small compared typical large language models (Chowdhery et al.) 2022 Brown et al.) 2020), these results serve as a proof- of-concept of FCM's effectiveness and sca scalability for larger model sizes. We leave further scaled-up experiments as promising future work. We compare with methods that use more high quality datasets including the official PaLM* and GPT-3. PaLM is trained on 780 billion tokens and GPT-3 is trained on 300 billion tokens. Other baselines include T5 and UL2 which are trained for 1000 billion tokens and ST-MoE which uses 500 billion tokens. The results show that FCM works with both smaller and larger models, ranging from 128 million parameters to 8 billion parameters. Surprisingly, the largest FCM model matches the score of PaIM with an average score of 64, despite the fact that PaLM* is trained on much larger dataset (780B tokens) of high-quality conversation and webpage data, while FCM is trained on the smaller C4 dataset (180B tokens). We further compare FCM with official PaLM 8B model on one-shot and few-shot experiments. Table 5 shows the results, FCM matches PaLM in most tasks, showing the promising capabilities of FCM.
Fiekko Fitura
100
25
$3.5
G4T
04.2
30
52
80
Figure 3: Fewshot and finetuning results on SuperGLUE. We compare PaLM* which is trained on 780B high quality data, PaLM trained on C4, and FCM. On zero-shot and one-shot learning, FCM matches or outperforms PaLM* and outperforms PaLM in few-shot and finetuning significantly.
4.3 ABLATION STUDY
FCM works best with random ratio. We evaluate the impact mask ratio on FCM using Su- perGLUE zero-shot benchmark. Table o presents the results of FCM with different mask ratios. Among them, sampling random ratio between [0.0, 0.15] performs significantly better than other choices. Sampling mask ratios from 0.0 to 0.1 or 0.15 perform generally better than using fixed
8

---

Under review as a conference paper at ICLR 2023
Table 4: Comparisons on SuperGLUE zero-shot benchmark with different model sizes. PaLM
denotes the published results of PaLM.
Model
BoolQ
CB
COPA
MultiRC
ReCORD
RTE
WiC
WSC
Avg
Methods that more high quality data
GPT-3 175B
60.5
46.4
91
72.9
90.2
63.5
0
65.4
61.2
PaLM 540B
51.8
93
83.5
92.9
72.9
59.1
89.1
78.8
GPT-3 13B
66.2
19.6
84
71.4
89
62.8
64.4
57.2
PaLM 8B
68.3
41.1
86
47.5
87.8
54.2
47
78.9
63.9
Methods that use 1000B tokens from C4
ST-MoE 269B
41.1
56
30.3
50
52.7
T5-XXL 11B
44.3
37.5
70
23
8..88
48.8
50.9
59.3
52.5
UL2 20B
63.1
41.1
85
36.2
88.1
60.7
49.8
79.9
63
Methods that use 180B tokens from C4
PaLM 128M
8.8
63
54.3
62.4
53.1
56.5
50.7
FCM 128M
55.7
8.9
69
55.1
62.1
56.3
52.1
56.1
51.9
PaLM IB
45.9
48.2
72
35.2
75.8
50.9
51.6
65.3
55.6
FCM IB
56
50
74
40.6
76.3
53.8
52.4
70.2
59.2
PaLM 8B
52
50
82
40.4
85.5
53.4
51.3
78.3
61.6
FCM 8B
62.1
48.2
84
54.1
85
48
51.1
79.1
64.0
Table 5: Comparison between FCM and PaLM on SuperGLUE zero-shot and few-shot benchmark tasks. PaLM denotes published results obtained by training on more high quality data. The model size is 8B.
shots
Model
BoolQ
COPA
MultiRC
ReCORD
RTE
WiC
WSC
Avg
PaLM
41.1
47.5
54.2
78.9
63.9
Zero-shot
PaLM
50
40.4
53.4
51.3
78.3
61.6
FCM 8B
62,1
48.2
84
54.1
85
48
51.1
79.1
64
PaLM 8B
64.7
41.1
82
50.6
87.8
57.8
47.3
81.4
64.1
One-shot
PaLM 8B
53.7
42.9
80
49.8
85,5
55.2
51.5
79
62.2
FCM 8B
59.6
51.5
83
56.5
84.9
47.3
46.9
86.8
64.6
PaL. 8B
68.9
57.1
82
41.1
88
56.7
52.4
83.2
66.2
Few-shot
PaLM
46.4
42.5
84.6
53.1
50.5
77.9
60.8
FCM 8B
57.7
50
85
46.5
83.9
48 4
49.5
87.9
63.6
mask ratio 0.1 or 0.15, indicating that fixed mask ratios could potentially introduce pre-train pre-training and inference gap, and sampling random mask ratio is a simple way to alleviate it.
Using mask tokens instead attention mask. Alternative to FCM, natural way of preventing future tokens from attending to past tokens is replacing tokens with special [mask] token. Using
mask tokens widely adapted in masked language modeling (Devlin et 2018 Liu al. 2019 and combining mask token with causal language modeling can be considered as special case of UniLM Dong al.) 2019). We perform an ablation study comparing FCM with mask token, and present the results in Table 7 Using mask tokens lead to performance degradation in zero- and few-shot experiments, and about the same results on finetuning experiments. We hypothesis that performance drop is due to the train and inference gap caused by introducing the [mask] ] token, which can negatively impact zero- and few-shot performance because the model is not finetuned to remove such gap.
Comparison with dropout. FCM random masking can be seen as a special type of dropout (Sri- vastava et al. (2014) applied only the input sequence layer wisely by using attention masking. We note that general dropout and FCM are complementary that they can be combined together. To compare random masking dropout, compare three models in Table (1) Pal.M, (2) PaLM

---

Under review as a conference paper at ICLR 2023
Table 6: Ablation of mask ratio SuperGI.UE. Comparisons SuperGI.UE zero-shot and shot benchmark between fixed mask ratio and random mask ratios using FCM. The model size is IB. FCM [x, y] denotes mask ratio randomly sampled between and y.
Model
BoolQ
CB
CОРА
MultiRC
ReCORD
RTE
WiC
WSC
Avg
zero-shot
PaLM
45.9
48.2
72.4
35.2
75.8
50.9
51.6
65.3
55.7
FCM [0.1, 0.1]
56.5
51.6
73.5
32.9
76.3
55.6
52
67.1
58.2
FCM [0.15,
48.2
75.5
22.6
75.9
52.7
49.8
66.1
55.6
FCM [0, 0.1]
57.9
51.8
69.6
33.3
76.8
48.4
51.6
67.7
57.1
FCM [0.0, 0.15]
56
50
74.1
40.6
76.3
53.8
52.4
70.2
59.2
FCM [0.0, 0.3]
52.5
53.6
69.4
42,9
75,4
49.8
48.4
66.1
57.3
one-shot
PaL.M
48.3
44.6
50.9
75.8
47.8
72
35.9
66.7
55.3
FCM [0.1. 0.1]
56.1
32.1
53.8
76.3
47.3
72
33.3
67.4
54.8
FCM 10.15,0.151
48.3
37.5
52.4
75.9
49.1
69
20.9
66.3
52.4
FCM [0. 0.1]
42.9
53.1
76.8
47.8
55.6
FCM [0.0, 0.15]
52.6
44.6
54.5
76.4
46.9
73
43.2
71.6
57.9
FCM [0.0, 0.3)
50.8
32.1
52
75.4
47.2
74
46.5
66.3
55.5
Table 7: Comparisons on SuperGLUE zero-shot, few-shot and finetuning benchmarks between using attention mask and using mask token. The model size is 1B.
Masking strategy
O-shot avg
1-shot avg
5-shot avg
finetune avg
Mask token Attention
$7.4
$7.0 $7.9
55.4
68.5
59.2
57.2
68.7
with dropout rate 0.1, (3) FCM with fixed random ratio 0.1, and (4) FCM with fixed random ratio 0.1 and dropout rate 0.1. We see that using dropout during large language models pre-training is harmful, decreasing the score from 55.7 7 to 55.4, which aligns with findings from prior work (Raffel al. 2020: Chowdhery et al. 2022). In contrast, combining dropout with FCM together can im- prove PaLM, improving the score 55.7 to 57.7, indicating that FCM and dropout are complementary techniques and leave further studies of this an interesting future work. We can see that using only FCM performs slightly better than combining dropout and FCM together. showing the effec- tiveness of FCM on performing the next token prediction task with randomly selected past tokens masked out.
Table Comparisons on SuperGLUUB zero-shot benchmark between between Random Masking vs.
Dropout, The model size IB.
Model
BoolQ
CB
COPA
MultiRC
ReCORD
RTF
WiC
WSC
Avg
PaLM
45.9
72,4
35.2
75.8
50.9
51.6
65.3
55.7
PaLM + Dropout
53.5
48.2
64,4
37.2
75.7
50.2
50.2
63.5
55.4
FCM 1o.1. 0.11+ Dropout
44
53.6
71
43.1
75.3
59.2
49.8
65.4
S7.7
FCM 0.11
56.5
51.6
73.5
32.9
76.3
55.6
52
67.1
58.2
5 CONCLUSION
In this paper, we propose FCM, novel pre-training paradigm using a causal transformer decoder. FCM is a combination of causal next-token prediction and random masking input sequence. Ex- perimental results show that FCM significantly outperforms the state-of-the-ar causal transformer model on wide range of zero- and few-shot as well finetuning benchmarks, and our model is readily extendable to various tasks.
hext-token-prediction
As FCM improves performance of causal language models few-shot and finetuning bench- marks, applying our approach to other language understanding tasks and language-iman image tasks (e.g., Flamingo (Alayrac al. 2022)) is promising direction for future work. Since method does not introduce extra computation, another direction would be investigating what is the impact FCM on compute scaling law (Hoffmann et al. 2022)
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, and Luke Zettlemoye. Cm3: A causal masked multimodal model of the internet. arXiv preprint arXiv: Arxiv-2001.07520, 2022.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Ju- lian Ibarz, Brian Ichter, Alex Irpan, Eric Jang. Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jes- month, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang- Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, and Mengyuan Yan. Do as can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv: Arxiv-2004.01991, 2022.
HannBBapistt Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. arXiv preprint arXiv:2204. 2022.
Vamsi Aribandi, Yi Tay. Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang. Vinh Q Tran, Dara Bahri, Jianmo Ni, Jai Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. Ext5: Towards extreme multi-task scaling for transfer learning. arXiv preprint arXiv: Arxiv-2111. 2021.
Mikel Artetxe, Jingfei Du, Naman Goyal, Luke Zettlemoyer, and Ves Stoyanov. On the role of bidirectionality in language model pre- trainingn arXiv preprint arXiv: Arxiv-2205.11726, 2022.
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unilmv2: Pseudo-masked language models for unified language model pre-training. arXiv preprint arXiv: Arxiv-2002.12811 2020.
Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. arXiv preprint arXiv: Arxiv-2207. 14255 2022.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam M. Shazeer. Scheduled sampling for se- quence prediction with recurrent neural networks. NIPS, 2015.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. arXiv preprint arXiv: Arxiv-191 1111111111,111111111111111111111 2019.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao. Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shiv- anshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. Gpt-neox-20b: An open-source autoregressive language model. BIGSCIENCE, acl.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Ben- gio. Generating sentences from a continuous space. arXiv preprint arXiv: 151 1.06349, 2015.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:
//github. com.b...ommae gowp://oggll///aaaa//
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards. Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian,
11

---

Under review as a conference paper at ICLR 2023
Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. arXiv preprint arXiv: Arxiv-2107.03377 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023111 2022.
Andrew M Dai and Quoe Le. Semi-super sequence learning. Advances in neural information processing systems, 28, 2015.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:/810.0440 2018.
Chris Donahue, Mina Lee, and Percy Liang. Enabling language models to fill in the blanks. Acl, 2020. doi: 10.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neu- ral Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 13042-13054, 2019. URL https: /ttp::////rrrretiiiiiii Gast://20032449555cccccc
newrips..pp... httpcccpppppp.....///////// - .html
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu. Zhilin Yang. and Jie Tang. GLM: general language model pretraining with autoregressive blank infilling. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 320-335. Association for Computational Linguistics, 2022. URL https: acl / 26
7/ac wwwp:/ow...ggg....oo/////
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv: Mxxiv-2204.5999, 2022.
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. Flax: A neural network library and ecosystem for JAX, 2020. URL
httt:://githbb. com wwt.:ccoomooooooooooo http///ooollllll/o///////o
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen- nigan, Eric Noland, Katic Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. arXiv preprint arXiv: 2xxix-2203.1555: 2022.
Vid Kocijan, Thomas Lukasiewicz, Ernest Davis, Gary Marcus, and Leora Morgenstern. A review of winograd schema challenge datasets and approaches. arXiv preprint arXiv: Arxiv-2004. 13831, 2020.
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Eduardo Blanco and Wei Lu (eds.),
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations, Brussels, Belgium, October 31 November 4, 2018, pp. 66-71. Association for Computational Linguistics, 2018. doi: 10. 10.18653/v1/d11 2012. URL https://doi www.org...... 16553/////llll. 8-
12

---

Under review as a conference paper at ICLR 2023
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Marie-Franciie Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-/1 / November, 2021, pp. 3045- 243.
3059. Association for Computational Linguistics, 2021. doi: 10. 000663//1/2221emmmlpmoooo
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising scquence-to-sequemn pre- training for natural language generation, translation, and comprehension. arXiv preprint arXiv: 2019.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Sone et al. Solving quantitative reasoning problems with language models. arXiv preprint 2rX.v:2206. 2022.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101 1000022001.00190,, 2021.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale gen- Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 158-167, Vancouver, Canada, jul 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1000 URL http://///.clleatooooooooooo////
cration:
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. aem computing surveys, 2021. doi: 10.1145/3560815.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? A new dataset for open book question answering. In Ellen Riloff, David Chiang, Julía Hockenmaier, and Jun' ichi Tsujii (eds.). Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pp. 2381-2391. Association for Computational Linguistics, 2018. doi: 10.18653/v1/d18. -2660. URL https: / /oi.. org /tlg/10.00 18..0.8653330 -
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task general- ization via natural language crowdsourcing instructions. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22- https:
27, 2022, pp. 3470-3487. Association for Computational Linguisties, 2022. URL
7/aclanthologys org/2022. acl-
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Van- derwende, Pushmeet Kohli, and James Allen. A corpus and evaluation framework for deeper understanding of commonsense stories. arXiv preprint arXiv: 16xi6-160..01666, 2016.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adver- sarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv: Arxiv- 1910. 14599, 2019.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context, arXiv preprint arXiv: Arxiv-1606.06 06031, 2016.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACI, 2018.
13

---

Under review as a conference paper at ICLR 2023
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. J. Mach. Learn. Res., 21:140:1-140:67, 2020. URL http://jmmn org/
papers/v21220-072 html
Adam Roberts, Colin Raffel, Katherine Lee, Michael Matena, Noam Shazeer, Peter Liu, Sharan Narang, Wei Li, and Yanqi Zhou. Exploring the limits of transfer learning with a unified text-to- text transformer. 2019.
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowyez, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bu- lian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepare Shepard, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. Scaling up models and data with t5x and seqio. arXiv preprint 1xXi::2033.11889 2022. URI. https://axxiv. wttp://////p/...../////
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487. 2022.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An ad- versarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artifi- cial Intelligence, AAAI 2020, The Thirty-Sccond Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances Artificial Intelli- gence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 8732-8740. AAAI Press, 2020. URL httts:://ossstt awai. org/ wwwggginne index.nixx..
Awt.p//AAAAA. http:///A///aatcce//eee/e/////// 6399
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference Learning Repre- sentations, 2022. URL https: wwtp:eiwwwwwweeeeeeeeeeeeeeeeeee wtwp:eewwwwiineetoooumuummmem forum 9Vrb9DowI4
Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In 2020 IEEE/CVF Conference on Computer Vision and Puttern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 4937 4946. Computer Vision Foundation IEEE, 2020. doi: 00.1009/CVVP2220022000004999 URL https: http://///ppecccss.ssss taecss.thecct com htaam/ccoa content.ttttttecoooocco.c......t Swweeent__e___e_eee CVPR wewuuueeeeaannnnnn
Matee_Matchiiiiii M Wwthhhhhhhhhhhhhaat Neura Networks_
CVPR 2at___0000_pp_______ html]
Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:
Arxiv-1911.02150 2019.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv: Arxin-2002..52022 2020.
Noam Shazeer and Mitchell Stem. Adafactor: Adaptive learning rates with sublinear memory cost. arXiv preprint arXiv: Arxiv-1804.04225 2018.
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky. Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. J. Mach. Learn Res., 15: 1958- 2014.
14

---

Under review as a conference paper at ICLR 2023
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. arXiv preprint arXiv: Arxiv-2104.09864, 2021.
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. Unifying language learning paradigms. arXiv preprint arXiv: Arxiv-2205.00 00,0.00511,... 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL foo00e011100111111144444------00 < http://ppp..cppppp////////////// ppff
Ben Wang and Aran Komatsuzaki. GPT-J-6B: 6 Billion Parameter Autoregressive Language May 2021.
Model. http:///tee///ea..oo.oooo/ooo//s
Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. What language model architecture and pretraining objective work best for zero-shot generalization? arXiv preprint arXiv: Arxiv-2204.05832, 2022.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetumed language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022a. URL https: id id=gE2rGCozdqR
net /forum?
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou, Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 1990, 2022b.
Ziang Xie, Sida I. Wang, Jiwei Li, Daniel Lévy, Allen Nie, Dan Jurafsky, and A. Ng. Data noising as smoothing in neural network language models. ICLR, 2017.
Vikas Yadav, Steven Bethard, and Mihai Surdeanu. Quick and (not so) dirty: Unsupervised se- lection of justification sentences for multi-hop question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing, EMNLP-IJCNLP 2019, Hong Kong. China, November 3-7, 2019, pp. 2578- 2589. Association for Computational Linguistics, 2019. doi: 10. 10.18653/11/D11-12226 URL 011653/vl/00 vlt/vlll//9999 9-160
https://doi. .arlboooorrgoo org/10. 18653/
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv: Arxiv-1906.08237 2019.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models. arXiv preprint arXiv: Arxiv-2005.01063 2022.
Wanrong Zhu, Zhiting Hu, and Eric Xing. Text infilling, arXiv preprint arXiv: Arxiv-1901.0011 2019.
15

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 IMPLEMENTATION AND TRAINING DETAILS
Our implementation uses Flax (Heek et 2020). JAX (Bradbury et al. 2018) and T5X (Roberts et al. (022)) Our architecture is based on PaLM Chowdhery et al. 2022) which introduces some modifications to GPT-3 (Brown et al., 2020) architecture to reduce compute cost. The dataset is provided by Tensorflow datasets. We use SentencePiece (Kudo & Richardson 2018) tokenizer. For PaLM and FCM trained on C4 datasets, the sequence length 1024 to reduce compute cost, though the official PaLM is trained with longer context length 2048. Following the settings of PaLM, input examples are concatenated together and then split into sequences of exactly 1024 tokens, so that there
padding tokens, but examples may be split in the middle. Input examples differ- entiated from one another with a special [eod] token For downstream tasks evaluation, including both fewshot and finetune benchmarks,
follow the dataset format and splits used Brown al. 2020); Chowdhery et al. (2022). Our experiments are conducted using cloud TPU v4, it has unified 32 GiB HBM memory space across the entire chip. For 128M models, training 180B tokens takes 18 hours on TPU v4-64. For 1B models, training on 180B tokens takes 25 hours on TPU v4-256. The training of 8B models on 180B tokens takes 100 hours on TPU v4-512.
A.2 HYPERPARAMETERS
In this section provide the training and evaluation hyperparameters of FCM. These configurations follow the training hyperparameters of PaLM Chowdherr et al. (2022).
Hyperparameter
Value
Dropout
0.0
Optimizer
Adafactor
Initial learning rate Ir
0.01
0.01 for the first 10,000 steps,
Learning rate decay
which is then decayed at a rate of 1/ 1/Vk,
where is the step number 2
Weight decay
Optimizer momentum
B1 = 0.9, B2 = 1.0 - 0-08
Global norm gradient clipping
1.0
Batch size
1024
Sequence length
1024
Table 9: Hyperparameters for training PaL. and FCM
Hyperparameter
Value
Dropout
0.1
Optimizer
SGD momentum
Momentum
0.9
Batch size
512
Sequence length
1024
Table 10: Hyperparameters for finetuning PaLM and FCM
A.3 FULL RESULTS
Table 12 includes the results for the FCM and the PaLM IB and 8B models across three random evaluation seeds. Following prior work, we only consider single checkpoint results from pre-trained language models. The variance across different evaluation seeds is small on most tasks.
A.4 ADDITIONAL EXPERIMENTS ON COMPARING WITH
16

---

Under review as a conference paper at ICLR 2023
Related FCM is fll-in-the-midde training, which splits documents into three pieces random and moves the middle piece to the end. This is similar to the procedure used in Aghajanyan et al.
2022 Donahue al. (2020): Bavarian et al. (2022) Artetxe et al. (202). FCM is orthogonal to fill-i-the-middle training, and in fact, our method can be easily integrated into such methods:
2022 and HYBUNI (Attttx et 2022 focus introducing bidirection- ality into causal masking objectives, where the masked tokens are moved to the end of the sequence in order make it possible attend to future positions from these masked tokens. The introduction of bidirectionality gives the model additional capabilities to attend to the future when predicting masked tokens.
CM3 (Aghajanyan et al.
On the contrary, our paper focuses exclusively on the setting of unidirectional language models. Unlike CM3 and HYBUNI, our approach focuses on improving autoregressive language models without altering the input sequence or changing the training objective. Instead, we only change the attention masks for unidirectional language models. fact, we can incorporate our attention masking method into bidirectional models such CM3 and HYBUNI, but it is beyond the scope of paper and we leave for future work.
In Table 11 we compare with simpler version of CM3 and HYBUNI, where we randomly split the sentence into [prefix, middle, suffix], move [middle] the end of the sequence, concatenate the three pieces using sentinel tokens, and train the causal language model to predict the sequence. We report zero- and few-shot results on SuperGLUE, The fill-n-the-middlee is denoted as FIM. We did not improvement in zero- and few-shot learning. BBvariin et al. 2002 also reported similar finding that few-shot performance does not improve by moving the infill regions to the end of the context.
CM3 demonstrates impressive results on cross modal generation and representation learning tasks, believe combining CM3 and FCM could further improve the performance which leave interesting future work.
Table 11: Comparison between FCM, PaLM, and iil-in-the-midlle (FIM) on SuperGI.UF zero-shot and few-shot benchmark tasks. All models are trained for 180B tokens on C4. The model size is IB.
of shots
Model
BoolQ
CB
COPA
MultiRC
ReCORD
RTE
WiC
WSC
Avg
PaLM IB
52
50
82
40.4
85.5
53.4
51.3
78.3
61.6
Zero-shot
FIM IB
50.5
47.7
78
39.8
83.7
47.9
50.1
77.9
59.8
FCM IB
62.1
48.2
84
54.1
85
51.1
79.1
64
PaLM IB
53.7
42.9
80
49.8
85.5
55.2
51.5
79
62.2
One-shot
FIM IB
52
43.1
78
49
83.1
48.2
45.9
80
59.9
FCM IB
59.6
51.5
83
56.5
84.9
47.3
46.9
86.8
64.6
PaLM IB
49
46.4
82
42.5
84.6
53.1
50.5
77.9
60.8
Few-shot
FIM IB
50.5
45.6
79
39.8
80.1
50.9
49.4
58.8
FCM IB
57.7
50
85
46.5
83.9
48.4
49.5
87.9
63.6
17

---

Under review as a conference paper at ICLR 2023 3
Table 12: Results across three random realizations. We use the same setup as in Brown et al. (2020); Chowdhery et al. 2022), including the splits for each task.
ANLIR3
ANLUR2
ANLIRI
CB
ReCoRD
WSC
Multire (Fla)
WIC
RTE
Copa
BoolQ
Winogrande
Winograd
-QA
Openbook
ARC-c
ARC-e
PIQA
StoryCloze
Lambada (EM)
Task
30,2/ 30.2/296
313/313/315
44.9/44//44
75.8/75.7722
66.5 16656666.
/7.1/7.1/52
47.8 47.8/478//5 47.8
52.0 1..0/53.1/5 542
71.0/330/740
48.0 48.0/48. 44.3 /487
54./5400550 54.0
67.1 1670 63.0
55.0/550045 /450
26.31 26./2600
18.0/48.0/0 48.1
710/710/710
6 67.4
48.6 /4..9/9//40 /9.3
30,0
30.6/ /303
30,5
31.3
44.6
75.8
66.7
57.1
47.8
53.1
72,0
48,3
$4.0
67.0
45,0
26.3
48,0
71.0
67.3
48.9
PaLM IB
310/311//22
139//331/200
764/7644776
715/312
510/574/592
169/469/46
545/545
730/730/730
528/526/522
558/558/550
675/674/667
432/432/432
211/274/221
159/4459/499
45.9
7L3/716/7166
667
49444944
31,2
014//34//31
30.6
33,0
44,8
76.4
71.2
57.2
46.9
54.5
73,0
52.6
55,8
67.4
43,2
27.2
45,9
71.6
09.16669666 67.1
66,9
49,5
FCM
IB
/764
19.99
/669
54.5
524
/996
One-shot
315/315/3211
306/
3277
129/429/442
85.8/85./ /
85.8
79.0
79.
498 /49.8/49.9
790/799
550/
790/
335/537/530
60.1 660.566000
795/795//79
470/470/497
/40/44/3411
602
75.5/554/75
75.1
058/658/66 15.9
31.7
00/0000/222 30.
30.6
0..7/329/225
32.7
42,9
85.5
79.0 79.0
79.0
49,8
790 79.0
79.0
55,2
000/800081000
80.0
53,7
60.5
79.5
47.0
34.0
66.1 66.1
60.1
75.5
75,0
65.8
8B
PaLM
1609
7488
335/333/34
340/34.033
335/335 3355/35/333 /3.6
50.//220/5 52.0
849
868
56.9
860/
47.34733/47
82.0/840/885 830
590/
/2.1622/
813/818/
4831485145
/ /3.00/300
602
76.5/765//0 /765
75.5
255/759//2
66.
33.8
33.7
33.5
51.5
/9/99/889
84.9
1669 86.7
86.8
56.0 56.6
56.5
87.00 87.4
86.8
/473
47.3
83.0
59.6
/222/6222
62.1
81.7
484
35.0
/
60,2
76.5
75.7
66.5 6665
66.5
8B
FCM
/3.1
81.7
60.3
/758
50.8
32./322//33
32.8
32.5/32.5/32 32.5
32,5
20.8/30./200 299
30.5
42.3/42.3/42
42,3
74./7499778
74.9
06.7166.1664 66.6
66.7
51.157.3157
57.2
489/489
48.9
510//331//5
53.1
10.1/710//00
70.0
2.11/482/48
48.1
48.1
53.6/336/55
53.6
65.0/646// 64.8
64.8
42.6/42./42.7
42.6
26112699/2 26.6
26.5
50.2/5002/5 50.2
50,2
72.0/2.1//22
72.0
2/8//5.9/66 65.9
65.9
48.2 48.2/48./44
48.2
PaLM
IB
489
34.0340/34 /346
34,2
130//33//34 /34
33.4
22.5/325/322
32.5
2811/88.146 /484
48,2
75.0/75.077
75.0
70.0/005
/00/70./// 70.1
70,2
482/449
48.2
50.0/500/
50.0 50.1
50.1
45.1745.1/44 /45.1
45.1
12.0/710/733
72.0
/68/468//4
46.8
55.9/558/55 533
55.0
700/700/0
70.0
43.5/43.//43 /43.
43.6
28.1/28.1
28.1
08.2188.1/8
48.2
716/716/713
71.6
16.66/5// 6666
66.7
66.766655
66.7
19.6/49.7
49,7
FCM
/48.
IB
466
700
28.1
149.7
Few-shn shot
32.8/22./333
31.33317/31.7
31.1/312/310
466/
84.8/ 04/8488/443
77.9/77.9/77
42.3/2..3/42
77.9/77./77
53.1/53.1/51
830/810/822
19.0/49.0//00
59.1/8897 65.0
79.5/79.5/ 79.5
49.0/
35.3/35.5355
64.0/44.0766
72.//7.1/7
75.6/559/75.
66.1/66.0/ 66.1
32.9
31.7
31.1
06.666.6// 46.0
46.4
84.6
77.9
42.5
77.9
53.1
82.0
49.0
61.0
79.5
49.04994499
49.0
35.5
64.0
77.1
75.8
66.1
SB
PaLM
34.1335.935/
35.1
13.9/33.9/33
33,8
32.1/3211223 /325
32.9
5L.0/510//
50.0
34.0/84.188
83.9
08.1/878/878
88.1
87.9
365/462/446
46.5
88.4/87A/88 87.9
87.9
0.4//8.4/44
48.4
$5.0/5.0/85.00
85.0/85.0/8
85.0
56.7/57.97558
57.7
62.3/62.4666
62.3
81.0/81/822
81,2
49.0/49.1/54
49,5
3655/371//5
36.5
/..0/61.8/6 64.5
64.0/
64.4
73/3733mm
77.3
76.0 7600766776
76.2
67./6265/66
67.5
8B
FCM
48.0
8336
81


Reviews:
[{"review_id": "q1dcF_2HndO", "reviewer": "Reviewer_wa4A", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to randomly mask past tokens in casual language models. \n", "summary_of_the_review": "This paper presents a new casual masking objective, and carries out apple-to-apple comparisons with PaLM 128M, PaLM 1B. In comparison with this paper, [1] studied 6 different training objectives (including the most relevant casual masking method) and carried out comparisons with these six training objectives with 5 different scales: 125M, 355M, 1.3B, 2.7B, 6.7B. Due to seriously ignore related articles and casual masking method from [2] is pretty similar as this paper (approach of [2] might be better), I choose to reject this article.\n\n\n*References*\n\n[1]. Artetxe, Mikel, et al. \"On the Role of Bidirectionality in Language Model Pre-Training.\" arXiv preprint arXiv:2205.11726 (2022).\n\n[2]. Aghajanyan, Armen, et al. \"Cm3: A causal masked multimodal model of the internet.\" arXiv preprint arXiv:2201.07520 (2022).", "strength_and_weaknesses": "*Strength*:\nThis paper is well written and easy to understand.\n\n*Weaknesses*:\n\nThe biggest weakness of this paper is ignoring related research papers and lack of comparison. \n[1] presents comprehensive study about different training objectives in zero-shot priming and finetuning. They compared 6 different training objectives in table 1 (https://arxiv.org/pdf/2205.11726.pdf): NXTUNI, NXTPRE, MSKUNI, MSKBI, HYBUNI, HYBPRE. Casual language modeling such as GPT models belong to NXTUNI. They controlled hyper-parameters and training data to be same and studied these six different objectives. The most relevant method to this paper is HYBUNI (casual masking objective). [2] is the representative work of causal masking objective. This paper presents this new causal masking objective without comparing it with the most relevant casual masking method (CM3). \n\nIn addition, this paper should carry out apple-to-apple comparisons with other models. For example, it needs to control model sizes (model parameters), batch size, hyper-parameters, training tokens, training devices, to be exactly the same. And control Total train compute (flops) to be similar. Therefore, the qualified apple-to-apple comparison is only PaLM 1B v.s. FCM 1B (in table 3), PaLM 128M v.s. FCM 128M (in table 4). For others, PaLM 8B used different training data with FCM 8B, and T5 used different model parameters. \n\n*References*\n\n[1]. Artetxe, Mikel, et al. \"On the Role of Bidirectionality in Language Model Pre-Training.\" arXiv preprint arXiv:2205.11726 (2022).\n\n[2]. Aghajanyan, Armen, et al. \"Cm3: A causal masked multimodal model of the internet.\" arXiv preprint arXiv:2201.07520 (2022).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "o0MuExs-Fs", "reviewer": "Reviewer_pAfF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed a technique to improve zero-shot and few-shot capabilities for Large language models (LLM). Specifically, when performing the next token prediction task, the proposed method will randomly select past tokens masked out to fine-tune the LLMs. Experimental results show that the proposed FCM improves PaLM in both zero and few-shot tasks.", "summary_of_the_review": "The novelty of this paper is limited.  Methods similar to this paper have been widely used in previous work. On the other hand,  the authors should give more analysis about why this method can work in LLMs. ", "strength_and_weaknesses": "Strength:\n1. The proposed method is simple and easy to reproduce.\n2. Experimental results show the proposed method can effectively improve the LLMs in both zero- and few-shot tasks.\n\nWeaknesses:\n1. This method can be seen as a variation of schedule sampling (Bengio et al., 2015) or a simplified version of UniLM (Dong et al., 2019). The novelty of this paper is limited.\n2. On the other hand,  the authors should give more analysis about why this method can work in LLMs. The training set of pre-training models contains a lot of noise data. I wonder if the proposed method is necessary.\n\n\nBengio et al., 2015. Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks.\nDong et al., 2019. Unified language model pre-training for natural language understanding and generation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jCFE3Ekou6", "reviewer": "Reviewer_8z3G", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel, simple, scalable and efficient pre-training approach for causal, decoder-only language models. The proposed approach is called Forgetful Causal Masking (FCM) and it randomly masks out past tokens. FCM pre-training is applied to PaLM to demonstrate significant improvement in the 0-shot performance on the SuperGLUE benchmark. FCM is also tested on a variety of 0-shot and few-shot tasks - PIQA, ARC, OpenBookQA, Winograd, WinoGrande, ANLI, StoryCloze, and LAMBADA tasks to demonstrate consistent performance improvement across such diverse tasks. \n", "summary_of_the_review": "The paper proposes a simple and scalable pre-training tactic - of causal random masking for causal language modeling. It doesn’t require additional parameters or computations. Extensive experimental evaluation is performed. While the performance gains seem modest to moderate, and somewhat inconsistent, they are largely positive across a large range of tasks. Though the technical novelty is limited, I expect the proposed mechanism to become a part of the pretraining toolbox for causal LLMs.\n", "strength_and_weaknesses": "## Strengths\n- Simple and scalable approach: no extra parameters or computations are added to pre-training using causal LM objectives.\n- Extensive experimentation\n- Performance improvements (modest) on a wide variety of tasks\n\n## Weaknesses\n\n### Training \n- __(W.0)__ Kindly share the computational resources used, memory requirements and the training time.\n\n### Results and Analysis\n- __(W.1)__ In Table 1, along with the mean of the 3 random realizations, the spread should be provided as well, given that many of the performance numbers are within a single percentage point. \n- __(W.2)__: The performance of FCM appears to be better than PaLM, but not significantly and consistently so. For significance, I’m using a criterion of at least 1% in absolute terms. It will be good if the authors can shed some light on the following:\n- __(W.2.a)__ __0-shot__: In 8/19 tasks, the performance improvement is < 1%.  \n- __(W.2.b)__ __1-shot__: In 9/ 19 tasks, the performance improvement is < 1% and 2/19 times it’s worse. \n- __(W.2.c)__ __few-shot__: In 4/ 19 tasks, the performance improvement is < 1% and 4/19 times it’s worse. \n- __(W.2.d)__ Scaling trends: If we look at the consistency of performance improvement as the number of shots increase, there are very few tasks, on which FCM consistently stays better than PaLM by at least a percentage point. On the NLU tasks, FCM seems to lose its advantage as the number of shots increase, sometimes becoming much worse.\n- __(W.3)__ __Classification finetuning__: results on SuperGLUE dev set only show significant improvement on 4 out of 8 tasks, with majority of the gains observed on WSC and CB. The authors should discuss the variability in performance gains.\n- __(W.4)__ __Scalability, 0-shot__: In Table 4, FCM 1B results seem better than FCM 8B on CB, RTE, and WiC tasks. Kindly discuss. \n- __(W.5)__ __Scalability, k-shot__: In Table 5, the performance of both PaLM and FCM sometimes, and more often for FCM, degrades as the number of shots increases. Kindly discuss.\n- __(W.6)__ __Ablation of mask ratio__: Irrespective of the average across all the tasks, FCM[0,0.15] is better only on 2 out of 8 tasks in the 0-shot setting and 3 of 8 tasks in the 1-shot setting. This shows that the choice of mask ratio is task dependent. Kindly discuss. \n- __(W.7)__ __Comparison with Dropout__: Contrary to the claim of dropout hurting FCM performance, it improves in 3 of 8 tasks. It improves performance by 10.2 absolute % points for the MultiRC task. It seems that the benefit of dropout is task dependent.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "iNCrV6EP0oN", "reviewer": "Reviewer_r8mR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a simple method of randomly masking past tokens during causal language modeling that boosts zero-shot capabilities  and fine-tuning results by a non-trivial margin. The problem motivation is capturing the best of both decoder and encoder LMs. While previous methods achieve this as well, the authors assert that the proposed method in this paper does not add any additional computational cost. ", "summary_of_the_review": "The authors propose a very simple methodology of simply masking out random tokens during causal language modeling which proves to be effective at enhancing both zero-shot capabilities and the downstream fine-tuning results. I highly recommend this paper be published at this conference. However, it would be great if the authors could elaborate more on the question of the proposed method in terms of the engineering implementation aspect.", "strength_and_weaknesses": "Strengths\n1. The method is very simple and well explained in the method section. Experimental details are also well explained.\n2. Different ablations of which factor of the proposed method contribute to the performance enhancement are provided.\n3. Proposed method boosts performance in the zero-shot setting as well as when fine-tuned to perform the downstream task. \n\nWeaknesses\n1. While the proposed method does not add any computational cost in theory, since previous tokens are randomly masked out, won't there be an engineering overhead since the previous hidden states cannot be cached which would have remained unchanged if the previous token weren't masked out otherwise? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject