Are Bert Family Good Instruction Followers?  A Study on Their Potential And Limitations
Authors: yisheng xiao
Juntao Li
Zechen Sun
Zechang Li
Qingrong Xia
Xinyu Duan
Zhefeng Wang
Min Zhang
Venue: ICLR 2024 poster (2024)
Keywords: Instruction tuning
Large language models
BERT family
Natural language generation
Primary area: generative models

Abstract: Language modeling at scale has proven very effective and brought unprecedented success to natural language models. Many typical representatives, especially decoder-only models, e.g., BLOOM and LLaMA, and encoder-decoder models, e.g., Flan-T5 and AlexaTM, have exhibited incredible instruction-following capabilities while keeping strong task completion ability. These large language models can achieve superior performance in various tasks and even yield emergent capabilities, e.g., reasoning and universal generalization. Though the above two paradigms are mainstream and well explored, the potential of the BERT family, which are encoder-only based models and have ever been one of the most representative pre-trained models, also deserves attention, at least should be discussed. In this work, we adopt XML-R to explore the effectiveness of the BERT family for instruction following and zero-shot learning. We first design a simple yet effective strategy to utilize the encoder-only models for generation tasks and then conduct multi-task instruction tuning.  Experimental results demonstrate that our fine-tuned model, Instruct-XMLR, outperforms Bloomz on all evaluation tasks and achieves comparable performance with mT0 on most tasks. Surprisingly, Instruct-XMLR also possesses strong task and language generalization abilities, indicating that Instruct-XMLR can also serve as a good instruction follower and zero-shot learner. Besides, Instruct-XMLR can accelerate decoding due to its non-autoregressive generation manner, achieving around 3 times speedup compared with current autoregressive large language models. Although we also witnessed several limitations through our experiments, such as the performance decline in long-generation tasks and the shortcoming of length prediction, Instruct-XMLR can still become a good member of the family of current large language models.

Published as a conference paper at ICLR 2024
ARE BERT FAMILY GOOD INSTRUCTION FOLLOWERS? A STUDY ON THEIR POTENTIAL AND LIMITATIONS
Yisheng Xiao , Juntao Li ; Zechen Sun ' Zechang Li?, Qingrong Xia ? Xinyu Duan 2
Zhefeng Wang?, 2, Min Zhang
Soochow University, China Huawei Cloud, China
fysxiaoo, scsuns suda. edu en, {1jt, minzhanggaauuua.ee... [lizechangl, xiaqingrong, waanninyy whwwwei com
ABSTRACT
Language modeling at scale has proven very effective and brought unprecedented success natural language models. Many typical representatives, especially decoder-only models, e.g., BLOOM and LLaMA, and encoder-decoder models, e.g., Flan-T5 T5 and AlexaTM, have exhibited incredible cntrucctio-followiiii capa- bilities while keeping strong task completion ability. These large language mod- els can achieve superior performance in various tasks and even yield emergent capabilities, e.g., reasoning and universal generalization. Though the above two paradigms are mainstream and well explored, the potential which are encoder-only based models and have ever been one of the most repre- sentative pre-trained models, also deserves attention, at least should be discussed. In this work, we adopt XML-R explore the effectiveness of the BERT family for instruction following and zero-shot learning. We first design a simple yet ef- fective strategy to utilize the encoder-only models for generation tasks and then conduct multi-task instruction tuning. Experimental results demonstrate that our fine-tuned model, Instruct-XMLA. outperforms Bloomz on all evaluation tasks and achieves comparable performance with mTO on most tasks. Instruct-XMLL also possesses strong task and language generalization abilities, indicating that Instruct-XMI.R can also serve a good instruction follower and zero-shot learner. Besides, Instruct-XMLR can accelerate decoding due to its non-autoregressive generation manner, achieving around 3 times speedup compared with current au- toregressive large language models. Although we also witnessed several limita- tions through our experiments, such the performance decline in gengeenrrttion tasks and the shortcoming of length prediction, instruct-XMLR can still become good member of the family of current large language models
the BERT family,
INTRODUCTION
Making large language models (LLMs) as good instruction followers further unleashes their power for different usages. Through instruction tuning with annotated examples and human preference feedback (if possible), the output of these LLMS can align with human expectations well on various tasks, e.g., understanding, generation, commonsense reasoning. under different settings, including zero-shot, few-shot, and of-houhf- (Ouyang et al., [022 Longpre et al., 20033). Many re- cent works have pointed the key factors and insights into the success of instruction following techniques on the popular decoder-only (e.g., GPT (Brown et al. 2020). LLaMA (Touvron et al. 2023)) and encoder-decoder frameworks (e.g., T5 (Raffel et al. (020a)), including but not limited to annotation quality (Zhou et al., 2023), data formats, and scaling (Chung et al. (002).. It can be predicted that more powerful alignment methods will occur for more model variants.
To enlarge the application range of alignment techniques, we alternatively explore the potential and limitations of the BERT family as instruction followers. To our knowledge, this the first work that attempts to provide suggestions for building instruction-followinn models the BERT family.
Junttao Li is the corresponding author. httpp::///// thub. com/ httm///ttterrrtteeeeKe.. www.nnatttt.iiateo natroet_.MMl//rree/maaa

---

Published as a conference paper at ICLR 2024
Our study is motivated by existing observations that: 1) the BERT family also be zero-shot learners (Zhao et al. 2022: Wang et al. 2023). Further equipping them with instruction understand- ing capabilities will expand their usage scenarios; 2) their masking language model objective can support open-ended long text generation and meanwhile achieve much faster decoding speed than the popular autoregressive fashion (Liang al., 2023b); 3) complicated real-world applications ally involves the collaboration of generative LLMs and expert models (Liang et al., [023c] [Shen
ct al. 2023), while the BERT family have moved countless tasks forward before the rising of large generative models and still hold the records for various downstream tasks.
Contributions Throughout this paper, we mainly (1) provide a basic method that can hint at the potential and limitations of the BERT family instruction followers; (2) present the key factors in making the BERT family competitive with the popular encoder-decoder and decoder-only models that have similar model scale and pretraining data; (3) disclose the risks and limitations (4) along with the possible reasons and solutions to give suggestions for further performance improvement.
Important Observations Although this work is at the preliminary stage, we still have some im- portant observations for reference and future research. First, the BERT family has the possibility to be good instruction followers that can generalize well across tasks and languages compared to the other popular LLMs with similar model sizes and data. Besides, our presented method is scalable, in which one can use more computation resources and data to improve performance significantly. Some of the problems are possibly owing to the limited capabilities of the backbone models since there are no previous BERT family models that are as powerful as the decoder-only and encoder-decoder counterparts yet. The inherent shortages of -on-autoregressive decoding also result in unconvinced and unreliability to feed different tasks and scenarios.
In the rest of this paper, we first provide the background of text generation via the BERT family and instruction tuning in Section 2 We then present the details for post-training the BERT family as a language generator in Section 3 and detailed experimental settings for instruction fine-tuning in
Section 4 Section 5 and 6 analyze their potential and the possible reasons behind the limitations.
RELATED WORK
2.1 TEXT GENERATION VIA BERT FAMILY
BERT (Devlin et al. 2019 revolutionized the field of Natural Language Processing by leveraging the Transformer architecture and large-scale pre-training. BERT a typical encoder-only architecture consisting of a multi-layer bidirectional Transformer encoder (Vaswani et al.] 2017) with stacked identical blocks. Building upon the foundation laid by BERT, researchers have further explored and expanded the capabilities of the original BERT architecture, leading to the development of the BERT family, such as RoBERTa (Liu et al. 2019). ELECTRA (Clark et al. 2020), DeBERTa (He et al, 2020), and XLM-R (Conneau al., 2020), among others. The bidirectional modeling characteristic and parallelizable masked language model (MLM) objective enables it to learn contextual word representations, facilitating the capture of comprehensive semantic information. Thus, the BERT family is renowned for its ability in natural language understanding (NLU) tasks. However, there is a scarcity of research that delves into their potential for text-generation tasks. Previous works (Dong et al., 22019: Wang & Cho 2019) have theoretically indicated that the BERT family can generate coherent and high-quality textual content. However, the primary usage of the BERT family has been focused on extracting contextual features rather than generating text by itself (Zhu et al.) 2019 Guo
ct al. 2020 Yang et al. 2020). Recent works (Chan & Fan 2019 Jiang et al. 202 2021|Su
Su et al. 2021
Liang et al. 2023b a) explore the application of the BERT family in non-autooeegressive generation using MLMs and receive positive feedback regarding performance. Their settings still follow the traditional pre-training and task- fine-tuning paradigm. In work, further explore the potential of becoming multitask instruction followers, which has been well explored in recent autoregressive language models but is blank area for BERT family.
2

---

Published as a conference paper at ICLR 2024
2.2 INSTRUCTION TUNING
Instruction tuning refers to the process of fine-tuning LLMs on an instruction dataset consisting of (instruction, oulput) pairs in supervised fashion, which narrows the gap between the next-word prediction objective of LLMs and the users' objective of having LLMs adhere to human instruc- tions (Zhang et al., 2023). The concept of merging diverse NLP tasks as generative tasks was pioneered by the T5 model (Raffel et al., 2020b). This method effectively simplifies the application of LLMs and paves the way for further advancements in instruction tuning models. Subsequent instruction tuning models like FLAN (Wei et al.) 2021] Chung et al. (022) Wei et al., 2021; Chung et al., 2022) and TO (Sanh et al., 2021 have further improved performance across diverse tasks by incorporating more task-specific instructions during the pre-training phase. Currently, instruction tuning represents an important research direction in NLP. The open-source community offers a va- riety of instruction datasets, such as xP3 (Muennighoff et al. (022), Alpaca (Taori et al., (2023), and Dolly (Conover et al., 2023), as well as instruction tie-ttnndd LLMs, such as BLOOMZ (Muen- nighoff et al.J 2022). FLAN-T5 (Chung al., 2022), and Alpaca (Taori al., 2023). However, the backbone of present instruction fine-tuned LLMS is mainly encoder-decoder and decoder-only based. The cetruuctioo-oolowiinn capabilities of the BERT family, which are encoder-only based models, are severely under-explored. In this work, we introduced Instruct-XMLR to explore the potential and limitations of the BERT family for instruction following.
METHODOLOGY
3.1 UTILIZING BERT FAMILY FOR LANGUAGE GENERATION
Dramatically different from the prevalent left-to-right autoregressive unidirectional language mod- els, the conditional independent factorization due the bi-directional nature of BERT family cap- tures more complex dependencies between tokens during training and also results in difficulty in generating reliable texts from scratch. In this section, we review the potential solution to utilize BERT family for language generation. As shown in previous work (Wang & Cho 2019 He et al.) 2022), BERT can be viewed as a Markov random field language model and then sample texts. Specifically, the sequence Y = (Yi, Y2, ..., Yr) can be viewed random variables of undirected fully-connected graph S, and this full graph can be defined as a clique in the Markov Random Field. Then the potential of this graph clique can be decomposes into sum of log-potential terms:
T
T
(S)
II (S) = exp Σ
log the (S)
(1)
where S denotes the fully-connected graph created from the original sequence Y, T denotes the number of random variables, i.e., the length of sequence Y. This can also be simplified by Con- ditional Distribution and Pseudo Lige-Likidoodd which maximizes the probability of each variable rather than joint probability of the entire graph:
(S)) 1 Σ log (u)(()))
Then for BERT family, Sy₁ can represent that y; is replaced by the masked token [MASK] in the entire sequence Y. Vi can represent each masked token. Then, the original training objective can be approximated as masked language model (MLM), where each potential (iS) is based on the model probability P(yr(S,;:0) 0). Given the sequence Y, we can achieve:
T (\)) = + - log log ~
1 _ R
1
log ((,,)
(3)
where R is the number of masked tokens sampled from Y. and the upper bound is the length of Y. Then we can expand this non-conditional language model to conditional scenarios, i.e., sequence to sequence generation. Given a training pair (X, Y), the model maximizes the following likelihood:
R
(Y) = RRC
Σ
log X(O))
(4)
3

---

Published as a conference paper at ICLR 2024
During inference, we begin with fully Y with all tokens masked and generate them in each step of [Y] (Wang & 2019). Notice that the conditional MLM and Mask-Predict are suitable for BERT family, which keeps the training and inference objective consistent with pre-training.
3.2 ONE-STEP TRAINING WITH DYNAMIC MIXED ATTENTION
We can draw inspiration from the previous practices for language generation; the encoder-decoder- based models feed the source sequence X into the encoder to extract the representations and use the decoder to learn the internal relationships of the target sequence Y. Then, a cross-attention module inserted into each decoder layer to aggregate the source representation and the target sequence. The key factor to language generation is how to learn the conditional probabilities P((|X) well, and this is the same for BERT family. However, unlike previous encoder-decoder- based models, the BERT family only contains a single multi-layer bidirectional Transformer encoder. To best match the traditional learning formats, we first build an additional model to perform like the decoder, then introduce mixed attention module (He et al. 2018 Liang et al. 2023b) to substitute for the original cross-attentie and self-attention modules in the decoder. Specifically, given a training pair (X, Y), the pre-training MLM containing I. layers where each comprises one self-attention layer and one feed-forward layer. We can get the source representation Hore of each encoder layer l:
Ft AnC
Se f Attenttinnnttr.... + Ward, Harr = FFN(F( + W
(5)
For target sequence Y, we randomly select partial tokens to be replaced by the [MASK] token and feed this corrupted sequence into the additional MLM. Then, this model tries to recover these masked tokens during training, and get the target representation Hige of each decoder layer I:
Higt = Mi xed_ettttto Att.ttteettttt AtttentionnnH @ H + Hige Egt FF)( + Higg
(6)
where + denotes the concatenation operation. Mixed-attention brings no additional parameters but just take the concatenated vector of source and target hidden states as key and value the original attention mechanism. As a result, this additional MLM can completely share parameters with the MLM encoder. Thus, we can adopt only one pre-training BERT model to save model parameters and accelerate the training process. However, the mix-attention mechanism first acquires the last layer's 'ss source representation. We must pass the model twice during training, leading to low training efficiency. As result, we introduce dynamic representation and learn to predict the masked tokens at the same pass as the model. This mechanism adopts each source representation of the corresponding previous layer rather the last layer
which allows the model to get source
Algt = Mi Mix.ee.tteettionne
Higs) + Hly
H!
FFN(H, Hgy)) + Aige
In practice, we can simply concatenate the source and the masked target sequence, then feed them into the model for training. Besides, as shown in Figure [ we prevents the query of each source token attending the target sequence in the attention module, which keeps consistent with the infer- ence process since there is target sequence in advance. Dynamic mixed-attention is more matching with the pre- training tasks of BERT family, making the idea of shared parameters more reliable.
X,
X X.YY Y.
X,
Xx
ix,
YY1
122
Ya
3.3 TRAINING AND INFERENCE PROCESS
The overview attention masking, Green and Blue ones de- note which can be attended while the white can not be attended.
Figure 1:
To better adapt generation tasks, given a training pair (X, Y), we uniformly mask to L (target length) tokens fol- lowing CMLM (Ghazvininejad al. 2019) rather the origi- nal fixed masking in BERT family. Besides, we also randomly mask a small ratio of tokens in source sequence to improve generality as mentioned in AMOM (Xiao et al. (023). Then the training ob- jective is to maximum the conditional MLM loss like the pre-training process:
M
MMLM =
Σ
log P(y: XM, 0),
(8)
4

---

Published as a conference paper at ICLR 2024
where M is the number of masked tokens in masked target sequence Yu. Notice the model do not need to predict the masked tokens in the source sequence XM-
During inference, we adopt the same Mask-Predict algorithm in CMLM (Ghazvininejad et al, 2019). which adopts multiple iterations to generate the final sequence. Specifically, given the to- tal decoding iteration T in advance, decoding iteration. In the following T - 1 iterations, a specific number of low-confidence tokens will be masked and re-generated. The number of masked tokens in each iteration can be computed as n T_í * Ly, where t denotes the current iteration number. The model selects the specific masked tokens in the next iteration based on the prediction probability, where tokens with the lowest probability will be masked, and their scores will be updated after the new prediction. Besides, unlike traditional left-to-right auto-regressive models, we should obtain the target length before initializing the fully masked target sequence. Naturally, we can directly give a length before inference. We also introduce a length prediction module following previous non-autoregressvve models where a special token [LENGTH] is used to predict the target length.
begin with the fully masked target sequence at the first
4 EXPERIMENTAL SETTINGS
In this section, we introduce our experimental settings in fine-tuning and evaluation process.
4.1 FINE-TUNING DETAILS
Backbone Model. Recent generative LLMs have been proven to be good instruction follow-
ers (Wei et al., 2021 Taori et al. (023), especially for these with relatively large size (Chung et al.]
2022 OpenAl 2023). To fully explore this capability of BERT family, we also select our back- bone model with large size version. As a result, we choose XML-R (Conneau et al., 2020), which pre-trained on around one hundred languages with masked language modeling objective and has two large versions, XML-Rxi and XML-RxXL (Goyal et al. (2021), containing 3.5 and 10.7 billion parameters respectively. Due to the cost of computing resources, We choose XML-Rxi. in our main experiments.
For fine-tuning instruction data, to make full use of the multilingual knowledge during the pre-training process of XML-R, select multilingual instruction dataset xP3 (Muen- nighoff et al. 2022). xP3 adds 30 new multilingual datasets with English prompts and serves as the multilingual version of P3 (Sanh et al. [02T). Overall, xP3 contains 46 languages and a similar language distribution of ROOTS (Laurençon et al. 2022). In the original xP3 paper (Muennighofi jet al., (202)), the authors use around 80 million samples for training, which is much more than the actual needs and increases the training expenses. As result, we sample partial instruction samples (around 1/80) from the original training corpus our experiment. Our sampled data only differs from the original data in sample size and remains strictly consistent in all other aspects, such as the composition ratio of each language, the number of tasks, prompt formats, etc.
Instruction Data.
Implementation Details. Our experiments use the pre-trained XML-RxL- which contains 36 lay- ers, 2560 hidden sizes, and 10240 feed-forward filter sizes. We implement all the experiments based on the Fairseq (Ott et al.] 2019) library on 8 NVIDIA A A100-PCIF--40IB GPU cards. We adopt the Adam (Kingma & Ba 2014) as optimization algorithm during training. The learning rate will warm up to 2c-5 in the first 500 updates and then decrease gradually with the polynomial decay schedule. We select the final model based on the validation performance.
4.2 EVALUATION PROCESS
Baseline Models. To make a relatively fair comparison between our fine-tuned model and other LLMs, we choose mTO-3.7B and BLOOMZ-3B since they have comparable model parameters and are also fine-tuned on the instruction dataset xP3. We give more comparison in Table [ These models are with different and covers the current popular architectures. Need to notice a big differ- exists in the number of training tokens, which always plays an important role on performance. Instruct- XMLRxt is trained on a bit more tokens than BLOOMZ-3B and half m during pre-training. While in fine-tuning Instruct-XML. MM..xc
Rx. is training on much fewer tokens.
5

---

Published as a conference paper at ICLR 2024
Instruction Tuning stage
Prestraining Stnge
Model
Backhone
Architecture
Parameter
Tok.
Datasets
Tok.
B1.OOMZ-3B
Bloom-3h 3b
Decoder-only
3B
15R
ROOTS
1.6T
366R
2048
mT0-3.7B
mT5-xl
Encoder-doooder
3.7B
158
mC4
6.4T
IT
1024
InstructXMMLAA
XML-Rxt
Encoder unly
3,5B
0.68
CC100
167B
0.5T
512
Table Comparison between our model MatrruttXMLLRx. and other baseline models. Tok. denotes the training tokens (billion). Len. denotes the sequence length limitation.
Tasks and Datasets. Following the previous work (Muennighoff ct al., 2022). we evaluate the model's ability of task generalization in three held-out tasks that are not contained in the fine- tuned instruction data: conference resolution, sentence completion, and natural language inference (NLD) Specifically, we use XWinograd (Tikhonov & Ryabinin| (021). XCOPA (Ponti et al.) 2020), XNLI (Conneau et al. 2018) dataset for each task, respectively. Besides, we also evaluate in machine translation as a language generation task, which can also serve as a held-in task since xP3 contains various translation samples in multiple languages sampled from Flores-200 datasets (Costa-jussá) cet al. 2022). We use WMT' translation dataset= which contains five language splits between English and others. We adopt the test set for evaluation, each containing around 3,000 sentence pairs. Fortunately, all these datasets contain the language splits that do not exist xP3, so we can also study and evaluate the model's ability for language generalization.
Evaluation Settings. During evaluation, we select five prompts from PromptSource (Bach et al., 2022] and then use them for all language splits of each dataset mentioned above. Finally, we report model need to get target length in advance, we adopt length prediction module for WMT 14 datasets and fixed length for others. Besides, we should adopt some constraints the design of prompts depending on the specific task formats due to the length prediction, more information can be found in Section 5.1
the average performance of these prompts. Since
5 ON THE POTENTIAL OF BERT FAMILY FOR INSTRUCTION FOLLOWING
In this section, we first present the potential of length prediction in successfully applying BERT family for instruction following. We then present the overall performance on three held-out tasks under the zero-shot setting to evaluate the model's 's ability task generalization. Finally, we further study the scaling effects, which serves as an important aspect assessing the potential for further improvement of model capabilities and performance.
5.1 LENGTH PREDICTION
For several non-autoreeressivee models, length prediction is an extra task to determine the target sequence length during inference (Ghazvininejad al., 2019 Qian al. (021). In comparison, the autoregressive models, c.g., the two baseline models BLOOMZ and mT0, generate the texts one- by-one in a left-to-right manner, and they can dynamically finish generation when meeting a special token indicating the end of sentences (e.g.. [EOS]). For Instruct-XMLRxu which adopts Mask- Predict algorithm (Ghazvininejad et al., 2019) for generating the texts as mentioned in Section 3.3 length prediction is an essential process and directly related to the quality of final generation results. Next, we discuss the potential of length prediction through our experiments.
Adopting the length predicted by the model-self is a common practice in original task-specific non-autoregressive models for various generation tasks, e.g., machine translation, summarization, and story generation. In their experiments, length prediction only brings tiny decline in per- formance compared with directly using the target length. It is also explainable that since these generation tasks are flexible, word adjustments eliminate the effect of minor length differ- ences during generation. Unlike several thove-mentioned generation tasks with flexible target lengths, some tasks with determined target lengths may heavily rely the length prediction. To verify this, we choose the XWinograd task and the prompt as {The in the sentence below refers
{optioni). True or False? (sentence) }. the label space is [True, False). Notice these two labels have different lengths after the sent encepiece

---

Published as a conference paper at ICLR 2024
tokenizer in Instruct-XMLRR, As result, once we set the length one in advance during infer- ence, the predictions are always {True]. If we set the length as two, the predictions are always (False), indicating that the predicted length directly determines the label. This is not reasonable situation and should be avoided in practical applications. As a result, we can add some constraints when selecting the prompts for these tasks.
Specifically, we can adopt length prediction module to get target length for traditional language generation tasks (e.g., machine translation since their target length can be flexible) and adopt fixed length for determined tasks ge.g., multiple choice tasks since they always have the target length as one). However, for such tasks whose labels have determined but different lengths, such as the task with label space (False, True}. J. we can transform the label space into (Yes, No} whose labels have the same length, and then adopt the corresponding fixed length without leaking information about target labels.
5.2 OVERALL PERFORMANCE
Many works have shown that large lan- guage models after multitask instruc- tion tuning can solve completely new tasks (Wei et al. 2021] Taori et al. 2023] Wang al.J 2022), we also examine our fune-tuned model: if Instruct-XMLR can successfully understand and com- plete the tasks which are not included in the fine-tuning process. Table 2 presents the results. To avoid the influence of language gener- alization discussed in Section 6.1 we only choose the languages included in the fine-tuned data here. We report the average accuracy of prompts in all tasks. We can find that Instruct-MMI.I also demonstrates strong task generalization ability. After fine-tuning only 1/25 tokens of baseline models, Instruct-XML. can significantly outperform a decoder-only model with comparable size BLOOMZ-3B in all tasks. Compared with the more competitive model mTO-3.7B. achieves better performance on XCOPA and comparable performance on XWinograd, but underper- forms little on XNLI. We attribute this failure to: (1) XNII is a multilingual dataset for traditional natural language inference (NLI) task, and m'TO with the encoder-decoder architecture is more ben- eficial to this task as also mentioned in Muennighoff et al. (2022), and (2) m0-33.7B is trained more much longer in both pre-training (I trillion v.s. 0.5 trillion tokens) and instruction tuning stage (15 billion v.s. 0.6 billion tokens), which can boost the performance of NLI tasks (Goyal et al. 202T
Models
XCOPA
XNLI
XWinograd
BLOOMZ-3B
0.519
0.482
0.523
mT0-3.7B
0.627
0.528
0.584
Instruct-XMLRxx
0.646
0.518
0.582
Table 2: Accuracy on three held-out tasks.
Pustruct-XMLL.
Hoffmann al. 2022
5.3 SCALING EFFECTS
Scaling law plays a vital role in the recent success LLMs (Hoffmann et al., 2022 Touvron et al., 2023). Since XML-R has different versions containing various parameters, we study the perfor- mance changes as the model size increases. Besides, we also focus on another layer of scaling, the number of training tokens during the fine-tuning process. Firstly, we also conduct experiments on relatively small models, XML-RBase and XML-RRaage with 270M and 550M parameters, respec- tively. | the results. We find an apparent growing trend the model parameters increase. After fine-tuned with the same data, Instruct- XMrrutttXMLLxx outperforms Instruct-XMLRpp and Instruct-XMLRR.n
all tasks, indicating the important role of model size in task general- ization. Then, we focus on the scaling effects of training tokens during instruction tuning process. Figure plots the performance changes in the training process. The performance of all tasks keeps improving as the training continues, which is consistent with our intuition. Overall, Instruct- XMLL also demonstrates the positive scaling effects, which presents potential improvements in the future.
6 POSSIBLE LIMITATIONS, REASONS AND SOLUTIONS
In Section 5 have noticed the potential of the BERT family for instruction following. However, more aspects remain to be explored for Instruct-XMi before becoming a superior instruction fol- lower. In this section, we investigate some limitations that may be caused by the specific factors of

---

Published as a conference paper at ICLR 2024
XWinograd
XCOPA
XNLI
Models
XCOPA
XNLI
XWinograd
0.70
0.60
Instruct-XMI.Rg.
0.583
0.386
0.521
Instruct-XMLRR
0.602
0,432
0.545
0.50
Instruct-XMLRxx.
0.646
0.518
0.582
0.18
0.28
0.38
0.4B
0.5B
0.68
Table 3: Effects on models scaling.
Figure 2: Effects on data scaling.
our model, e.g., the backbone model, including the original architecture and training corpus, the tra- ditional non-autoregressive generation manner we adopted during inference, etc. Then, after finding these limitations, we also give some possible solutions.
6.1 LANGUAGE GENERALIZATION
Since Instruct-MMLR and two baseline models are multilingual, they should support multiple lan- guages. Naturally, we aim to explore their abilities of language generalization, i.e., whether the model can perform well in languages not contained in fine-tuned datasets. Language generalization is also widespread concern of language models and is related strongly to language distribution in the training data.
We select the language splits that are not included in the fine-tuned data each task for evaluation. Table 4 presents the results. We can find that Instruct- XMLR outperforms BLOOMZ-3B but fails behind mTO-3.7B in all tasks, es- pecially on XNLI dataset. Considering the same language composition in fine-
Models
XCOPA
XNLI
XWinograd
BLOOMZ-3B
0.427
0.391
0.499
mT0-3.7B
0.605
0.533
0.569
Instruct-XMLRxx
0.599
0.466
0.539
Table Results of language generalization.
tuning datasets, we explain this to the difference between their pre-training stages. Firstly, all these languages are outside the pre-training data of BLOOMZ-3B, the declines for BLOOMZ-3B are easy to understand. Regarding Instruct-XMIL data and show similar composition distribution. As shown in Table 1 m 7B is trained with twice as many tokens than Instruct-XMM XMLR and learns more robust language knowledge. As a result, Dnstruct-XMLR is inferior to m.T0337BB in the ability of language generalization.
and m these languages are all the pre-training
6.2 GENERATION TASKS
As mentioned in numerous previous works (Gu et al.) 2018 Xiao et al. 2022). the non-autoregressive models remove the target-side dependency dur- ing training. They will result in a perfor- mance drop in generation quality com- pared with the traditional autoregressive models. As Instruct-XMLR also adopts the non-autoregressive manner to generate texts, this performance decline still exists. What's 's more, unlike the generative pre mre-trainin of baseline models, the pre-training paradigm of the BERT family is not designed for language generation, leading to difficulty in learning well just in the instruction tuning process. To evaluate the generation ability of InStruct-XMMLR, we choose WMT' 14 trans- lation datasets, which are widely used in non-autoregressive works. Table S presents the results. We use SacreBLEU (Post 2018) as our evaluation metric for all translation directions. We split the dataset into two subsets whose source and target language is English (EN-X and X-+NN). We report the average performance of prompts. Besides we also report the speedup by computing the number of tokens per second when translating the texts. As T0-3.7BB is the lowest, the correspond- ing speedup 1.0x. Results show that although Instruct-XMLRR. can speed up the decoding by around 3 times that baselines due to the non-autoregressive manner, the performance drops seriously compared with mO03.7BB.
Models
EN
X
X
+
EN
Speedup
BLOOMZ-3B
4.15
8.30
1.1x
mT0-3.7B
9.41
22.82
1.0x
Instruct-XNe Porruct-MMLR..
5.72
13.51
3.1x
Instruct-XMLRRy--M
8.83
18.31
3.0x
Table 5: Results machine translation tasks.
https:://giituub..omm///////
8

---

Published as a conference paper at ICLR 2024
We further analyze the main internal reasons: non-autoregressive generation manner the pre- training paradigm. As mentioned in (Muennighoft et al. 2022). our fine-tuned dataset contains much more short target texts for training. Thus, Instruct-MMLRy still can not learn superior generation abilities. As a result, we aim to enhance the learning on generation tasks during the instruction tuning process. Specifically,
random sample a subset containing all translation pairs but keep the language distribution the same as the original one. We get Destruct-XMLRxte -MT after fine-tuning on this purer instruction dataset with only a quarter updates steps of Instruct-MMLRxx We can witness a significant performance improvement, indicating that the pre-training paradigm is more decisive. Therefore, necessary to enhance the learning on generation tasks for the BERT family; even a suitable generative post-training process is optimal before instruction tuning.
6.3 UNRELIABLE PREDICTED LENGTH
As mentioned in section 5.1 we should transform the label space by designing the prompts for several tasks. However, we also rec- ognize that this can not be adapted to all tasks in practical appli- cations. For example, it unrealizable for a language modeling task whose targets have different lengths of various samples. Be- sides, length prediction also seems not suitable since the length of each sample is determined. Therefore, we conduct quantitative experiments to analyze the effects of length prediction various tasks with determined target length. Specifically, multiple choice questions task with options (A, B, C, D}, denoted as setting A; (2) the specific XWinograd task mentioned above with label space (True, Fals se} without transforming the label space, denoted as setting B; (3) a language modeling task LAMBADA (Paperno et al. 2016), where the target words be generated are with various length, this is more challenging setting, denoted as setting C. Table 6 presents the results. We report the accuracy of length prediction (the column Accuracy) and the performance gap compared with using fixed target length or the ground length (the column Gap). We can find that: length prediction is unreliable when meeting challeng- ing testing scenarios. For setting A, since the model may predict the incorrect length, e.g., one as two, the result cab be {C:} or [C.). which is equal to the label {C}. For settings B and C, length prediction almost fails, leading to a seriously declining performance. Therefore, we look forward to more robust mechanism to determine the target length for non-eetureee models.
Settings
Accuracy
Gap
A
0.87
0.02
B
0.42
33.2
C
0,05
24.3
Table 6: Effects on length.
select the following evaluation settings: (1)
6.4 FEW-SHOT PROMPTING LEARNING
Few-shot prompting, also known as in-context learning (ICL), is first mentioned in GPT-3 (Brown jtt al. 2020) and allows the LLMss to learn specific abilities to solve different tasks with only a few examples concatenated in the demonstration. Then, L.MSs can perform well in various without up- dating model parameters. ICI. has become a mainstream method to boost the performance of various LLMs. Unfortunately, Instruct-XMLA has not yet demonstrated this ability during our experiments. We also concatenate several examples simply before the test example. However, this setting con- fuses the model and results in disorganized predictions. We attribute this the bi-directional nature of BERT family since ICL has only been proven effective in autoregressive unidirectional language model. Therefore, we need more efforts to explore this and leave it future work.
CONCLUSION
This paper thoroughly explores the potential and limitations of BERT family as instruction followers. We find that our fine-tuned Instruct-XMLE can also generalize well across tasks and languages compared with current popular LLMS of model architecture with comparable model parameters. Besides, we also witnessed several limitations through our experiments, such as the performance decline of cong-generation task and failure in few-shot prompting learning. We further analyze the reasons and point out that some of the problems are possibly owing to the limited capabilities of the backbone models since there are no previous BERT family models that are as powerful as the decoder-only and encoder-decoder counterparts yet. Therefore, we hope that researchers can pay more attention to BERT family, and make them become competitive members in the family of current large language models.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We would like to thank the anonymous reviewers for their constructive comments. This work was supported by the National Science Foundation of China (NSFC No. 62206194), the Natural Science Foundation of Jiangsu Province, China (Grant No. BK20220488), and the Young Elite Scientists Sponsorship Program by CAST (Grant No. 2023QNRC001).
REFERENCES
Stephen H Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, et al. Promptsource: An in- tegrated development environment and repository for natural language prompts. arXiv preprint arXiv:2202.01279, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020..
Ying-Hong Chan and Yao-Chung Fan. A recurrent bert-based model for question generation. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pp. 154-162, 2019.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li. Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling isstuction-finetunce language mod- els. arXiv preprint arXiv:2210. 1416, 2022.
Kevin Clark, Minh-Thang Luong, Quoe V Le, and Christopher D) Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint vrXiv:2003.1055. 2020.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguisties, 2018.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer and Veselin Stoyanov. Unsuper- vised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440-8451, 2020.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tumed Ilm, 2023, URL httph::t///// wwww www.:ddaaaaaccccccoo wttp:/ccc//o//o/ccceeeee 12/dolly-first open wwwpeencoomeciallllllo
viabliillleee inn.ruttiiinttttiiiiooioo.oooo tuned 11mm
Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centere machine translation. arXiv preprint arXiv:2207.04672, 2022.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long and Short Papers), pp. 4171-41888 2019.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuee Wuen Hon. Unified language model pre-training for natural language understanding and generation. Advances in Neural Information Processing Systems, 32, 2019.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask- decoding of conditional masked language models. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural I Language Processing (EMNLIP-IJCNIPP) pp. 61 6112-6121, 2019.
Parallel
10

---

Published as a conference paper at ICLR 2024
Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, and Alexis Conneau. Larger-scale trans- formers for multilingual masked language modeling. arXiv preprint arXiv:210S.00552, 2021.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non-autoregressive neural machine translation. In International Conference on Learning Representations, 2018.
Junliang Guo, Zhirui Zhang, Linli Xu, Hao-Ran Wei, Boxing Chen, and Enhong Chen. Incor- porating bert into parallel sequence decoding with adapters. Advances in Neural Information Processing Systems, 33: 13843-051885,, 2020.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhaneee bert with disentangled attention. In International Conference on Learning Representattons, 2020.
Tianyu He, Xu Tan, Yingee Xia, Di He, Tao Qin, Zhibo Chen, and Tie- Yan Liu. Layer-wise co- ordination between encoder and decoder for neural machine translation. Advances in Neural Information Processing Systems, 31, 2018.
Zhengfu He, Tianxiang Sun, Kuanning Wang, Xuanjing Huang, and Xipeng Qiu. Diffusion- bert: Improving generative masked language models with diffusion models. arXiv preprint arXiv:2211.15029 2022.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint 22X1v:2203.15555 2022.
Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, and Wei Xu. Neural CRF model for sentence alignment in text simplification. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 7943-7960. Association for Computational Linguistics, 2020.
Ting Jiang, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang. Liangjie Zhang, and Qi Zhang. Improving non-autoregressive generation with mixup training. arXiv preprint a 10.221011111 11151, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2014.
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. Advances in Neural Information Processing Systems, 35:3 11 1809-3333 3826, 2022.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting promoting objective function for neural conversation models. arXiv preprint 150.03055 2015.
Xiaobo Liang. Juntao Li, Lijun Wu, Ziqiang Cao, and Min Zhang. Dynamic and efficient inference for text generation via bert family. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2883-2897, 2023a.
Xiaobo Liang, Zecheng Tang, Juntao Li, and Min Zhang. Open-ended long text generation via masked language modeling. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 223-242333 241, 2023b.
Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, et al. Taskmatrix. ai: Completing tasks by connecting foundation models with millions of apis. arXiv preprint arXiv:2303. 2023c.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. CommonGen: A constrained text generation challenge for generative common- sense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020,
November 2020.
Chin- Yew Lin and Eduard Hovy. Manual and automatic evaluation of summaries. In Proceedings of the ACL-022 Workshop Automatic Summarization, pp. 45-51, 2002.

---

Published as a conference paper at ICLR 2024
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1XXX::197..1199/ 2019.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301. 00233011.688. 13688, 2023.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong. Hailey Schoelkopf, et al. Crosslingual gen- eralization through multitask finetuning. arXiv preprint arXiv:2211.01755 2022.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1797- 1807, 2018.
OpenAl. Gpt-4 technical report. 2023.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 2022.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint 160v::1606.00331, 2016.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Ko- rhonen. Xcopa: A multilingual dataset for causal commonsense reasoning. arXiv preprint arXiv:2005.000 00333, 2020.
Matt Post. A call for clarity in reporting BLEU scores. Proceedings of the Third Conference on Machine Translation: Research Pupers, October 2018.
Lihua Qian, Hao Zhou, Yu Bao, Mingxuan Wang, Lin Qiu, Weinan Zhang, Yong Yu, and Lei Li. Glancing transformer for non-autoregressive neural machine translation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pp. 1993-2003, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020a. URL http:
//iml..oro/pppprs//// wtwppaaarrs.alaaaaaaaaaaapp wttp:rsss/i//2000001 20- html
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020b.
Alexander M Rush. Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 379-389, 2015.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, An- toine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint 1 2021.
12

---

Published as a conference paper at ICLR 2024
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.175800 2023.
Emily Sheng and David Uthus. Investigating societal biases in poetry composition system, 2020.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
Yixuan Su, Deng Cai, Yan Wang, David Vandyke, Simon Baker, Piji Li, and Nigel Collier. Non- autoregressive text generation with pre-trained language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pp. 234-243, 2021.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction- following model. Stanford Center for Research on Foundation Models. https://crfm. stanford. calu2023/03/1//ulpaes himl, 2023.
James Thome,, Andreas Vlachos, Christos Christodoulopoolls, and Arpit Mittal. FEVER: large-
scale dataset for fact extraction and VERification. In NAACL-IILT, 2018.
Alexey Tikhonov and Max Ryabinin. It's all in the heads: Using attention heads as a baseline for cross-lingual transfer in commonsense reasoning, 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation 2023.
language models. arXiv preprint
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Alex Wang and Kyunghyun Cho. Bert has a mouth, and must speak: Bert as a markov random field language model. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pp. 30-36, 2019.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabshhi, Yeganch Kordi, Amirreza Mirzaci, An- jana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, al. Super-natuuralinstuuttiossss Generalization via declarative instructions on 1600+ nlp tasks. arXiv preprint Arviv:2204.000 077705, 2022.
Yue Wang, Lijun Wu, Juntao Li, Xiaobo Liang, and Min Zhang. Are the bert family zero-shot learners? a study their potential and limitations. Artificial Intelligence, pp. 103953, 2023.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Lc. Finetuned language models are zero-shot learners. arXiv preprint (xXiv:2009001653 2021.
Yisheng Xiao, Lijun Wu, Junliang Guo, Juntao Li, Min Zhang, Tao Qin, and Tic-yan Liu. A sur- vey non-autoregressive generation for neural machine translation and beyond. arXiv preprint arXiv:2204.09226, 2022.
Yisheng Xiao, Ruiyang Xu, Lijun Wu, Juntao Li, Tao Qin, Tie- Yan Liu, and Min Zhang. Amom: adaptive masking over masking for conditional masked language model. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 13789-13797, 2023.
Jiacheng Yang, Mingxuan Wang. Hao Zhou, Chengqi Zhao, Weinan Zhang, Yong Yu, and Lei Li. Towards making the most of bert in neural machine translation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 9388-9385, 2020.
13

---

Published as a conference paper at ICLR 2024
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Per- sonalizing dialogue agents: have dog. do you have pets too? In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistic, pp.
2018
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey, arXiv preprint arXiv:2308. 10792, 2023.
Xuandong Zhao, Siqi Ouyang, Zhiguo Yu, Ming Wu, and Lei Li. Pre-trained language models can be fully zero-shot learners. arXiv preprint urXiv:2212.069555 2022.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305. /206, 2023.
Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin. Wengang Zhou, Houqiang Li, and Tieyan Liu. Incorporating bert into neural machine translation. In International Conference on Learning Representations, 2019.
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1
PROMPTS
We provide the prompts adopted in our main experiments here.
Task
Index
Prompt Template
Suppose [[peemise]) Can we inter that "(hhpaateeiss))" Yes, mayhe?
[premise]]] } Are we justimied in saying that "(hypothhsiis) Yes, no, or mayhe?
XNLI
3
Given that premise Does it follow that ([hypohhesis] J" Yes, no, or maybe?
+
Given that [[premise]] ] Therefore, it mmst he tree that "[ "[Utppothesis] ] Yes, no, or mayhe?
{premise) }} Rased on the previous passage, mue that "([ypathosis)) 33"? Yes, no, or mayhe?
(premise)) ]] Select the most plansible effacticmse: ((Option 1]] [[Opxion 21)
{ (premiiee) I As comsoquenee... This happened because... Help mo pick the more plausiblc optio:: ([Option 13) (:Opion 2))
XCOPA
1
{(premisee) } hesitating herween two options. Help me choose the effecticnsse: [(Option 1}} ((Oprion 2}}
+
[premise)] We are looking effecticass,, what best option? [[Option 1]] [[Option 2]}
[[Option 13) [[Option 2)) [[reemise] ] becousefen:
(
[premise J in sentence refers [[Opiin 12) Yes No? [[premise] The the sentence
2
(premiss)) } What
((Option 13} or {(Option 2}3?
XWinogra
abowe refers [[Option Yes No?
What does the, stand for the helow sentence! [[premise]] ((Opiio [[premise }} Replace
(OOgiion 2}}
the above sentence with the option: [Option [[Option
1
text in {(rc-lang]]: }: [Sonrce) ]} The same text in [fg--bng]
(2)
Transaate the following text from [(re-hung] } to t Higt-bog] ((Source) [[Souree]
14
Transtation
④
the same text in [[grrlang)]]:
{[Source) previous in {(rr- lang))
the Text {(ro-lang)): (Source) Transtation of
((tgt-bagg)):
15
((tgt-bmng))
Table 7: Different prompts adopted in our experiments. Texts within quotes denote the input quences. {(sre-lang)) and {[tgt-lang)) denote the source and target languages.
A.2 RESULTS
As we only report the average performance in the main body of our paper, we present different results based on the different prompts and different languages splits here.
Models
Prompts
EN
FR
PT
ZH
Jp
RU*
0.511
0.530
0.475
0.474
0.452
0.454
0.548
0.482
0.525
0.536
0.490
0.524
BLOOMZ.31 3B
0.557
0,530
0,502
0.567
0.485
0.498
0.501
0.494
0.483
0.524
0.503
0.540
0.529
0.609
0.533
0.568
0.651
0,551
0.583
0.560
0.527
0.575
0.566
0.574
0.575
0.569
0.571
mTM-1B
0.619
0.566
0.597
0.585
0.603
0.590
0.582
0.502
0.560
0.546
0.615
0.590
0.608
0.629
0.597
0.568
0.553
0.518
0.532
0.554
0.550
0.470
0.563
0.619
0.570
0.607
0.539
0.505
Instruct-XMLLRs
0.567
0.566
0.548
0.627
0.565
0.517
0.568
0,602
0.567
0.647
0.592
0.502
0.637
0.578
0.589
0.631
0.600
0.546
Table 8: Results of conference resolution task on XWinograd datasets. The language with is not contained in fine-tuned dataset.
15

---

Published as a conference paper at ICLR 2024
EN->X
X-+EN
Models
Prompts
FR
HI
DE
CS
RU
FR
HI
DE
CS
RU
!
10.12
2.61
0.70
0.19
0.28
8.15
5.46
4,91
2.15
3.77
32.65
6.57
2.64
0.16
0.48
33.13
17.55
15.18
2.57
13.97
BLOOMZ-3B
7.99
1,93
0.53
0.23
0.29
8.17
5.44
4.59
2.24
3.55
7.07
1.79
0.43
0.24
0.25
6.76
4.67
4.12
1.95
3.19
20.98
3,47
1.56
0.14
0.54
24.32
12.56
8.92
1.59
8.57
24.9
28.78
24.35
27.33
9.58
7.01
4.26
31.52
27.28
28.05
27.85
mT003B
24.53
6.72
4.64
23.98
22.12
8.32
7.01
5.03
28.34
24.00
24.37
23.42
11.78
3.73
5.65
19.55
10.36
17.76
15.91
14.63
4
14.97
7,03
5.23
2.57
0.75
19.59
13.16
15.83
14,94
7.62
12.26
4.83
2.67
2.18
0.73
18.57
12.18
14.62
14.05
1.80
Patruct-XMLRgg
15.84
7,00
5.20
2.49
0.73
20.53
13.72
16.52
15,38
8.27
14.99
7.16
5.42
2.69
0.77
17.79
12.29
13.81
13.34
5.89
12.73
6,51
4.99
2.53
0,76
19.33
12.84
14.92
14.03
6.79
16.53
7.75
21.88
13.35
17.83
16.32
20.76
9.52
23.71
Instoct-XMLLxxMMT
16.23
7.94
16.42
7.74
8.16
5.47
3.51
17.69
16.53
15.34
16.60
7.82
7.59
3.44
4.06
22.13
15.33
18.64
17.36
18.65
Table 9: Results of machine translation task on WMT' datasets.
Prompix
FN
VI
111*
0.482
11540
0.510
0.514
0.908
0.418
0.437
0.463
0.477
0.398
0.408)
0.404
01:37
01377
0.357
0443
0.512
0.501
0.515
(1:6.2
0.419
0.429
(1466
0.490
0.411
0.410
0.40%
0.639
0.439
0.161
BLOOMZ
0.488
0.505
0.435
0.535
0.536
0.516
0.508
0.380
0.369
0.356
(
0.518
0.568
0.542
0.538
0.498
0.492
0.499
0.531
0.506
0.535
0.533
0.53
0.505
0.521
0.477
0.509
0.552
0.548
0.542
0.902
0.406
0.467
0.532
0.506
0.42
0.588
0.5%
0.51
0.500
0.505
0.599
0.535
0.578
0.513
0.583
0.556
0.514
0.547
0.544
0.560
0.565
0.55
0.599
(1)
0.515
0,566
0.547
0,541
0.504
0.894
0.488
0.530
0.513
0.45
0.504
0.43
0.455
0.407
0,500
11559
0.545
0.529
0318
0.508
Ieinsct-MMB....
0.554
0.530
0.529
0.521
0.515
0.455
0.417
0.497
0.502
0.538
0.332
0.535
0.507
0.45
0.485
0.522
0.504
0.473
0.505
0.45
0.477
0.426
0.500
Table 10: Results natural language inference task on XNLI datasets.
Models
Prompts
ID>
SB
VI
7H
ET
HT*
IT
QU
TH*
TR*
0.74
0.51
0.76
0.76
0.43
0.53
0.47
0.46
0.48
0.62
0.44
0.53
0.68
0.67
0.46
0.44
0.43
0.46
0.50
BLOOMZ-3B
0.65
0.47
0.70
0.49
0.38
0.55
0.42
0.54
0.42
0.47
0.25
0.76
0.61
0.66
0.69
0.74
0.69
0.68
0.70
0.49
0.69
0.71
0.77
0.57
0.70
0.67
0.79
0.68
0.70
0.70
0.50
0.71
0.69
mT0-3B
0.72
0.58
0.69
0.67
0.76
0.72
0.67
0.68
0.45
0.73
0.68
0.75
0.59
0.66
0.68
0.74
0.70
0.65
0.67
0.48
0.69
0.70
0.40
0.36
0.39
0.39
0.33
0.44
0.45
0.35
0.34
01.42
0.38
0.82
0.58
0.71
0.75
0.74
0.68
0.57
0.73
0.58
0172
0.68
0.73
0.60
0.71
0.75
0.70
0.71
0.57
0.70
0.63
0.71
0.64
Instrict-MMLRxt
0.66
0.61
0.67
0.71
0.69
0.62
0.54
0.68
0.58
0.66
0.68
0.77
0.62
0.67
0.76
0.68
0.70
0.54
0.65
0.63
0.65
0.64
0.43
0.53
0.40
0.40
0.47
0.43
0.47
0.37
0.41
0.42
0.39
Table 11: Results of sentence completion task on XCOPA datasets.
16

---

Published as a conference paper at ICLR 2024
A.3 THE PERFORMANCE VARIANCE OF DIFFERENT PROMPTS.
Since we adopt five different prompts for each task, analyzing how stable these models perform across these different prompts is essential. As result, we present the performance variance of dif- ference prompts here. Table 12 Table and Table presenn the results. We report the performance range, i... the performance boundary, show the performance variance of difference prompts. We can find that: (1) The performance difference across prompts of Dustruct-XMLRR is nearly compared with two baseline models in all tasks and even smaller in XCOPA, XNLI, and machine translation tasks, indicating that our method does not bring instability across prompts. (2) The performance difference XCOPA is relatively big, but this is the case for all models. Instruct-XMLR performs better than other models.
Models
XCOPA
XNLI
XWinegrad
BLOOMZ-3B
0,519 #00289
0.482 +00026
0.523 +00033
mT0-3.7B
0.627 1.0255
0.528 1..016
0.584 10.000
Instruct-XMLRRr
0.646 #:0200
0.518 1:0000
0.582 100043
Table 12: Accuracy on three held-out tasks.
Models
XCOPA
XNLI
XWinograd
BLOOMZ-3B
0.427 10.165
0.391
10.019
0.499 10.06
mT0-3.7B
0.605 #0.208
0.533
+0.021
0.569 +00028
Instruct-XMLRRy
0.599 18.184
0.466 +0.007
0.539 +0.034
Table 13: Results of language generalization.
Models
EN + X
X ->> EN
BLOOMZ-3B
4.15 14.35
8.30 18.18
mT0-3.7B
9.41 14.70
22.82 17.18
Instruct XMLRxx.
5.72 #1.19
13.51+137
Iyst
8.83 11.82
18.31 15.02
Table 14: Results of machine translation tasks.
A.4 EVALUATION ON MORE TASKS AND DATASETS.
Beyond the tasks shown in 4.2 we also evaluate more tasks and datasets. Table 15 and Table 16
present the corresponding results.
Tasks and Datasets. We evaluate our models in more tasks, including language generation and understanding. For language generation, except the machine translation task we have included in Table 5] we select various tasks with different characteristics, e.g., text summarization task which has relatively long sequences, text simplification and paraphrase generation tasks whose inputs and targets have the similar expression, controllable generation task which requires the model to gener- ates texts meeting certain controllable constraints as humans wish reliably, and dialogue generation task which has relatively flexible outputs. For text summarization task, we use XSUM (Narayan) et al. 2018) and Gigaword (Rush et al. 2015). which contain the online articles and single sentence summary pairs from the British Broadcasting Corporation and summary from Gigaword corpus, re- spectively. For text simplification task, we adopt WIKI-AUTO (Jiang et al.) 2020) which contains aligned sentences from English Wikipedia and Simple English Wikipedia. We adopt Quora Question Pairs (QQP) for paraphrase generation task, where each question pair is semantically equivalent. For controllable generation task, we adopt COMMONGEN (Lin et al. 2020) in which the model needs to generate coherent sentence using the given common concepts. For dialogue generation task,
17

---

Published as a conference paper at ICLR 2024
Text Summarization
XSUM
Gigaword
Rouge-1
Rouge-L
Rouge-1
Rouge-L
Bloomz-3B
35.12 11132
27.97 19.35
31.45 10.60
28.94 10.04
m
37.85 00.4
30.28
10226
34.31 +0.18
31.54 10.10
Instruct-XMLRy
35.62 =0.22
27.90
=0.23
30.64
#0.11
28.52 #0.21
PettructXMMRRx w/GL
36.00 =0.19
27.84 10.10
32.60 +0.55
29.85 10.00
Text Simplification
Paraphrase Generation
WIKI-AUTO
QQP
Bleu-2
Distinct-2
Bleu-2
Distinct-2
Bloomz-3B
41.90 =6.2
0.57 10.03
38.24
10.79
0.54 +0.28
mTT033.BB
55.92 1.83
0.64 +0.01
51.93
10.39
0.63 =0.08
Instruct- Cestruct-XMIRRe XMLL
47.91 1.85
0.65 10.01
43.50
+2.49
0.67 +0.01
Instruct-XMLRR. w/GL
56.93 22.00
0.76 +0.00
60.02 +0.72
0.69 +0.01
Controllable Generation
Dialogue Generation
COMMONGEN
PersonaChat
Bleu-2
Distinct-2
Bleu-2
Distinct-2
Bloomz-3B
35.20 =0.26
0.44 +0.01
16.39 +0.04
0.39 #0.01
mTO03.7B
31.86 00.4
0.36 10.04
9.17 0.02
0.39 +0.01
Instruct-XML.R
21.68 0085
0.52 #0.02
2.88 0004
0.52 +0.03
w/GL
35.44 1.03
0.48 10.01
17.30 +0.01
0.53 +0.01
Table 15: Results various language generation tasks.
Fact Verification
Sentiment Analysis
FEVER
SST-2
Poem
Accuracy (%)
Accuracy (%)
Accuracy (%)
Bloomz-3B
81.23 +4.67
95.40 #0.30
18.67 #2.33
mTO-3.7B
83.47 +3.37
82.30 +1.50
14.67 +1.67
Instruct-XMIL. Instruct-XMLRRt w/GL
88.10 +120
92.20 #0.60
25.33 #0.67
88.10 1.20
92.20 +0.60
25.33 #0.67
Table 16: Results on various language understanding tasks.
adopt PersonaChat (Zhang et al. 2018), which contains around 1,0k data triples formatted as (pro- file, conversation, response). For language understanding tasks, we select FEVER (Thoree al., 2018) for the fact verification task, which aims to verify the correctness of textual claims against textual sources. We also use SST-2 (Socher al., 2013) and Poem (Sheng Uthus (020)) datasets to evaluate the sentiment analysis task, which analyzes the sentiment information given the specific sentence.
Evaluation Metrics. We utilize various evaluation metrics for these tasks. For text summarization task, we adopt ROUGE Fl score (Lin & Hovy 2002) following previous works. For other language generation tasks, iee. text simplification, paraphrase generation, controllable generation, and dia- logue generation, the n- u-gram level precision and the diversity of the generated texts. For language understanding tasks, report the corresponding accuracy.
adopt BLEU (Papineni al. 2002) and Distinct (Li et al.) 2015) to measure
18

---

Published as a conference paper at ICLR 2024
Results. As shown in Table 15 and Table 16 except the performance of standard Instruct-XMMLR we also report the performance of Instruct-XMLR w/GL which decodes that we use ground truth length rather than the predicted length during inference. We aim to analyze the effects of length pre- diction for different tasks. We can mainly find that: (1) Instruct-XMI.R is more suitable for language understanding tasks. It outperforms m'OO and BL BLOMMZ. in most testing scenarios (except underper- forming BLOOMZ on sst-2). (2) We can notice two groups for language generation tasks according the effects of length prediction. The length prediction usable for text summarization, text simplification, and paraphrase generation tasks. The performance in these tasks of Instruct-XMI. outperforms BLOOMZ most testing scenarios but underperforms mTO. This is the same case for the machine translation task as shown in Table 4 in the main body. (3) For other language genera- tion tasks, i.e., COMMONGEN and PersonaChat, although Instruct-XMI. can achieve promising performance with ground truth length, adopting length prediction results in significant performance declines. This is easy to understand, i.c., these tasks have no strong alignment relationship between source length and target length. Hence, Instruct-XMLL fails to predict the reliable length based on the source representation. More reliable alternative solutions should be explored to determine the target length for these tasks in the future. (4) The Distinct score of Instruct-XMLL is superior in all generation tasks, indicating that Instruct-XMLR can generate more diverse texts for all text generation tasks. We attribute this to the non-autoreereesivee generation (Mask-Predict algorithm) generation manner where the dependency of different tokens is released compared with adopting the standard next token prediction algorithm in traditional autoregressive models.
A.5 MORE EXPLORATION FOR FEW-SHOT PROMPTING LEARNING
As mentioned in Section 6.4 Instruct-XMLR has not yet demonstrated the ability of few-shot prompting during our experiments. Since this capacity is first noticed in GPT-3 (Brown al.. 2020), current researchers believe that few-shot prompting learning arises in the pre-training stage and benefits from its generative pre-training paradigm. However, the different pre-training paradigm of instructXMMLE (i.e., simply predicting the masked tokens without generative paradigm) fails to equip the capacity of few-shot prompting learning. As a result, We have tried to make up for this flaw by including some few-shot instruction data during training rather than the only zero-shot instruction data we adopted in our main experiments. Unfortunately, this also doesn't 't bring any profit. Table 17 and Table present the corresponding results. We find that adopting few-shot prompting learning even harms the overall performance.
Models
XCOPA
XNLI
XWinograd
Instruct-XMLRyg w/ zs
0.646
0,518
0.582
w/ ICL 4
0.548
0.414
0.547
CostructXMMIRR w/ fs
0.642
0.489
0.565
w/ ICL4 4
0.571
0.442
0.553
Table 17: Accuracy on three held-out tasks. w/ zs denotes the model trained on zero-shot instruction data, w/ fs denotes the model trained on mixed zero/few-shot instruction data, w/ ICL denotes that we adopt 4 examples in the demonstration for few-shot prompting learning.
Models
XCOPA
XNLI
XWinograd
Instruct-XMLRyt w/ zs
0,599
0.466
0,539
w/ ICL 4
0,521
0.397
0.506
Dastu w/ i's
0,591
0.451
0,528
w/ ICL 4
0.536
0.419
0.514
Table 18: Results of language generalization. w/ zs denotes the model trained on zero-shot instruc- tion data, w/ fs denotes the model trained mixed zero/few-shot instruction data, w/ ICL 4 denotes that we adopt examples in the demonstration for few-shot prompting learning.
19

---

Published as a conference paper at ICLR 2024
A.6 RESULTS OF XML-R WITHOUT INSTRUCTION TUNING
To analyze the effects of instruction tuning for our model, we compare Instruct-XMLR and the original pre-trained XML-R without instruction tuning. However, it is difficult to collect the evaluation results of XMLR-w/ instruction tuning owing to its poor performance on these tasks. We provide a few representative examples for intuitive understanding in Table 19 We can find that XML-R fails to complete the task without instruction tuning, i.c., it always generates the special tokens [PAD] without actual meaning XCOPA, XNLI, and WWinograd tasks and generates texts unrelated to the specific prompt in machine translation.
impossible
Task
Type
Output
Source
Mata saya menjadi dan bengkak, Select the plaasible Saya terizagis. B:
TTrret
XCOPA
Instruct-MLRRss
XML- RXL
LEAD]
Source
Supposs "And he said, Mama, I'm home"" Can we infer that "He dida' it say word."
Target
XNLI
InstuuctXMLLRst
No
XML.RRa.
[PAD]
Source
the help
given.
sentence above refers or
XWinograd
Target
Yes
Yes
XML-RxL
[PAD]
Source
A Gemma: Kinder brauchen Stabilitii und Sicbetheit. The English:
Target
Children need stability and curtainty.
MT
Instruct-MMLL XMLRax
Children need safety and security. Pragen und Antworien zum.
XML-Rsa.
Source
Translate the following lest from German Englishr. Auch Fast Food erlamt.
Target
Fast food is also permitted.
Inltnus-MML.R. Ryt.
Fast food acceptable.
Translate page:
Table 19: Examples of XML-R with/without instruction tuning. MT denotes machine translation.
20


Reviews:
[{"review_id": "aUAhFGU9vD", "reviewer": "Reviewer_fL5N", "summary": "The authors adopt XML-R to explore the effectiveness of the BERT family for instruction following and zero-shot learning. They first design a simple yet effective strategy to utilize the encoder-only models for generation tasks and then conduct multi-task instruction tuning. Experimental results demonstrate that our fine-tuned model, Instruct-XMLR, outperforms Bloomz on all evaluation tasks and achieves comparable performance with mT0 on most tasks. Besides, Instruct-XMLR can accelerate decoding due to its non-autoregressive generation manner.", "questions": "Have you tried to use a larger model?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The idea is interesting and novel. It is great to know BERT family can also do instruction tuning. It can potentially lead to another series of research. \n2. The proposed method shows competitive performance compared to auto-regressive methods. And it can achieve 3 times speed up. \n3. The proposed method is simple and effective.\n4. The paper is well-written and easy to read.", "weaknesses": "1. The method is not tested on longer sequence generation. The classification tasks don't need a longer output sequence. And machine translation fits the non-auto-regressive methods. Need to test it on the tasks like dialogue, summarization, etc. \n2. It would be better to analyze how stable the method is. How do the prompt templates affect the performance. Need to report the variance or significant test.\n3. The proposed method cannot handle fewshot prompt learning.\n\nOverall, although the exploration is quite interesting, the limitations are also obvious and need further exploration. It would also be interesting to see negative results on a broader range of tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r52NZfx8Bc", "reviewer": "Reviewer_oFHF", "summary": "This manuscript studies the instruction following capabilities of the BERT-family of language models, featuring an encoder-only architecture, which is one of the first of its kind. The work proposes a series of approaches that have made this possible, which eventually results in Instruct-XMLR (from XML-R), revealing promising task and language generalization abilities, previously undiscovered.", "questions": "It's a fascinating topic to explore beyond decoder-only autoregressive models. Would the authors agree that the benefit in text generation mainly comes from the next-token prediction formulation itself, and not necessarily the decoder architecture? One such example may be [1].\n\n[1]: RWKV: Reinventing RNNs for the Transformer Era", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This is quite an interesting work with a refreshing perspective on language modeling via encoder-only architecture. I really appreciated the author's detailed analysis of relevant works, especially the mention of the work Wang & Cho (2019), which appears to be crucial to this work. The approaches, and training processes are detailed and well-motivated, and the benchmarks are extensive.", "weaknesses": "1. Some minor issues regarding citation formats (e.g. \"Following the previous work Muennighoff et al. (2022), we evaluate the\nmodel’s ability of task\" is not directly readable, may need to include parenthesis)\n2. As the authors have pointed out, the text generation capabilities, one of the arguably most important capabilities of language models, is still weak for Instruct-XMLR. This may limit the significance of this work.\n3. Also arguably not a deal-breaker, but the compute resource (8xA100, unclear 40GB or 80GB, PCIe or SXM) and the model size (3.5B params on 0.6B tokens) may still be too small compared to cutting-edge decoder-only equivalents.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "12kVokdri2", "reviewer": "Reviewer_bkXV", "summary": "We've been witnessing the rise of instruction-tuned language models since T0, FLAN, NaturalInstructions, Alpaca, etc. These models typically have a decoder-only (Alpaca) or encoder-decoder (T0, FLAN) architecture. This paper wants to study the question of whether encoder-only models such as the BERT family, can also be instruction-tuned and exhibit zero-shot task-generalization abilities. \n\nTo do so, they first develop a new MLM training scheme that mimics the Seq2Seq training paradigm by concatenating source and target sequences to feed into BERT but with proper masking so that the source sequence tokens won't attend to the target sequence tokens. When training, they mask some tokens from the target sequences to do MLM training. During inference, they first predict the length of the sequence and then iteratively predict masked tokens (all tokens are [MASK] at the beginning). \n\nThey then did instruction tuning on the 3.5B XML-R backbone model with the multilingual xP3 dataset. The performance is somewhat comparable to baselines like BLOOMZ-3B and mT0-3.7B, although still lagging behind on language generalization, generation tasks, few-shot learning, etc.\n\n\n----------------------------------------------------------\nPOST-REBUTTAL UPDATE:\nI'm raising my score from 5 to 6.", "questions": "- Why don't you also include an XMLR without instruction-tuning baseline so that we can directly see the relative improvement coming from instruction tuning? \n\n- I am a bit confused by the method description in Section 3.2. Initially, you were describing it as if you train a separate \"decoder\" MLM, and H_{src} and H_{tgt} are from the two different models. But what I think you are actually doing is that, you just concatenate the source and target sequence and feed into the same BERT model, but with a modified masking scheme, such that source sequence tokens cannot attend to target sequence tokens.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- I think the experiments are pretty solid. \n- The writing is generally clear and well-organized.", "weaknesses": "- I'm not exactly sure what's the core contribution of the paper - it reads too much like an experiment report in this current draft. My biggest takeaway is that you can also do instruction tuning on encoder-only models. But why is that particularly \"surprising\"? To be fair, I'd be surprised it actually works much better than decoder-only or encoder-decoder models because then it means that the popular approach is wrong and they should go back to encoder-only models instead. But based on your experiments, it's not the case, decoder-only and encoder-decoder models still seem better at similar sizes. \n\n- The experiments are not exactly controlled in the sense that, your baselines - BLOOMZ-3B and mT0-3.7B, are not even based on the same backbone model as Instruct-XMLR. This means that many confounders (e.g., pretraining data / steps) exist for making any scientific conclusions based on the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)