BERTs are Generative In-Context Learners
Authors: David Samuel
Venue: NeurIPS 2024 poster (2024)
Keywords: in-context learning
masked language modeling
bert
language modeling
evaluation
inference
Primary area: natural_language_processing
TL;DR: This paper explores the in-context learning capabilities of masked language models, challenging the common view that such abilities are only present in causal language models.

Abstract: While in-context learning is commonly associated with causal language models, such as GPT, we demonstrate that this capability also 'emerges' in masked language models. Through an embarrassingly simple inference technique, we enable an existing masked model, DeBERTa, to perform generative tasks without additional training or architectural changes. Our evaluation reveals that the masked and causal language models behave very differently, as they clearly outperform each other on different categories of tasks. These complementary strengths suggest that the field's focus on causal models for in-context learning may be limiting – both architectures can develop these capabilities, but with distinct advantages; pointing toward promising hybrid approaches that combine the strengths of both objectives.

BERTs are Generative In-Context Learners
David Samuel
Language Technology Group
University of Oslo
davisamu@uio.no
Abstract
While in-context learning is commonly associated with causal language models, such as GPT, we demonstrate that this capability also 'emerges' in masked language models. Through embarrassingly simple inference technique, we enable an existing masked model, DeBERTa, to perform generative tasks without additional training architectural changes. Our evaluation reveals that the masked and causal language models behave very differently, as they clearly outperform each other different categories of tasks. These complementary strengths suggest that the field's focus on causal models for in-context learning may be limiting both architectures develop these capabilities, but with distinct advantages; pointing toward promising hybrid approaches that combine the strengths of both objectives.
Introduction
Masked language models used dominate the field of natural language processing due to their adaptability across diverse tasks and their superior performance compared to causal language models (Radford et al., 2018; Devlin al., 2019). Between 2018 and 2020, the field witnessed surge in the development of these models (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020, inter alia). However, the field dramatically shifted with GPT-3 and its introduction of in-context learning the ability to infer and perform tasks from prompts and examples without any finetuning (Brown et al., 2020). This capability eliminated the need for task-specific training data and deep-learning expertise, making such models far more practical for real-world applications. This perceived advantage led many researchers and practitioners abandon masked language models in favor of GPT-style architectures.
Scaling of in-context learning performance (1-shot)
DeBERTa
GPT-3
Language understanding
Language modeling
Translation
Question answering
Average performance
65
50
60
15
35
50
50
0.1
0.8
2.6
13
175
0.1
0.8
13
175
0.1
2.6
175
0.1
0.8
2.6
175
Parameters (B)
Parameters (8)
Parameters (B)
Parameters (8)
Figure 1: The average 1-shot performance across four groups NLP tasks scaling abilities of DeBERTa (four sizes in red) with GPT-3 (eight sizes in blue). Even though these models rely on different training objectives, they scale a similar lin-linear manner overall. Yet, on task- by bask basis, the pretraining methods lead to substantial differences between them.
We compare the
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Previous studies of "emergent' in-context learning abilities have focused almost exclusively on causal language models, creating a widespread assumption that this capability is unique them (Saunshi et al., 2021; Olsson et al., 2022; Wei et al., 2022; Wang et al., 2023, inter alia). In this paper, we challenge this assumption by demonstrating that in-context learning can emerge in masked language models as well. In-context learning is more general phenomenon and should not be studied with singular pretraining objective in mind. Moreover, the assumed inability of masked language models to perform (generative) in-context learning has rendered them outdated et al. (2023): "BERT-style models are very restricted in their generative capabilities. Because of the classification heads, we strongly do not recommend using this class
explicitly noted by Tay
cumbersomeness of of autoencoding models moving forward and consider them somewhat deprecated."
In this paper, we challenge these prevailing assumptions about masked language models (MLMs). We present empirical evidence showing that DeBERTa, an ML.M released just one month after GPT-3, is equally adept at in-context learning. Our findings suggest that the capacity for in-context learning not tied the training objective, but can be achieved across different types of language models. To our surprise, we found that DeBERTa does not simply mimic the performance of GPT-3 the two model behave very differently - DeBERTa is clearly much better on tasks such as language understanding, and, on the other hand, much worse on tasks such as closed-book question answering. This suggests that masked and causal language modeling are two complementary training objectives and that there is great potential for training method that combines the strengths of both objectives. Finally, scaling (performance improvement with increased size of pretrained language models) is crucial feature of modern language models; demonstrate that MLMs do scale in-context learning (Figure 1).
We introduce a simple inference technique that transforms an MLM into generative model without any further training. Using publicly available DeBERTa checkpoints, we show that the MLM training objective not only provides versatile way of encoding text, but is also competitive in text generation and text completion ranking. This claim is tested by following the same evaluation suite as GPT-3, speculating on an 'alternative reality" in which masked language model is the first model reported to achieve the so-called 'emergent' in-context learning abilities. While other masked language models could potentially demonstrate similar capabilities, we deliberately target DeBERTa because of its large size and its mongth-generalizttion abilities. Ultimately, our goal is to demonstrate that MLMs can perform in-context learning and that they can be surprisingly good doing so.
Outline First, Section 2 (Method) describes the inference methods used to evaluate the in-context learning abilities of an off-the-shelf masked language model. Then Section 3 (DeBERTa) describes the details of the particular model used in this study. Section 4 (Evaluation) details the evaluation setup and compares DeBERTa with GPT-3. Finally, Section 5 (Related work) talks about other relevant work within this topic, and the paper concludes with Section 6 (Conclusion).
Text generation
Ranking (pseudo log-likelihood)
F'n BEXT and
MASK
HOUSK
AIP
CEC
Haute
HouSE
Jeaues
talk
aze
logP
I'n
10.22
BEXT
Houod
JOCP
CLU
House
Jesuac
House
cie
couk
log
SEAT
I'm BEXT
DEST
lug (akk
Figure 2: Illustration of the proposed methods for using a masked language model for text generation and text ranking We show how to adapt a masked language model for in-context- learning tasks through simple input reformatting, requiring no additional training. LEFT: Text generation is achieved by 1) appending [MASK] tokens to the input prompt, 2) predicting the next token for the first mask, and 3) iteratively appending new masks and predicting tokens. RIGHT: A similar approach is used to retrieve a peeudo-log-lieelhhee used to rank multiple sequences by their individual likelihoods. Both methods maintain the model's original architecture while enabling new capabilities through careful input formatting.
score of a text sequence that can be

---

2 Method: text generation and ranking with masked language models
The goal of this article is reuse existing pretrained masked language model for (generative) in-context learning. We achieve this without any additional training or finetuning, our method only slightly changes the sequence of input tokens, as illustrated in Figure 2. There are two methods used to solve tasks with in-context learning: text generation where the model completes a given prompt (e.g. for translation) and ranking where the model chooses an answer from several options (e.g. for multiple choice questions).
2.1 Text generation
Masked language models are trained semi-supervised fill-in-the the-blanks tasks and so they cannot be used to generate straight out of the box. One possibility is to interpret these models as Markov random fields and produce text by Gibbs sampling (Wang and Cho, 2019). However, simpler and more consistent way to produce text to do the familiar left-to-right autoregressive generation we could place [MASK] token next to text prompt and let the model generate next token by unmasking the appended token then, when we repeat this process in a loop, we can generate text in the same way causal language models (and apply the same advanced generation techniques).
This straightforward inference scheme would be enough if the pretraining process were designed with this use case in mind. However, since our goal is repurpose an existing masked language model, we have to complicate the method with two modifications that are also illustrated in Figure
1. Masked language models are typically trained with special end-of-sequence [SEP] token. This token always present during pretraining and so we also have to include it as the last token during inference.
2. However, the addition of this end-of- -e--uene token creates a problem - raises the probability that the masked token should end the sequence (for example with full stop). Thus, in order to obtain a less restricted continuation, we include additional [MASK] tokens pad the space in front of the mad-offsequenne token. Specifically, we use two additional masks for the DeBERTa models." This decision later ablated in Appendix B.
In the end, this approach gives probability distribution over the next token prediction, thus we can use any existing method for searching or sampling an output sequence. We follow GPT-3 and use beam search with four candidate beams for all generative tasks.
Limitations While this method works with the same quadratic time complexity (in sequence length), it is slower in practice because it is not possible to cache the intermediate self-attention key and value vectors. Instead, these have to be recomputed every step due to the bidirectional nature of the model. While our current implementation prioritizes demonstrating the core capability over optimization, several promising approaches could address these computational limitations future work. For example, using prefix language modeling selectively updating hidden vectors could significantly improve efficiency. We leave these optimizations for future work to maintain focus on establishing the fundamental ability of MLMs to generate text.
2.2 Ranking
Many of the existing tasks for evaluating LLMs can be formulated as classification tasks where models have to select the correct answer from a number of different options. Brown et al. (2020) rank the candidate completions based on their estimated conditional likelloodo,,,, which can be computed exactly by the chain rule (where wy @ w1 wi is completion of prompt e):
log PPw @ wT ......u wk (c) log P(w; |c + wy ..... .o....-11) w)-1)
(1)
While this equation matches the training objective of causal language models, it is not suitable for masked language models because they are not trained to estimate P(w; | wp... u;-1). Instead,
Note that this is not an arbitrary number but it is model -seeifipecciiic- DeBERTa models were pretrained to unmask spons of masked tokens where the longest allowed spans three tokens long (He al., 2021).
3

---

Wang and Cho (2019) proposed to modify Equation (1) to make it more appropriate for BERT- like models. Salazar et al. (2020) then empirically showed that the resulting geeudo-lo-likeliinee score can be used to accurately rank text sequences by their likelihood. Specifically, the PIJ score is approximately proportional to the conditional probability of a text sequence and is computed as:
(PLL)
log P(w @ w1 wk (c) X
log P(w, + wp... = @ [MASK] + W.+1.... wkk
(2)
However, this approximation gets very inaccurate when there are strong local dependencies between tokens. As a counterexample, the estimated likelihood of the multi-token word 'superealifragilstiic expialidocious' is seven orders of magnitude higher than that of the single-token word 'super', which clearly an incorrect estimation of the relative frequencies of these words.
We improve on this behavior by interpolating between the mathematically correct unidirectional derivation in Equation (1) and the bidirectional approximation in Equation (2). Our approach is to simply mask two additional tokens in the right context to reduce the effect of local dependencies while still taking into account the global bidirectional context. This process is illustrated in Figure 2. We conduct an ablation study of this approach in Appendix C.
Limitations Even though Equations (1) and (2) look similar, the later sum is substantially more compute intensive when calculated with transformer architecture for token sequence of length k, the first equation can be computed with passing single sequence through language model, while the second equation needs k sequences to be processed. However, Salazar et al. (2020) showed that the PLL score can be accurately estimated in a single pass after a short sulf-supervised finetuning.
2.3 Length generalization
potentially limiting factor of using BERT-like models is that they are typically pretrained shorter sequences than causal language models - arguably because the training of modern causal models is already optimized for in-context learning, which requires processing of long few-shot prompts. DeBERTa is not an exception to such pretraining: it was only trained with relatively short maximum sequence length of 512 tokens (He al., 2021). Fortunately, the architecture of DeBERTa can easily process much longer sequences than seen during training due to its use of relative positional embeddings with logarithmic buckets (Raffel al., 2020).
OPT
DeBERTa
Pretralning
Length gensraliation
Pretraining
Length generalization
Noodle postonn
62
33
49
0.5
06
28
06
3.0
255
381
261
2024
15:96
2245
1012
K1.0C 0132
12:204
255
381
THE
2024
15/6
2043 1012 4006 01.0% 2132 17:204
Sequance
Sequence
Figure 3: Length generalization measured with a 'needle a haystack' benchmark The z-axis indicates the total size of the 'haystack' and the y-axis indicates the position of the 'needle'; the values show the average exact-match accuracy for a particular configuration. Unfortunately. GPT-3 is closed-source model and the original version is not accessible, of GPT-3, OPT by Zhang et al. (2022). which should perform similarly on this task because of the the same transformer architecture GPT-3. In particular, it uses absolute positional encoding, which strictly limits any model from generalizing to longer inputs than trained on.
use open- replication
"This is because the tokenizer splits the long word into 9 subwords and each of them is assigned an almost certain likelihood given the bidirectional context. The largest 1.5B DeBERTa estimates the pseudo log- likeelhodood of the first word to -2.1 while the second (very common) word has pseudo log- likelihood of -9.6.

---

We measure the extent to which DeBERTa generalizes to longer sequences with the 'needle in haystack' test from RULER (Hsich et al., 2024). Specifically, in our formulation of this task, random 6-digit number (needle) is hidden in a long collection of essays (haystack). We then measure the exact-match accuracy of retrieving the hidden number given two variables: the total sequence length and the position of the needle in the haystack (more details about the evaluation setup are given Appendix E.I).
The results in Figure demonstrate that DeBERTa generalizes to sequences well beyond its training length, which is enabled by its relative positional encoding. For comparison, we also show results from OPT (Zhang et al., 2022), which uses absolute positional encoding like GPT-3. As expected from models using absolute positional encoding, performance drops sharply beyond the training length. This comparison highlights the importance of positional encoding choice for length generalization, independent of whether the model masked or causal. In practice, this observation means that DeBERTa should be able to handle as many task demonstrations as models trained with longer sequences.
3 DeBERTa family of language models
This study uses the largest openly available English masked language model, DeBERTa with 1.5 billion parametrs, and its smaller configurations - O.IB 0.4B and 0.9B (He et al., 2021). DeBERTa is an improved version of BERT language model (Devlin et al., 2019) that uses an advanced attention mechanism with relative positional embeddings apart from being trained on a larger corpus and with larger number training steps.
Training corpus Compared to GPT-3 and modern large language models, DeBERTa was pretrained on a relatively small and clean text corpus totalling 78GB of data after deduplication, the corpus is comprised of the English Wikipedia (12GB), BookCorpus (6GB; Zhu al., 2015), Open WebText (38GB; Gokaslan and Cohen, 2019), and STORIES (3 (31GB; Trinh and Le, 2019). This is almost an order of magnitude less data than what was used to pretrain GPT-3. Notably, our strong results despite this data disparity could suggest that masked language models are more data-efficient than causal models for developing in-context learning capabilities. This claim would however need to be evaluated with comprehensive study. In comparison, GPT-3 uses 570GB of filtered CommonCrawl. WebText2 (roughly 26GB), two web-scraped book corpora (roughly 17GB and 76GB), and the English Wikipedia (roughly 4GB, estimated from Brown et al. (2020)).
Total training compute Interestingly, even though DeBERTa uses substantially smaller training corpus, it trained on more than three times more tokens than GPT-3 (1 trillion compared to 300 b However, the loss is computed only on 15% of tokens (150 billion) and it is not clear what would be the effective number of tokens used for pretraining. Nevertheless, the total compute used for training depends on the number of input tokens and it is roughly 8.0 1021 FLOPs for the 1.5B DeBERTa, and 2.4 1 FLOPs for the 1.3B GPT-3.
Causal conversion for HuggingFace We have converted the officially available DeBERTa check- point into HuggingFace (Wolf et al., 2020) implementation of JutoModelForrausalll (following the method in Section 2.1), and released it openly at https://hf http://ff../ottg/////// wttpo////////e////ee xxlarge- fixed. The weights of this model are exactly the same as the official release from nicrosoft/debers v2 v2- xxlarge but we have fixed some bugs found in the original modeling script in addition to implementing the text generation abilities. Similarly, we have also converted the smaller De- BERTa models and released them as Ltg/deberta base- fixed, ltg/deberta: large- fixed, and Ltg/deberta taleerree--l---e----- fixed.
'This means that DeBERTa was trained on dozens of repetitions of its training corpus, not unlike other popular masked language models (Devlin et al., 2019; Liu et al.. 2019) suggesting that this type of models operates under different 'training laws' than causal language models (Muennighoff et al., 2023).
'Specifically: 1) incorrect name of the output embedding weights in the checkpoint file, 2) non-functioning implementation of the enhanced mask decoder (EMD), and 3) missing truncation of the relative positional indices.
5

---

Evaluation
As our goal is to compare two language models released around the same time in 2020 - GPT-3 and DeBERTa we replicate the evaluation setup used for GPT-3 (Brown et al., 2020) and apply it to the latter model. This also means that we follow GPT-3 and divide the tasks into generative ones (such as machine translation) and into classification tasks (such as BoolQ) the first group uses the method described in Section 2.1 and the second type of task uses the ranking described in Section 2.2. Generation is performed with beam search (4 candidate beams), and ranking uses the modified F(completion context) PLL scores (and the normalized unconditional probability of completions Picompletion answer context) for ARC and OpenBookQA), again replicating the choices for GPT-3). We also use the exact same prompt templates, with the exception of the machine translation task its template did not produce any meaningful output, and so we decided to use the simple prompt template from Garcia et al. (2023) instead. More details on the evaluation setup can be found in Appendix E. Note that using prompts optimized for GPT-3 is slightly unfair to all other models, as prompting has strong influence on performance (Gonen al., 2023), but we believe that makes the results more convincing than if we were to do extensive prompt engineering.
To show the strengths and weaknesses of DeBERTa in (generative) in-context learning, we evaluate it on four groups of tasks and compare to the results from Brown et al. (2020). The four groups are language understanding (SuperGLUE), language modeling (text completion and Winograd-like tasks), machine translation, and question answering (closed-bol bboo question answering and commonsense reasoning). We detail each of these groups of tasks below.
Before looking into the details of each group, we show the overall aggregated scores for each group in Figure and Figure 4. The first figure shows how the performance of both models scales with their size, while the latter figure compares the in-context learning abilities of the two language models. We also provide qualitative evaluation of text generation in Appendix A and full results in Appendix F.
The average 0-shot, 1-shot and few-shot performance
DeBERTa
GPT-3
Language understanding
Language modeling
Translation
Question answering
75
Avcrage performanc
25
70
75
65
70
15
60
65
5
0
SeW
0
New
tew
0
few
Number of shots
Number of shots
Number of shots
Number of shots
Figure 4: The performance improvement with increased number of in-context examples We compare the in-context learning ability of 1.5B DeBERTa (in red) with 1.3B GPT-3 (in blue) using prompts without any completed examples (0-shot), prompts with a single randomly sampled gold sample (1-shot), and prompts with few examples (4 64 examples, depending figure demonstrates that a masked language model behaves similarly to a causal language model in the in-context learning regime. More detailed few-shot evaluation is in Figure
the task). This
4.1 Language understanding (SuperGLUE)
We use SuperGLUE (Wang et al., 2019) as a popular collection of standard NLP tasks, allowing us to evaluate the performance on different aspects of natural language understanding.
In total, this benchmark consists of eight datasets, selected to be difficult for the contemporary (finetuned) language models. The Boolean Questions dataset is yes/no reading comprehension dataset evaluated with accuracy (BoolQ: Clark et al., 2019); CommitmentBBnk a three-class textual entailment dataset evaluated with accuracy and F, score, where the multi-class F, is computed as the unweighted average of the F, per class (CB; de Marneffe et al., 2019); the Choice of Plausible Alternatives dataset is causal reasoning task evaluated with accuracy (COPA; Roemmele et al.,

---

2011), Multi-Sentence Reading Comprehension is multiple choice dataset, evaluated with exact- match (of all answers per question) accuracy and Fio score computed over all flattened answers (MultiRC; Khashabi et al., 8); Reading Comprehension with Commonsense Reasoning Dataset is another reading comprehension dataset, is evaluated with the exact-match accuracy and token-level Fi score (ReCoRD; Zhang et al., 201 8); the collection of Recognizing Textual Entailment datasets a textual entailment task evaluated with accuracy (RTE; Dagan et al., 2006; Bar-Haim al., 2006; Giampiccolo et al., 2007); the Word-in-Contes dataset is word sense disambiguation dataset evaluated with accuracy (WiC; Pilehvar and Camacho-Colladon 2019); and finally, the Winograd Schema Challenge evaluates coreference resolution capabilities (WSC; Levesque et al., 2012).
Results We show the resulting scores of evaluation with the same prompts as GPT-3 in Table and Appendix D. DeBERTa clearly outperforms its contemporary and scales much more favorably than the family of GPT models (Figure 1). Interestingly. the average performance gets close to the reported performance of the largest 175B GPT-3 (68.4 4 vs. 68.9, I-shot). However, this average score still far from the performance of a finetuned DeBERTa, which more than 20 percentage points higher (He et al., 2021); the average few-shot performance of DeBERTa is slightly better than finetuned BaRe-large (Devlin al., 2019; Wang et al., 2019).
the 1.5B DeBERTa
Table I: Natural language understanding results All results in this table are evaluated with accuracy (higher is better). The table shows the performance of the largest available DeBERTa (1.4 billion parameters) and of similarly GPT-3 model, the best results are boldfaced. The average score is calculated over averaged task scores (in case a task uses more than one metric).
BoolQ
CB
COPA
MultiRC ReCoRD
RTE
WIC
WSC
Average
O-shot
GPT-3
62.4
19.6
77.0
13.6
84.1
56.0
50.0
61.5
55.9
DeBERTa
80.8
66.1
78.9
6.6
87.1
64.3
50.5
71.2
65.4
1-shot
GPT-3
63.7
48.2
74.0
13.6
83.0
49.5
49.2
62.5
57.8
DeBERTa
82.1
76.1
84.2
15.6
87.4
64.1
50.3
69.6
68.4
few-shot
GPT-3
64.1
69.6
77.0
20.8
83.1
50.9
53.0
49.0
60.0
DeBERTa
82.1
75.0
90.4
16.9
87.4
62.2
50.8
75.0
69.6
4.2
Language modeling, Winograd-style and text completion tasks
The tasks in this category defined in familiar language-modeling form and focus partic- ularly difficult cases of language modeling, cases that involve commonsense reasoning, language understanding, and intricate coreference resolution.
In this group, we consider four NLP tasks: HellaSwag is a text completion task where a language model has choose the most appropriate multi-word ending, the examples are adversarially filtered to be difficult for language models but easy for humans (Zellers et al., 2019). Story consists of five-sente cee-seneence-long stories, the goal is to select the best final sentence based on commonsense knowledge (Mostafazadeh et al., 2016). Winograd is a language language-modeling modeling formulation the WSC task from SuperGI.UE (Levesque et al., 2012). WinoGrande is similar Winograd in its form, but is adversarially mined to contain more difficult examples of coreference resolution (Sakaguchi et al., 2020). We do not include the LAMBADA benchmark (Paperno al., 2016) here because Brown et al. (2020) used an unknown preprocessing step that disallows direct comparison with GPT-3.
Results We show the in-context-learing results from this group in Table 2, where we evaluate the 1.5B DeBERTa with a comparable GPT-3 model. The scores showcase consistently stronger performance of the masked language model, similarly to the language understanding tasks. One difference those tasks the rate of scaling, which appears be similar between the two types of language models (Figure 1).

---

Table Results of text completion, language modeling and Winograd-style tasks All tasks are measured with accuracy, we show the performance of the largest available DeBERTa (1.4 billion parameters) and of a similarly-sized GPT-3 model, the best results are boldfaced.
HellaSwag
StoryCloze
Winograd
Winogrande
Average
0-shot
GPT-3
54.7
73.4
76.9
58.7
65.5
DeBERTa
62.0
83.6
74.0
70.2
I-shot
GPT-3
53.5
74.2
76.9
59.1
65.9
DeBERTa
62.4
84.6
80.7
63.6
72.8
few-shor
GPT-3
54.9
76.1
76.9
59.1
64.8
DeBERTa
62.5
84.8
85.6
68.8
75.4
4.3 Translation
Translation is a useful benchmark for language models as it evaluates their ability to understand text in language and produce fluent text in another language. Even though the performance on the translation tasks is arguably very dependent on the composition of training data (especially when we concerned with monolingual English models), include translation demonstrate the generative performance of masked language models.
To directly compare DeBERTa with GPT-3, use the same SacreBLEU metric (Post, 2018) and the same bitexts. Thus, even though there are more recent (and arguably more thought-out) datasets, we use the French-English pair from the outdated 2014 shared task at the Workshop Statistical Ma- chine Translation (WMT14; Bojar et al., 2014), and also the Romanian-Enge English and German Engllish pairs from the WMT16 workshop (Bojar al., 2016). Our approach differs only in using a different prompt template, as we had to opt for the prompt from Garcia et al. (2023) to get consistent transla- tions: "(Ssource language): ($source text]\\n (Starget language): (Starget text)".
Results The SacreBIEU scores on each language pair are given in Table 3. Unlike in the previous two task groups, the tables have turned, and the causal language model clearly outperforms the masked model all comparisons. We believe that the subpar performance of DeBERTa can be (at least) in part explained by its relatively small and clean monolingual training corpus (Section 3), because the performance on this task highly dependent on the presence multilingual data the corpus (Lin al., 2022). The rate of improved translation performance with larger scale appears to be similar between the two models (Figure 1).
Table Machine translation results We report SacreBLEU scores (Post, 2018) with signature BLEU+case mixed=numrefs. 1+smooth. eosttootteeeetok........ intl+version. 1. 2. 20 (higher is better). The table shows the performance of the largest available DeBERTa (1.4 billion parameters) and of a similarly- sized GPT-3 model, the best results boldfaced.
DE EEN
EN-+DE
FR-EN
EN-
RO RO-+EN
EN-RO
Average
0-shot
GPT-3
3.6
2.4
3.6
2.8
3.6
3.1
3.2
DeBERTa
2.4
1.6
1.7
0.3
1.7
0.1
1.3
I-shot
GPT-3
25.8
13.4
27.0
19.3
26.8
10.3
18.8
DeBERTa
23.7
5.4
23.5
9.7
17.7
2.5
13.8
few-shot
GPT-3
30.5
17.7
32.2
26.1
30.1
12.9
24.9
DeBERTa
25.1
6.6
24.5
10.8
18.9
4.1
15.0
8

---

4.4 Closed-book question answering and commonsense reasoning
important quality of modern-day large language models their ability to learn and retrieve world knowledge, and to have degree of common sense. The final group of tasks attempts to evaluate these two qualities.
This category of tasks consists of seven datasets in total: Natural Questions (NQs; Kwiatkowski et al., 2019) and Web Questions (WebQs; Berant al., 2013) are closed-book inestton-anssweee datasets sourced from natural web queries; while the original datasets are accompanied by relevant articles that contain the answer, we only ask models question and then evaluate the exact-match accuracy of their answers. TriviaQA is very similar dataset, but based on online quizzes (Joshi et al., 201 7). The next four tasks fall more into subcategory of commonsense reasoning datasets. The Physical Interaction: Question Answering dataset evaluates how well language model is grounded in the real physical world (PIQA; Bisk et al., 2020). The A12 Reasoning Challenge is dataset sourced from grade-school science questions that evaluates knowledge and reasoning abilities; this task is divided into ARC-Easy and ARC-Chan IRC-Chaleenge splits, based on their difficulty (Clark al., 2018). Finally, OpenBookQA evaluates the understanding of common knowledge (Mihaylov et al., 2018).
The question-answeringg performance is given in Table 4. Apparently, the results of DeBERTa substantially worse closed-book question answering compared GPT-3. We believe that this highlights a more general disadvantage of the MLM training objective - the model can often retrieve world knowledge from the rich bidirectional context during training, not needing to language models
Results
store it in its learned weights; similar effect has been shown in retrieval-augmennt (Samuel al., 2024). However, the commonsense reasoning abilities are comparable between the two models. The scaling behavior is again similar between the two models (Figure 1). The same is also true about the improvement when given more in-context examples, which are especially important for the tasks evaluated with exact-match accuracy, where the goal is not only to answer correctly but also match the expected style and form of the gold answers (Figure 4).
Table 4: Closed-book question answering and commonsense reasoning The first three tasks are measured with the exact-match accuracy and the rest measured with classification accuracy. The table shows the performance of the largest available DeBERTa (1.4 billion parameters) and of similarly-sized GPT-3 model, the best results are boldfaced. A detailed description the evaluation method is given in Appendix E, full results are in Appendix F.
Open- BookQA
NQs
TriviaQA
WebQs
PIQA
ARC-C
ARC-E
Average
O-shot
GPT-3
4.4
19.7
4.6
75.1
35.5
53.8
46.8
34.4
DeBERTa
0.8
72.9
36.5
55.1
45.8
I-shot
GPT-3
5.4
26.5
9.2
74.4
36.4
55.9
46.4
36.3
DeBERTa
2.6
14.3
5.1
73.0
37.1
55.1
45.7
33.3
few-shot
GPT-3
9.7
32.1
19.6
74.3
36.7
59.1
50.6
40.3
DeBERTa
4.4
17.9
9.9
74.5
39.6
57.7
50.4
36.3
Related work
Few-shot finetuning with masked language models While our work demonstrates the emergence of in-context learning in masked language models, prior research has explored different approaches few-shot learning with these architectures. The dominant paradigm has been few-shot finetuning, where the model's weights are updated using a small number of examples. Studies by Schick and Schüizze (2021), Gao al. (2021), and Xia al. (2022) showed promising results with this approach. However, these methods require additional training steps with complicated training objective,

---

making them more complex to implement compared to the simple prompting based in- in-context learning demonstrated in our work. Despite the generally lower performance of in-context learning compared to few-shot finetuning (Liu et al., 2022), its simplicity and immediacy have made it the preferred choice many practical applications.
Other large masked language models Our choice of DeBERTa for this study was motivated by its unique combination of size and capability to handle extended context lengths. While larger masked language models exist, such as Megatron BERT with 3.9 billion parameters (unfortunately not publicly available; Shoeybi et al., 2019) and XoBMRRoBERTa with 10.7 billion parameters (Goyal et al., 2021). they have limitations that make them less suitable for studying in-context learning. Megatron BERT lacks mechanisms for length generalization, which is crucial for processing long prompts with multiple examples, while XLM 4LM-RoBERTa's 's multilingual nature and restricted sequence length of 512 tokens would confound our analysis. DeBERTa's ' architecture, particularly its relative positional embeddings, makes ideal candidate for exploring how masked language models scale with in-context learning.
Hybrid masked-causal models Our empirical findings, particularly the complementary strengths of masked and causal models demonstrated in Section 4, suggest significant potential in combining these approaches. Several architectures have already explored this direction, even if inadvertently: T5 (Raffel et al., 2020), BART (Lewis et al., 2020) and GLM (Du et al., 2022) introduced autoregressive fill-in-the- thee-llnkkkk objectives; CM3 developed causal mask approach (Aghajanyan et al., 2022); and PrefiXLM implemented a partially bidirectional causal model (Dong et al., 2019; Raffel et al., 2020). These efforts align with observations about the distinct advantages of masked and causal objectives. The recent work by Ding et al. (2024) provides theoretical support for this direction, demonstrating that prefix language models, which combine aspects of both architectures, are particularly well-suite for in- in-context learning.
6 Conclusion
This paper demonstrates that masked language models can be capable in-context learners. We show that these models- often considered deprecated and limited only to finetuning can match and sometimes even exceed the performance of their causal counterparts in this domain. Our evaluation reveals that masked and causal models exhibit remarkably similar characteristics in terms of overall performance, scaling behavior, and improvements with additional in-context demonstrations. Most notably, we validate these capabilities using DeBERTa without any architectural modifications or additional training. We achieve this through carefully designed inference methods that unlock the model's 's latent generative abilities.
Our findings point to several promising directions for future research. First, DeBERTa's performance could likely be enhanced through straightforward improvements such training on larger and more diverse corpora, increasing model scale, and extending the pretraining context length. More fundamentally, the complementary strengths where each architecture excels different tasks - suggest an exciting opportunity to develop hybrid approaches that combine the best of both paradigms. Rather than viewing these as competing architectures, future work might explore how to synthesize their distinct advantages into more capable and versatile language models.
observed between masked and causal models
These results argue for broader reconsideration of how we approach language model architecture and training. The field's recent focus on causal models, while productive, may have prematurely discounted the potential of alternative approaches that are not limited to unidirectional text processing. Our work demonstrates that the path forward likely involves embracing architectural diversity rather than converging on a single dominant paradigm.
10

---

Acknowledgments and Disclosure of Funding
I am deeply grateful to Lilja Øvrelid, Andrey Kutuzov, and Erik Velldal for providing insightful feedback, for their never-ending encouragement and support, and for making Oslo a warm and welcoming place.
This work is fully funded by the University of Oslo. The computations were performed on resources provided through Sigma2 the national research infrastructure provider for high-performance computing and large-scale data storage in Norway. We acknowledge Norway and Sigma2 for awarding this project access to the LUMI supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CSC (Finland) and the LUMI consortium through project 5000144.
References
Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, and Luke Zettlemoyer. 2022. CM3: A causal masked multimodal model of the internet. Preprint, N.Xiv:2201.07:2222
Roy Haim, Ido Dagan, Bill Dolan, Lisa Ferro, and Danilo Giampiccolo. 2006. The second PASCAL recognising textual entailment challenge. Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533-1544, Seattle, Washington, USA. Association for Computational Linguistics.
Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA:
Reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7432-7439
Ondiej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Kochn, Johannes Leveling, Christaf Monz, Pavel Pecina, Matt Post, Herve Saint- Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. Findings of the 2014 workshop statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12-58, Baltimore, Maryland, USA. Association for Computational Linguistics.
Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva. Christof Monz, Matteo Negri. Aurélie Névéol, Mariana Neves, Martin Popel. Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia. Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 131-198, Berlin, Germany, Association for Computational Linguisties.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Susan Carey and Elsa Bartlett 1978. Acquiring a single new word. Proceedings of the Stanford Child Language Conference, 15:17-29.
Christopher Clark, Kenton Lee, Ming-Wei Chang. Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 2924-2936, Minneapolis. Minnesota. Association for Computational Linguistics.
Peter Clark, Isaac Cowhey, Oren Etzioni. Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord, 2018, Think you have solved question answering? Try ARC, the A12 reasoning challenge. Preprint, arXiv: 1833:1803.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pages 177-190, Berlin, Heidelberg. Springer Berlin Heidelberg.
11

---

Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The CommitmentBank: Investi- gating projection in naturally occurring discourse. Proceedings of Sinn und Bedeutung, 23(2):107-124.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Pupers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Nan Ding, Tomer Levinboim, Jialin Wu, Sebastian Goodman, and Radu Soricut. 2024. CausalLM is not optimal for in-context learning. In The Twelfth International Conference on Learning Representations.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Zhengxiao Du Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang. and Jie Tang. 2022. GLM: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume Long Papers), pages 320-335, Dublin, Ireland. Association for Computational I Linguistics.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing ( (Volume I: Long Papers), pages 3816-3830, Online. Association for Computational Linguistics.
Xavier Garcia, Yamini Bansal, Colin Cherry, George Foster, Maxim Krikun, Melvin Johnson, and Orhan Firat. 2023. The unreasonable effectiveness of few-shot learning for machine translation. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10867-10878. PMLR.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACI-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1-9, Prague. Association for Computational Linguisties.
Aaron Gokaslan and Vanya Cohen. 2019. OpenWebText corpus.
Hila Gonen, Srini Iyer, Terra Blevins, Noah Smith, and Luke Zettlemoyer. 2023. Demystifying prompts in language models via perplexity estimation. In Findings of the Association for Computational Linguisties: EMNLP 2023, pages 10136-100 1014,, Singapore. Association for Computational Linguisties.
Naman Goyal, Jingfei Du, Myle Oit, Giri Anantharaman, and Alexis Conneau. 2021. Larger-scale transformers for multilingual masked language modeling. CoRR, 00000022..
Pengeheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTa: Denondig-enhanned BERT with disentangled attention. In International Conference Learning Representations.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations.
Cheng- Ping Hsich, Simeng Sun. Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. RULER: What's the real context size of your long-context language models? arXiiv preprint 240.2220040665444
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zeulemoyer. 2017. TriviaQA: large seale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611, Vancouver Canada. Association for Computational Linguistics.
Carina Kauf and Anna Ivanova. 2023. A better way to do masked language model scoring. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 925-935. Toronto, Canada. Association for Computational Linguistics.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay. and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long Papers), pages 252 262, New Orleans, Louisiana. Association for Computational Linguistics.
12

---

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield. Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein. Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew Dai, Jakob Uszkoreil, Quoc Le, and Slav Petrov. 2019. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics,
Zhenzhong I Lan, Mingda Chen, Schastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut, 2020.
ALBERT: A lite BERT for self-supervised learning of language representations. In International Conferencee on Learning Representations.
Hector J. Levesque, Ernest Davis, and Leora Morgenstem. 2012. The Winegrad schema challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, KR 12, page 552-561. AAAI Press.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising aequence-to-sequencc pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages 7871-7886 Online. Association for Computational Linguistics.
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale. Jingfei Du, Ramakanth Pasunuru. Sam Shleifer. Punit Singh Koura. Vishrav Chaudhary, Brian O'Horo, Jeff Wang. Luke Zettlemoyer, Zomitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. 2022. Few-shot learning with multilingual generative language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Longuage Processing, pages 9019-9089 905999952,444 Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Haokun Liu, Derek Tam, Muqeeth Mohammed, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022. Few-shot carameter-efficient fine-tuning better and cheaper than in-context learning. In Advances in Neural Information Processing Systems.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Dangi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. Preprint, arXiv: 11692.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018 Can suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381-2391, Brussels, Belgium. Association for Computational Linguistics.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839-849, San Diego California. Association for Computational Linguistics.
Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. 2023. Scaling data-constrained language models. In Thirty-seventh Conference on Neural Information Processing Systems.
Catherine Olsson, Nelson Elhage. Neel Nanda, Nicholas Joseph. Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatficld-Dodds, Danny Hemanndez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodci, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2022. In-context learning and induction heads. Preprint, arXiv:2209. 11895
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi. Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1525-1534, Berlin, Germany, Association for Computational Linguistics.
Mohammad Taher Pilehvar and Jose Camacho-Collae uating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 1267 1273, Minneapolis, Minnesota. Association for Computational Linguistics.
2019. WIC: the word-in-context dataset for eval-
13

---

Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Brussels, Belgium. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understand- ing by generative pre-training.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li. and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1).
Melissa Roemmele, Cosmin Bejan, and Andrew Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning, AAAI Spring Symposium Technical Report.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. WinoGrande: An adversarial Winograd Schema Challenge at scale. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):873228744)
Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages (899-27122 Online. Association for Computational Linguistics.
David Samuel, Lucas Georges Gabriel Charpentier. and Sondre Wold. 2024. More room for language: Investigating the effect of retrieval on language models. Preprint, (ttl::2444.19999
Nikunj Saunshi, Sadhika Malladi, and Sanjeev Arora. 2021. A mathematical exploration of why language models help solve downstream tasks. In International Conference on Learning Representations.
Timo Schick and Hinrich Schütze. 2021. It's not just size that matters: Small language models are also few-shot learners. In Proceedings the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339-2352, Online, Association for Computational Linguistics.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM:: Training multi-billion parameter language models using model parallelism. CoRR. abs/1909.08053.
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. 2023. UL2: Unifying language learning paradigms. In The Eleventh International Conference on Learning Representations.
Trieu H. Trinh and Quoc V. Le. 2019. A simple method for commonsense reasoning. Preprint, arXiv: 800.08877..
Alex Wang and Kyunghyun Cho. 2019. BERT has mouth, and it must speak: BERT as a Markov random field language model. In Proceedings of the Workshop Methods for Optimizing and Evaluating Neural Language Generation, pages 30-36, Minncapolis, Minnesota. Association for Computational Linguisties.
Alex Wang. Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. SuperGL.UF: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc.
Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. 2023. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. In seirty-seventh Conference Neural Information Processing Systems.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sehastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Fd H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy I Liang, Jeff Dean, and William Fedus. 2022. Emergent abilities of large language models. Transactions on Machine Learning Research. Survey Certification.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac. Tim Rault, Remi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu. Canwen Xu, Teven Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-at natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online. Association for Computational Linguistics.
14

---

Mengzhou Xia, Mikel Artetxe. Jingfei Du, Danqi Chen, and Veselin Stoyanov. 2022. Prompting ELECTRA: Few-shot learning with discriminative pre-trained models. Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pages 11351-11361, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Farhadi, and Yejin Choi. 2019. HellaSwag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-48000 Florence, Italy. Association for Computational Linguistics.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen. Shuohui Chen, Christopher Dewan Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: Open pre-trained transformer language models. Preprint, arXiv:2205.0106: 01068.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 19-27.
15

---

A Examples of text generation
To also give sense of the quality of the produced by DeBERTa, we include some examples of text generation in this section. We use the exact same example prompts that were used in the GPT-3 paper (Brown et al., 2020), provide a fair estimate of the generative qualities. All text completions were generated with nucleus sampling (Holtzman et al., 2020). 5 We compare the largest DeBERTa 1.5B with OPT 1.3B, an openly available replication of the GPT-3 1.3B (Zhang al., 2022).
A.1 Learning and using novel words
Based on studies in developmental linguistics (Carey and Bartlett, 1978), this task tests the ability to understand and productively use new words; specifically using a word in a sentence after seeing defined only once. We qualitatively test this ability in a generative one-shot setting, using the prompts provided below there, the human-provided prompts rendered normal while the generated completions are rendered in boldface. The prompts are taken from Brown et al. (2020, Section 3.9.5).
appropriate example sentences than OPT. While the 'farduddle' and 'screeg' sentences from DeBERTa are not very descriptive, the rest of sentences are informative and fitting the word definitions. Note how the model tried invent a plural inflection of 'yalubalu', the suffix '-a' is morphologically plausible, but the stem is fumbled, possibly because of subword tokenization. The examples generated by OPT are of lesser quality; it either repeats the definition (in 'farduddle", repeats the ohot example (in 'yalubalu') or provides an unfitting example (in 'screeg').
Results Overall, DeBERTa provides
"whatpu" a small, furry animal native to Tanzania. An example sentence < that uses word whatpu is: were traveling Africa and we saw these very cute whatpus.
To do a "farduddle" means jump up and down really fast. An example of sentence that uses the word farduddle is: did a farduddle.
DeBERTa:
OPT:
farduddle jumped up and down really fast.
"yalubalu" is type of vegetable that looks like big pumpkin. example of sentence that uses the word yalubalu is: We were in the market and
DeBERTa:
these very big yambolula.
OPT:
We were traveling in Africa and we saw these yalubalu.
"'Burring"" is a car with very fast acceleration. An example sentence that + uses the word Burringo
DeBERTa:
were driving very fast Burringo.
OPT:
We Burringos all over Africa this trip.
"Gigamuru" is type of Japanese musical instrument. An example of sentence that uses the word Gigamuru is:
DeBERTa:
Japanese musician, who plays the Gigamuru, is very famous.
OPT:
was walking down street and I saw this very cute gigamuru.
Usiin these hyperparameters: top k=64, top_p=0.9, temperature=0.2 22
16

---

To "screeg" something is to swing a sword at it, An example of sentence that uses + the word screeg is:
DeBERTa:
sword-screeg.
OPT:
screeging the computer.
A.2
Correcting English grammar
English grammar correction is another task used for qualitative evaluation of GPT-3. Provided three demonstrations, the model is given an incorrect English sentence and is tasked to correct it. The prompts are taken from Brown et (2020, Section 3.9.6) and are shown below.
Results While the corrections are not perfect, DeBERTa outputs more sensible completions com- pared to OPT, similarly to the previous task. A notable correction is in the second example, where DeBERTa tried to improve the understandability, not only grammar.
Poor English input: eated the purple berries.
Good English output: I ate the purple berries.
Poor English input: Thank you for picking me as designer. I'd appreciate it.
Good English output: Thank you for choosing me as your designer. I appreciate it.
Poor English input: The mentioned changes have done.
Good English output: requested changes been made.
Poor English input: I'd be than happy work with you in another project.
DeBERTa:
Good English output: I'd be more than happy to work with in another project.
OPT:
Good English output: I'd be more than happy to work with you in another project.
Poor English input: Please provide me with short brief of the design you're re + looking for and that'd be if you could share some examples or project you
- before.
DeBERTa:
provide with a brief design you looking and examples of your previous work.
English output:
OPT:
Good English output: I'll provide with brief of the design you' re looking for and that'd be nice if you could share some examples project did before.
Poor English input: The patient was died.
DeBERTa:
Good English output: The patient has died.
Good English output: The patient was revived.
Poor English input: We think that Leslie likes ourselves.
DeBERTa:
Good English output: think that Like ourselves.
OPT:
Good English output: think that Leslie likes herself.
English input: Janet broke Bill the finger.
DeBERTa:
Good English output:Janet broke Bill's 's finger.
OPT:
Good English output: Janet broke Bill on the finger.

---

Poor English input: Mary arranged for, in St. Louis, John to rent a house cheap.
DeBERTa:
Good English output: Mary arranged for, in St. Louis, John rent a house.
OPT:
Good English output: John rented house cheap.
B Ablation study of text generation
We empirically evaluate different approaches for text generation with DeBERTa language models. In particular, we study how many additional mask tokens should be used during autoregressive generation, and we also compare our approach with previously proposed methods based on Markov- chain Monte Carlo sampling (Wang and Cho, 2019). For this analysis, we use German-to-Ennlissh machine translation as representative generative task. We evaluate different generation methods using one-shot prompts (following Section 2.1) and the largest 1.5B DeBERTa model. The translation quality is measured with SacreBLEU score, using the same signature as in the main experiments.
The results shown Table demonstrate that using additional mask tokens during gen- eration substantially improves the performance of DeBERTa. This aligns with our hypothesis that additional masks help to reduce the effect of the end-of sequence token on the generated text. The results also show that while adding fourth mask token still marginally improves the performance, the gain is negligible compared to using three mask tokens.
Results
We also compare our autoregressive approach with the Gibbs sampling method proposed by Wang and Cho (2019). There we relied on the same hyperparameters that are suggested in the official implementation: 500 total iterations, 250 burn-in iterations with top-100 sampling and temperature of 1.0.6 Their sampling-ban -baed approach performs substantially worse than autoregressive generation, regardless of the initialization strategy. We noticed that it often produces infinite token repetitions or locally-coherent but globally-disconneete pieces text.
Table 5: Ablation study of different generation methods applied to DeBERTa Evaluated using one-shot setting and the largest DeBERTa 1.5B model on German-to-English translation with SacreBLEU score.
Generative method
DE-
Autoregressive generation: mask
10.0
Autoregressive generation; 2 masks
22.4
Autoregressive generation; 3 masks (our method)
23.7
Autoregressive generation; 4 masks
23.9
Markov-chain Monte- Carlo (random initialization; Wang and Cho, 2019)
1.6
Markov-chain Monte-Carlo (mask initialization; Wang and Cho, 2019)
2.6
C Ablation study of ranking implementation
We mentioned some drawbacks of calculating the pseudo-log likeliho-k score (PLL) as per Salazar al. (2020), and how we mitigate these problems, in Section 2.2. This section supports our decision with quantitative analysis of different ranking approaches. We test them on the ReCoRD task from SuperGI.UF (Zhang et al., 2018; Wang al., 2019), where the goal is to rank different named entities based their appropriatness. Since the problem of the original PLL is in estimating likelihoods of long multi-loken expressions, we choose this task to highlight the differences. An example of
Available GitHub: https://github... www.g/ttt...../yuuuuu//oeoo dt/bert- gen.
18

---

With bright lights illuminating his efforts from below, Mr ($answer: : Stephan Sanf} appears be on the set of prompt-formatted sample from ReCoRD is given below, the possible completions (all of which are named entities) are boldfaced:
Suspended hundreds of feet in the air amid glistening pillars of ice illuminated with ghostly lights from below, this could easily be a computer generated scene computer-generated
from the latest sci-fi blockbuster movie. But in fact these ethereal photographs taken in real life, and show Siegrist, 43, ascending the Voringsfossen icefall which is of gigantic glacier in Eidfjord, Norway. The stunning images were captured by fellow mountaineer and photographer Thomas Sanf. While the 500ft frozen waterfall is regularly scaled by climbers during daylight, he said he wanted to capture the beauty the falls by night.
sportsman and climber Stephan
Stunning images captured by photographer Thomas Sanf as climber Stephan Siegrist, 43, scaled frozen waterfall
The Voringsfossen fall is liquid for most of the year, but winter freezes into 500ft cliff favoured by climbers
Hundreds of adventurers attempt the climb by day, but very few attempt the ascent
b at night, as pictured here
sci-fi movie
Previous work After running the initial experiments, were informed about the related work by Kauf and Ivanova (2023). Their study addresses similar problem of the naive pseudo-log- ligeliieihoood scoring as this paper, but they do not target strongly correlated contiguous tokens in general, they focus on words that are split into multiple tokens as they observed that PLL overestimates the likelihood of those sequences. We include their two proposed solutions in the comparison of different ranking methods.
Results In Table we compare the original implementation of PLL by Salazar et al. (2020) that only masks the target subword; our approach that also masks next two subwords; other two alternatives that mask two four subwords total; the PLL-word- 12r PLL-whole word scoring functions by Kauf and Ivanova (2023); and the exact unidirectional computation of log-likelihood (using the same input formatting as for generation). The additional masks clearly help to make better estimates while the exact computation seems to not be appropriate for inherently bidirectional models.
Table 6: Ablation study of different ranking methods applied to DeBERTa zero-shot setting and the largest DeBERTa 1.5B model on ReCoRD.
Evaluated using
Ranking method
ReCoRD (EM)
ReCoRD (F₃)
Pscudo ligllikcl;hoo;; mask (Salazar et al., 2020)
80.9
81.6
Pseudo log- lig-like;;hood; 2 masks
86.0
86.8
Pseudo log-likelihoodk masks (our method)
87.1
87.9
Pseudo bogllikllhood; masks
86.9
87.8
Pseudo log- liellike;hhod; word 12r (Kauf and Ivanova, 2023)
78.2
78.8
Pseudo liellike;hood; worl (Kauf and Ivanova, 2023)
75.8
76.5
Exact log- logelikeldoood (unidirectional)
77.2
77.8
19

---

D Detailed SuperGLUE results
This appendix provides a detailed analysis of DeBERTa's 's performance on the SuperGLUE benchmark, focusing on two aspects: the relationship between the number of demonstration examples and model performance (Figure 5), and the task sssk-fpecific scaling behavior (Figure 6).
Effect of shot count Figure 5 demonstrates how DeBERTa's performance changes as we increase the number of in-context examples from 0 32. Unlike the main results where we optimize the number of shots for each subtask independently, here we deliberately use the same number of shots across all tasks to provide controlled comparison with GPT-3's results from Brown et (2020); there, SuperGLUU tasks are the only ones with such detailed few-shot evaluation.
The plot reveals several interesting patterns:
1. The steepest improvement occurs between 0 and shots, suggesting that even small number of examples provides substantial benefit.
2. The overall trend is very similar between DeBERTa and GPT-3, which again shows that masked language models can be just as good in-context learners as causal language models.
3. The performance DeBERTa decreases after 8 or more shots. We believe that this is mainly caused by its imperfect processing of contexts longer than 512 tokens as discussed in Section 2.3.
Average GLUE performance vs, number of shots
DeBER (1:4B)
GPT-3 (1758)
Average
Number Shots
Figure 5: The average performance on the SuperGLUE benchmarks a flunction of number of shots As opposed to the other SuperGLUE few-shot results, where we select the number of shots for each subtask according to the performance on its training split, here all subtasks are evaluated with the same number of shots. In this way, we can compare DeBERTa 1.5B directly to Figure 3.8 from Brown et (2020), which gives the same evaluation for GPT 175B (unfortunately not for smaller, more comparable, models).
Task-specific scaling analysis Figure 6 breaks down the scaling behavior for each SuperGL task individually, revealing the varying impacts of model size across different language understanding challenges. The plots demonstrate that while DeBERTa and GPT-3 both exhibit generally positive scaling trends, their scaling patterns differ substantially across task types. Most notably, DeBERTa shows steeper scaling curves than GPT-3 on the majority of tasks. This suggests that bidirectional attention mechanisms may provide particular advantages for scaling some language understanding capabilities.
20

---

Average 1-shot SuperGLUE performance vs. paremeter count
DeBERTa
GPT-3
70.0
65.0
62.5
57.5
55.0
52.5
50.0
0.1
0.4
0.8
1.3
2.6
6.7
13
175
parameters billions)
BoolQ
COPA
MultiRC
90
80
30
Accuracy
60
80
20
70
70
10
40
Parameters
Parameters
Parameters
Parameters
ReCoRD
RTE
WiC
WSC
100
80
Accuracy
90
70
60
60
70
50
40
50
0.1
0.8 2.6
175
0.8 13
175
0.1
0.8
175
0.6 13
175
Parameters
Parameters
Parameters
Parameters
Figure 6: Detailed evaluation of natural language understanding As opposed to the overall results presented Figure these plots show the scaling behavior on each subtask of the SuperGLUE benchmark. We can see the DeBERTa out-scales GPT-3 consistently across all tasks, with the exception of WiC where both models fail show any learning ability.
E
Evaluation details
This appendix provides more details about our evaluation setup make easier to reproduce our results; the source code is available at https: / http:/gggttu.cco/ttggggoot/ www.ltgoslo///eeecooooooooo context. In general, we follow Brown al. (2020) in all decisions about the implementation.
Newline separator Note we use newlines the displayed prompts for better readability, but do not actually them the DeBERTa tokenizers cannot encode newline character (they convert to the standard whitespace character instead). Instead, we convert all newline characters to double-escaped An string with whitespace character, which then acts paragraph/information separator.
Few-shot prompting For each evaluated sample, the example demonstrations are randomly selected (without replacement) from the training set of each task; if the training set available, we sample from the only available dataset split, making not select the same sample the evaluated
21

---

one. We format these examples using the respective prompt templates and concatenate them together. joined by two newline characters. The numbers of shots used for each task are given in Appendix
E.1 Needle a haystack
The needle is randomly generated 6-digit number (from 100 000 to 999 999). The prediction is produced via constrained generation that only allows sequences of tokens that form 6-digit numbers. We consider prediction to be correct only if it exactly matches the needle, which means that trivial baseline has the accuracy of 1/900000.
The evaluated models are prompted with the template shown below. Similarly to Hsieh et al. (2024), we use Paul Graham's essays fill the context ($prefix lines and $suffix lines)." The essays are sentence-segmentel and concatenated to fill the desired total sequence length.
special magic number hidden within the following articles. Make memorize it. will quiz you about the magic number afterwards.
Eeprefix_liness
The magic number is ($needle).
($suffix lines}
Question: What is the special magic number mentioned in the provided text?
Answer: The special magic number mentioned in the provided text is
E.2 Language understanding
This section provides the prompts used for the eight SuperGLUE tasks, all prompt templates are taken from the GPT-3 evaluation setup.
BoolQ This task is evaluated by ranking texts formatted as shown below with two possible $answers, yes
($passage)
question: ($question)?
($answer: -yes/no)
CB Evaluated by ranking texts formatted as shown below with three possible $answers, true, false neither.
(Spremise)
question: (Shypothesis); true, false, neither?
answer: ($answer: truee//asssaaseeeetteerrr
CОPA We rank two possible substitutions of $answer that follow the premise. $connector is formatted based on the question type: "because' if the type is 'cause', otherwise 'therefore' is used.
(Spremise)
($answer)
Available online http:p:/www..euugraaaaa.oooorooe
22

---

MultiRC
The potential answer is substituted for Soption and then we rank two possible substitu- tions for $answer: [True] or [False].
READING COMPREHENSION ANSWER KEY
(Sparagraph)
($question)
(Tanseer-(Truee/((aaas))
(Soption)
Here rank the possible entity names as substitutions for $answer. Note that $para- graph often includes summaries that, in way, act as few-shot examples (see the formatted zero-shot example in Appendix C).
ReCoRD
(Spassage)
RTE Ranking of two possible completions in this binary classification task: True or False.
($premise)
question: ($hypothesis) True or False?
answer: ($answer: - True/False)
WIC Another binary task with two possible substitutions: or no. Note that this prompt template was working for any GPT-3 models and is also working for the models evaluated in this paper, all models are just randomly guessing the answers (Table 1).
($sentencel)
($sentence2)
question: Is the word "[wword)' used in the same way in the two sentences above?
answer: (Sanswer:-yes/ng
WSC
We rank possible substitutions for $answer.
Final Exam with Answer Key
Instructions: Please carefully read the following passages. For each passage, you must identify which noun the pronoun marked in *bold* refers to.
Passage: (Stext)
Question: In the passage above, what does the pronoun ""(ssan_text)"" refer to?
Answer: ($answer)
23

---

E.3
Language modeling
This group of tasks uses very straightforward prompt templates as of these tasks are different variants of text completion.
HellaSwag Here, the task is to select the most likely completion ($answer) that follows after $context.
(Sactivity_label) (Scontext
StoryCloze
The goal is to select the most suitable $answer that completes a story laid out by four previous sentences.
($sentence_] 1} {$sentence_2 2} {$sentence_]) _3 {$sentence_) 4} ($answer)
Winograd $answer should be substituted by the correct entity (coreference resolution).
(Scontext_prefix) ($answer) (Scontext_suffee _ufrsitt__ffii
Winogrande
Same as Winograd:
(Scontext_prefixx
($answer) Ecconext_suffixe
E.4
Translation
All language pairs and all evaluation setups (zero-shot, one-shot and few-shot) use the same prompt template given below. This is the only time when decided differ from the GPT-3 setup as its very simple (and non-informative) prompt was not working for one-shot evaluation of DeBERTa models. The models then asked complete the prompt using beam search decoding with beams and the default HuggingFace hyperparameters* and the generation is stopped after producing the special newline character \nn.
($source_ Shoourcellanguage)) ($source_text)
{$target_ Laagut_ee__
For reference, here are the two prompt templates used for GPT-3 (for cero-shot and for one/few-shot, respectively):
0: what is the Langgugg__ translation of [isource_tex)) A:
($source_text)
'nttps: http://eee.............o.aa/aa.. framero wtt.://mww....eeettaiiiii...ooom generate

---

E,5
Closed-book question answering
This group of tasks mixes two types in-context evaluation: text generation (Natural Questions, TriviaQA and Web Questions) and text ranking (PIQA, ARC and OpenBookQA). The prompt setup exactly follows GPT-3.
Natural Questions Here, the goal is to generate an answer based on $question.
0: ($question)
A:
TriviaQA
Same as Natural Questions:
Q: ($question)
A:
Web Questions
Same as Natural Questions:
Q: ($question)
A:
PIQA
Here, the goal is to select the most suitable text completion by substituting for $answer.
($goal}
($answer)
ARC (challenge) and ARC (easy) This multiple choice test, the correct $answer has to be selected.
Question: (Squestion)
Answer: ($answer)
OpenBookQA
Similarly, the goal is to select the correct $answer.
($question)
($answer)
25

---

F All results
For reference, we list all results of the DeBERTa models evaluated throughout this paper Table The GPT-3 results that were used for comparison are published in Brown et al. (2020, Table H.1).
Table 7:
Results of all evaluations performed in this paper The second and third column shots the dataset splits and evaluation metrics, both of them replicating the GPT-3 evaluation setup. Note the BLEU scores used for evaluating translation are SacreBL.EU scores with signa- ture BLEU+case. www.eeeiiieemeemeeeeee www.mmst.....tttttoooo wttp:pttok.ntteeeeeeee. intl+versi+vnn 1.2.28.
Task
Split
Metric
n shots
O-shot (1.5B)
1-shot (0.1B)
1-shot (0.4B)
1-shot (0.9B)
1-shot (1.5B)
r-shot (1.5B)
BoolQ
acc.
4
57.5
68.6
76.1
75.0
CB
dev
4
46.1
23.8
39.8
47.1
57.0
57.6
COPA
dev
MANC
64
78.9
67.0
78.0
80.6
84.2
90.4
MukaRC
dev
EM DEC.
4
5.6
24
7.0
11.1
15.6
16.9
MultRC
61.6
574
57.0
67.9
69.2
ReCoRD
EM
87.1
62.3
73.6
86.8
87.4
87.4
ReCoRD
dev
87.9
63.0
74.3
87.5
88.1
88.2
RTE
dev
D0C.
8
64.3
50
53.1
64.5
65.0
62.2
WSC
dev
30C
50.5
49.5
49.6
49.6
50.3
50.2
acc.
67.3
69.6
75.0
Average
HellaSwag
dev aoc.
16
62.0
36.9
51.3
58.7
62.4
62.5
StoryClize
Test auc.
32
83.6
69.5
77.0
82/4
81.6
81.8
Winograd
test
acc.
74.0
59.3
68.1
76.2
80.7
85.6
Winogrande
aCc.
63.6
Average
72.8
75.4
DE-EN
BLEU
16
2.4
0.2
4.6
20.0
23.7
25.1
EN-DE
Test
BLEU
16
1.6
0.2
0.4
3.4
5.4
6.6
FR EN
test
BLEU
1.7
0.2
8.7
21.9
23.5
24.5
9.7
10.8
RO-EN
BLEU
1.7
16.0
17.7
18.9
EN-RO
BLEU
16
0.1
02
1.2
2.5
4.1
Average
1.3
0.1
3.1
11.2
13.8
15.0
Natural Questions
test
EM apc.
0.8
0.6
2.1
2.6
4.4
TriviQAA (wiki)
13,6
17.9
Web Questions
EM
1.5
1.0
4.5
5.1
9.9
PIQA
dev
a0c.
32
72.9
62.4
69.6
71.6
73.0
74.5
ARC (challenge)
Test
32
36.5
25.3
33.2
35.9
37.1
39.6
ARC (sasy)
Rest
300
55.1
39.6
46.3
53.3
55.1
57.7
OpenBookQA
acc.
42,8
46.4
50.4
Average
31.4
23.2
28.0
32.0
333
36.3
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
All claims are directly supported by the results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
There are explicit sections about limitations in 2. Method.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
27

---

Justification:
This paper is purely empirical.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
This paper only proposes simple inference methods, and the evaluation setup is fully described. see-ale-inteed--ee publishh addpted model-eode, We published the code at https: /tts::///tttbbb///// http://oo///ooo///eerreeeee -ttttttttt........ttttt.... context.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
28

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The paper should provide enough information fully reproduce our results, but we--wee--eeeeeee tosteend-t--uullle the source We published the code at https://github. com/ com/Ltgosloo/bet context.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (https://nips.cm for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wttpcccppplccccccc hpppullic/guids//Co/////o Cooe/Sobeeeeeeeeeeee wwtpbbimssssoooooiiiiiooo liy)) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
As described above.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: This paper replicates previous work, including its evaluation setup.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
29

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
This paper does not involve any training, only inference with negligable cost. However, we still give the compute cost of the pretrained language models (even though they were not pretrained as part of this paper).
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics /ttturi///errips http:///errriiipccuuu//////teiie
lines?
Answer: [Yes]
Justification: Yes, we only evaluate language models on published benchmarks.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
No apparent societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
30

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Section References
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assett are released, the license, copyright information, and terms of use package should be provided. For popular datasets, paper swirswitee thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
31

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Not applicable
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer:[NA]
Justification: Not applicable
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not applicable
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "kWAyhb5ks3", "reviewer": "Reviewer_iPtc", "summary": "This work proposes a simple modification on masked LMs such that they can be used as a generative way and conduct experiments as generative models. The claim of this work is that, through this modification, masked LMs can be used as in-context learners as GPT-3, adapting to new tasks with further fine-tuning. The experiments demonstrate the performance improvement with DeBERTa and a few categories of NLP tasks.", "questions": "Please refer to the previous section.", "limitations": "The paper has listed the limitations of the proposed methods, for example, how to improve the proposed algorithms. However, given the lack of technical depth, some of the listed limitations should be addressed in this work, instead of future work.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Overall, this work demonstrate the possibility of using masked LMs in a generative way. Although the modification is simple, the experiment results on some benchmark evaluation tasks are impressive. \n\nIn addition, the method description with limitations show the full picture of the proposed methods, which gives readers a comprehensive understanding about this work.", "weaknesses": "Despite of the impressive results, the work and the experiments are still in the preliminary stage, where more work can be done or needed to be done. For example\n\n- The proposed modification may be too simple, and more technical modification is needed to improve the efficiency. For example, in section 2.1, the opportunity of providing a more technically solid work is simplified as “We believe that this limitation can be fixed, …” Intuitively, I agree, although I do expect more technical novelty from this part.\n- Similar concern also applies to section 2.2, where it says “We improve on this behavior by interpolating between …”. Again, more technical discussion and novelty on this part will be deeply appreciated.\n\nWithout appropriate technical depth, I am wondering what the technical novelty of this work could be. \n\nIn addition, the experiment design is less convincing. The title and most of the discussion are about masked LMs, while the experiments are only about DeBERTa. \n\nThere are also some writing issues about this paper, for example, \n\n- Although this paper is about in-context learning, the major modification is about turning masked LMs to generative models. So, it’s not clear to me how this can be particularly tied to in-context learning\n- What the notation means in equations 1 and 2?\n- I think there may be some typos in equation 2\n\nAbout experiments\n\n- It would be great to list the inference costs of DeBERTa and GPT-3\n- What are the values of $k$ for few-shot learning?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time and effort spent on this review. While we may not completely agree with all your points (see below), they will be very helpful in making the paper more clear in the next version.\n____\n> The proposed modification may be too simple, and more technical modification is needed to improve the efficiency. For example, in section 2.1, the opportunity of providing a more technically solid work is simplified as “We believe that this limitation can be fixed, …” Intuitively, I agree, although I do expect more technical novelty from this part.\n\nWe believe that this is out of scope of this paper as its main focus is to demonstrate in-context learning of MLMs. Doing a good job at optimizing the efficiency of the generative method would most likely involve some changes to the MLM training objective, which is definitely an interesting topic, but it should be addressed in a separate paper, it is not appropriate for a footnote in this paper.\n____\n> Similar concern also applies to section 2.2, where it says “We improve on this behavior by interpolating between …”. Again, more technical discussion and novelty on this part will be deeply appreciated.\n\nWe will expand on this in an extra section in the Appendix that will describe the ablation experiments for our proposed method. In a nutshell, since the problem of the original PLL is in estimating likelihoods of long multi-loken expressions, we ablate this \"interpolation\" on the ReCoRD dataset, which involves answers with long named entities.\n\n|Ranking method|ReCoRD (EM)|ReCoRD (F₁)|\n|--------------|-----------|-----------|\n|PLL; 1 mask (original PLL)|80.9|81.6|\n|PLL; 2 masks|86.0|86.8|\n|PLL; 3 masks (our method)|87.1|87.9|\n|PLL; 4 masks|86.9|87.8|\n|Exact log-likelihood (used in causal LM) |77.2|77.8|\n____\n> In addition, the experiment design is less convincing. The title and most of the discussion are about masked LMs, while the experiments are only about DeBERTa.\n\nOur main claim is that MLMs can function as in-context learners, for this, it is enough to show that one such model exhibits these abilities. There are technical reasons for choosing DeBERTa over other models, we describe them in Related Work as well as in Section 2. In short, we need a model that is capable of processing long inputs (> 512 tokens), that is large (>1B parameters) and that is primarily trained on English. As far as we know, DeBERTa is the only model that satisfies this.\n____\n> Although this paper is about in-context learning, the major modification is about turning masked LMs to generative models. So, it’s not clear to me how this can be particularly tied to in-context learning.\n\nAll 1-shot and few-shot experiments clearly demonstrate in-context learning. Being able to generate text is necessary to show in-context learning in tasks such as machine translation. We don't think that we fully understand your point, could you please elaborate on this comment if we didn't address it?\n____\n> What the notation means in equations 1 and 2? I think there may be some typos in equation 2.\n\nThere is indeed a small typo, the last $w_{i−1}$ should be $w_{i+1}$, thanks! In terms of variables and notation used, $w_0 \\oplus w_1 \\dots w_k$ is a completion of a prompt $c$, then Equation 1 is just a simple chain rule while Equation 2 describes approximation of the previous equation with MLM; we will introduce this notation better in the future version.\n____\n> It would be great to list the inference costs of DeBERTa and GPT-3.\n\nWe didn't include it because we were not sure if it would be beneficial for the reader or not, we rely on the HF implementation of DeBERTa, which is (in our opinion) far from optimized; instead we include more general limitations. However, we also see the benefit of providing concrete numbers. When measuring the cost of generating 256 tokens from a 256-long prompt; OPT (~GPT-3) with cache: 3.8s/it, OPT without cache: 12.4s/it, DeBERTa (without cache): 20.2s/it.\n____\n> What are the values of $k$ for few-shot learning?\n\nWe completely agree that this should be described in the paper, we will put this information together with all results into a separate table in Appendix, in a similar fashion to the original GPT-3 paper. For completeness, this is the table with all results and few-shot values:\n\n| Task | Split | Metric | n shots | 0-shot (1.4B) | 1-shot (0.1B) | 1-shot (0.4B) | 1-shot (0.9B) | 1-shot (1.4B) | n-shot (1.4B) |\n|------|-------|--------|---------|---------------|---------------|---------------|---------------|---------------|---------------|\n|BoolQ|dev|acc.|4|80.8|55.7|60.5|78.4|82.1|82.1|\n|CB|dev|acc.|4|66.1|39.6|57.5|68.6|76.1|75.0|\n|CB|dev|F₁|4|46.1|23.8|39.8|47.1|57.0|57.6|\n|COPA|dev|acc.|64|78.9|67.0|78.0|80.6|84.2|90.4|\n|MultiRC|dev|EM acc.|4|6.6|2.4|7.0|11.1|15.6|16.9|\n|MultiRC|dev|F₁ₐ|4|61.6|57.2|57.4|57.0|67.9|69.2|\n|ReCoRD|dev|EM acc.|4|87.1|62.3|73.6|86.8|87.4|87.4|\n|ReCoRD|dev|F₁|4|87.9|63.0|74.3|87.5|88.1|88.2|\n|RTE|dev|acc.|8|64.3|50|53.1|64.5|65.0|62.2|\n|WiC|dev|acc.|16|55.2|49.6|49.6|49.6|50.3|50.2|\n|WSC|dev|acc.|16|71.2|62.3|65.0|67.3|69.6|75.0|\n|HellaSwag|dev|acc.|16|62.0|36.9|51.3|58.7|62.4|62.5|\n|StoryCloze|test|acc.|32|83.6|69.5|77.0|82.4|84.6|84.8|\n|Winograd|test|acc.|32|74.0|59.3|68,1|76.2|80.7|85.6|\n|Winogrande|dev|acc.|32|61.0|49.8|54.8|60.6|63.6|68.8|\n|DE--EN|test|BLEU|16|2.4|0.2|4.6|20.0|23.7|25.1|\n|EN--DE|test|BLEU|16|1.6|0.2|0.4|3.4|5.4|6.6|\n|FR--EN|test|BLEU|16|1.7|0.2|8.7|21.9|23.5|24.5|\n|EN--FR|test|BLEU|16|0.3|0.0|0.3|4.6|9.7|10.8|\n|RO--EN|test|BLEU|16|1.7|0.2|4.3|16.0|17.7|18.9|\n|EN--RO|test|BLEU|16|0.1|0.0|0.2|1.2|2.5|4.1|\n|Natural Questions|test|EM acc.|16|0.8|0.1|0.6|2.1|2.6|4.4|\n|TriviaQA (wiki)|dev|EM acc.|16|6.9|0.9|3.8|13.6|14.3|17.9|\n|Web Questions|test|EM acc.|32|1.5|0.3|1.0|4.5|5.1|9.9|\n|PIQA|dev|acc.|32|72.9|62.4|69.6|71.6|73.0|74.5|\n|ARC (challenge)|test|acc.|32|36.5|25.3|33.2|35.9|37.1|39.6|\n|ARC (easy)|test|acc.|32|55.1|39.6|46.3|53.3|55.1|57.7|\n|OpenBookQA|test|acc.|96|45.8|35.0|41.8|42.8|46.4|50.4|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4bXiHnr1ol", "reviewer": "Reviewer_ayxu", "summary": "This paper investigates whether BERT style masked language models can perform in-context learning in multiple LM benchmarks used in the GPT-3 paper, and specifically compares the results of DeBERTa to the GPT-3 model. The paper shows that in multi-choice Q&A, winogrand style quizzes the DeBERTA model can match (sometimes outperforms) GPT-3 model, whereas DeBERTa down performs in machine translation benchmarks. To be able to evaluate these models on generative tasks, the paper also introduces a simple way to use BERT like models in auto-regressive generation, which is itself a significant challenge in the field.", "questions": "- In machine translation benchmarks, were there any examples above L>512, and what the reported results did for those examples for GPT-3?\n- In Figure 3, you write “[w]e use an open-source replication of GPT-3, OPT, which should perform similarly on this task”. How did you know this? I would delete this claim unless you discuss the evidence. I think it would be unnecessary to discuss it here. So, I suggest removing that claim.\n- Figure 4 is a little odd. Why don’t you have an actual scaling curve with the actual number of shots but instead have 0, 1, and few labels?", "limitations": "As noted in the paper, The sampling procedure is too slow to use in practice since for each token the model needs to process the entire sequence again due to bi-directional attention.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents extensive results for MLMs for in-context learning which were missing and highly required in the field. \n- The paper proposes simple (slightly hacky) procedures for (i) auto-regressive sampling from MLMs, (2) calculating logprobs of a sequence of tokens with MLMs.", "weaknesses": "- The comparison of sampling procedures missing. I would like to see what scores you would get if you calculate logprobs by just summing individual pseudo-likelihoods of the tokens without doing +2 additional mask tokens. Similarly, what happens if you use Wang and Cho, 2019 method for text generation?\n- The few-shot scaling curves are not complete: only provided 0-shot, 1-shot, and the best few-shot result.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n____\n> The comparison of sampling procedures missing. I would like to see what scores you would get if you calculate logprobs by just summing individual pseudo-likelihoods of the tokens without doing +2 additional mask tokens. Similarly, what happens if you use Wang and Cho, 2019 method for text generation?\n\nWe will include an ablation experiment for the proposed ranking method that also addresses comparison. The main problem of Wang and Cho (2019) was consistency in our preliminary experiments, we will compute more quantitative results in the next few days.\n\n|Ranking method|ReCoRD (EM)|ReCoRD (F₁)|\n|--------------|-----------|-----------|\n|PLL; 1 mask (original PLL)|80.9|81.6|\n|PLL; 2 masks|86.0|86.8|\n|PLL; 3 masks (our method)|87.1|87.9|\n|PLL; 4 masks|86.9|87.8|\n|Exact log-likelihood (used in causal LM) |77.2|77.8|\n____\n> The few-shot scaling curves are not complete: only provided 0-shot, 1-shot, and the best few-shot result.\n\nWe were limited by the official GPT-3 results, which were done in the exact same way (https://arxiv.org/abs/2005.14165, Table H.1). But this is a good point, we will include additional data points for the few-shot experiments in a separate Appendix.\n____\n> In machine translation benchmarks, were there any examples above L>512, and what the reported results did for those examples for GPT-3?\n\nAll machine-translation experiments were done with 16 shots, if we look at German-English translation, for example, the average length of the whole prompt is 1113 with std of 147 (this also demonstrates the need of length generalization beyond 512 tokens). It is impossible to comment on the GPT-3 results, all we know are the final scores and that they used 64 shots to get them.\n____\n> In Figure 3, you write “[w]e use an open-source replication of GPT-3, OPT, which should perform similarly on this task”. How did you know this? I would delete this claim unless you discuss the evidence. I think it would be unnecessary to discuss it here. So, I suggest removing that claim.\n\nBecause OPT uses the same transformer architecture as GPT-3, in particular, it uses absolute positional encoding. This strictly limits any model from generalizing to longer inputs than trained on -- thus we can confidently say that GPT-3 is not able to process longer inputs than 2048, exactly like OPT. This is different from modern LMs, which usually use rotary positional encodings, or from DeBERTa with relative positional encoding. We definitely agree that this should be explained more clearly in the paper.\n____\n> Figure 4 is a little odd. Why don’t you have an actual scaling curve with the actual number of shots but instead have 0, 1, and few labels?\n\nAs commented above, this is because we decided to follow the methodology from the GPT-3 paper, which uses different few-shot settings for different tasks. This makes sense in practice, because the average length of samples differs a lot between datasets (entire documents in reading comprehension vs. sentences in machine translation). We believe it makes sense to keep this Figure as it is, to not lose the ability to compare against GPT-3, but we will add an additional Appendix section with your suggested approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dajRcIQ5Ay", "reviewer": "Reviewer_7Ges", "summary": "This paper argues that masked language models (MLMs) are just as capable as autoregressive language models at in-context learning. To demonstrate this, the authors propose a generative inference technique that allows DeBERTa to generate text, as well as a hybrid autoregressive/MLM pseudo-log-likelihood estimation technique that allows DeBERTa to rank text sequences by likelihood.\n\nDeBERTa and GPT-3 models are compared on a series of NLP benchmarking tasks in an in-context learning setup. While GPT-3 excels at translation and QA, DeBERTa excels at text completion and SuperGLUE tasks.", "questions": "Questions:\n* When computing PLL for sequence ranking, would it be possible to simply remove all of the right context and give three MASK tokens followed by SEP at each probability estimation?\n* Could you describe some specific ways in which one could combine autoregressive and masked LMs to truly get the best of both worlds? My (admittedly naïve) ideas about how one would do this all involve some mixture of objectives during pre-training. This seems like it would implicitly end up just giving you the advantages of MLMs, since a lot of their behavior comes from the fact that they receive more information (from the right context) than autoregressive systems.\n\nSuggestions/Typos:\n* L15-16: inter alig -> inter alia\n* When discussing MLM scoring (Salazar et al., 2020), could you also cite the improved version introduced by Kauf & Ivanovna (2023) here: https://aclanthology.org/2023.acl-short.80/", "limitations": "The limitations of directly comparing models that perform inference with right context versus without could be more directly discussed. Otherwise, I think the authors have done a good job of discussing limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper argues against the recent exclusive focus on scaling autoregressive language models, showing instead that masked language models also have significant potential as generative in-context learners. Relatedly, it is demonstrated that these two architectures are better for very different types of tasks. This could inform future efforts in training large-scale systems.\n2. The limitations are acknowledged explicitly for each of the proposed techniques, and are all written in a constructive way that suggest clear future directions.\n3. The writing and visuals are clear and enjoyable to read.", "weaknesses": "1. The text ranking procedure seems unfair to autoregressive models: the MLMs are given access to the right context, except for the two tokens that directly follow the one being predicted. This would give them unfair advantages on certain tasks that require more long-term dependency resolution, or which would benefit from lookahead (e.g., syntactic evaluation tasks); thus, the comparison seems a bit unprincipled in these tasks. Would it be possible to devise a technique that does not give the model any of the right context (e.g., shortening the end of the sequence to just [MASK] [MASK] [MASK] [SEP] at each prediction step, instead of giving the rest of the tokens before [SEP])? This would remove any semantic hints.\n2. The proposed inference techniques are model-specific. Thus, future efforts using MLMs will need to take into account the quirks of the MLM objective that a particular model uses when writing pseudo-log-likelihood estimation functions. The authors explicitly acknowledge this limitation.\n3. Using OPT as a comparison to DeBERTa in the length generalization experiments seems unfair: this model is informally known to perform poorly on many tasks compared to more recent architecturally similar models like Pythia or Llama. Thus, presenting these results side-by-side could mislead readers into believing that MLMs are unilaterally capable of length generalization while autoregressive models are not. I know the stated point of this paper is to compare models released around the same time, but actually, the paper seems to be about comparing different types of LM objectives at performing generative in-context learning. Thus, I think using some of the models from the cited RULER paper would be an acceptable and more fair comparison---or even just a set of Llama or Alpaca models.\n4. It is repeatedly stated that hybrid architectures making use of both autoregressive and masked LM elements is a promising future direction. However, no real evidence is given for this. It is true that MLMs and autoregressive models excel at different things, but why would this mean that combining them would give us the best of both worlds, as opposed to the worst (or simply some mean interpolation between their performances, rather than an overall gain over both)? A few papers are cited as examples, but these seem less like hybrid models and more like entirely distinct ideas (e.g., T5 is cited, but encoder-decoder models like these have different pros and cons even relative to encoder-only or decoder-only models).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review!\n____\n> The text ranking procedure seems unfair to autoregressive models: the MLMs are given access to the right context, except for the two tokens that directly follow the one being predicted. This would give them unfair advantages on certain tasks that require more long-term dependency resolution, or which would benefit from lookahead (e.g., syntactic evaluation tasks); thus, the comparison seems a bit unprincipled in these tasks. Would it be possible to devise a technique that does not give the model any of the right context (e.g., shortening the end of the sequence to just [MASK] [MASK] [MASK] [SEP] at each prediction step, instead of giving the rest of the tokens before [SEP])? This would remove any semantic hints.\n\nWe completely agree with this point, except for your statement that it is *unfair* and a weakness :) Both models are given the full context, but CLMs are limited to process it left-to-right, which indeed seems (and most likely is) suboptimal. MLMs do not mask-out half of the attention matrix and that can definitely be beneficial for some tasks. See below for your suggested ranking method.\n____\n> Using OPT as a comparison to DeBERTa in the length generalization experiments seems unfair: this model is informally known to perform poorly on many tasks compared to more recent architecturally similar models like Pythia or Llama. Thus, presenting these results side-by-side could mislead readers into believing that MLMs are unilaterally capable of length generalization while autoregressive models are not.\n\nThank you for this point, this is not the how we expected these experiments to be interpreted, we will need to frame it more clearly. From our point of view, we decided to include the length generalization experiment because this ability of DeBERTa was absolutely essential to demonstrate in-context learning (in the end, this was the reason why we chose DeBERTa instead of other MLMs). Since the overall theme is comparison to GPT-3, it felt natural to also compare the two models here. But indeed, what is really compared is absolute positional encoding (in GPT-3) and relative positional encoding (in DeBERTa), not CLM vs. MLM, we will make it more clear.\n____\n> When computing PLL for sequence ranking, would it be possible to simply remove all of the right context and give three MASK tokens followed by SEP at each probability estimation?\n\nAbsolutely, we will include this as an ablation study in the Appendix, it is the last row in the following table:\n|Ranking method|ReCoRD (EM)|ReCoRD (F₁)|\n|--------------|-----------|-----------|\n|PLL; 1 mask (original PLL)|80.9|81.6|\n|PLL; 2 masks|86.0|86.8|\n|PLL; 3 masks (our method)|87.1|87.9|\n|PLL; 4 masks|86.9|87.8|\n|Exact log-likelihood |77.2|77.8|\n____\n> Could you describe some specific ways in which one could combine autoregressive and masked LMs to truly get the best of both worlds? My (admittedly naïve) ideas about how one would do this all involve some mixture of objectives during pre-training. This seems like it would implicitly end up just giving you the advantages of MLMs, since a lot of their behavior comes from the fact that they receive more information (from the right context) than autoregressive systems.\n\nAs we hinted in the paper, the MLM training objective can be improved to be better for generation by two modifications: removing the last `[SEP]` token and shifting the outputs by one to the right. Then it should also be easy to do the mixture of objectives, as you suggest, which would make the resulting LMs more flexible during inference, if nothing else.\n____\n> When discussing MLM scoring (Salazar et al., 2020), could you also cite the improved version introduced by Kauf & Ivanovna (2023) here.\n\nThank you for this link, we were not familiar with this work, which indeed seems relevant and should be cited.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fGWQjk0r79", "reviewer": "Reviewer_gXpi", "summary": "This paper proposes a finding, indicating that masked language models like BERT can be used as in-context learners. This work suggests that there is potential for a hybrid training, where masked/causal language models can be involved, to take advantage of both objectives. The authors show that masked and generative LMs can outperform each other on different categories of tasks.\n\nAfter reading the authors' response, I think they made some good points and I think I will give accept to this paper.", "questions": "As I explained the weaknesses, authors are encouraged to answer the questions I made in the weaknesses section. For details:\n\n1. Which direction do you plan to conduct your research: \"either provide a wide and systemically empirical study or provide a solid mathematical proof\"? Please specify that and provide the relevant content.\n2. At least add Llama-2, Llama-3, Phi-2, Phi-3, Gemma-2B/7B as the exemplar generative LMs. For masked LM, if you cannot prove DeBERTa is the best masked LM (or find literature that supports your model choice), you should at least add BERT, RoBRETa, DistillBERT, T5.", "limitations": "1. Lack of experiments to support the statement/conclusion of this paper.\n2. Unclear the research conduct path.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Rethinking and attempting to discover the advantage of masked LM is very impressive, especially in the era of generative large language models, where most researchers put their efforts into improving the generative LLM performance. The figures are pretty clear.", "weaknesses": "1. In Figure 1, the authors demonstrate that DeBERTa outperforms GPT-3 in language understanding. But this is more about the natural advantage of the masked language model. The motivations for providing such experimental data are quite vague to me. Additionally, I don't see the advantages of masked models in the other three tasks (Figure 1). In logic, if one method only occasionally shows its advantage, then this method can lack robustness and generalizability. Furthermore, since the masked language models lack large versions (For example 175B), it's quite necessary to justify why use the smaller masked LM over the large and powerful generative LM.\n\n2. This paper mainly uses GPT-3 and DeBERTa to perform experiments. Several issues can weaken this paper. In the era of LLM, GPT-3 can be outdated. There are amount of generative LMs, which are redesigned and well-trained to make them much more lightweight and powerful than GPT-3. Even DeBERTa can demonstrate certain advantages over GPT-3, such experimental results cannot fully support the statement of this paper. In the same way, masked language models have many implementations. The authors need to prove that either DeBERTa is the best masked LM or provide more elements using other masked LMs.\n\n3. As a technical paper, either provide a wide and systemically empirical study or provide a solid mathematical proof.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review!\n____\n> In Figure 1, the authors demonstrate that DeBERTa outperforms GPT-3 in language understanding. But this is more about the natural advantage of the masked language model. The motivations for providing such experimental data are quite vague to me.\n\nIndeed, one contribution of our paper is that we have shown that MLMs have advantage on this kind of tasks. It is not clear to us how this is a weakness. It might seem 'natural' in hindsight, but we are not aware of any prior works that show that the MLM objective is better for learning how to understand language (without the additional influence of finetuning).\n____\n>  Additionally, I don't see the advantages of masked models in the other three tasks (Figure 1). In logic, if one method only occasionally shows its advantage, then this method can lack robustness and generalizability.\n\nMLMs are also clearly better on the second group of tasks (HellaSwag, StoryCloze, Winograd, Winogrande), see Figure 1 and 2. But we never claimed that MLMs are better overall, we clearly conclude that MLMs should be combined with CLMs during pretraining, to combine their advantages.\n____\n> Furthermore, since the masked language models lack large versions (For example 175B), it's quite necessary to justify why use the smaller masked LM over the large and powerful generative LM.\n\nWe compare language models of comparable size, using the existing pretrained models. It is completely out of budget for us to pretrain a 175B MLM for these experiments. However, we included scaling experiments to demonstrate that MLMs scale just as well as CLMs (Figure 1).\n____\n> This paper mainly uses GPT-3 and DeBERTa to perform experiments. Several issues can weaken this paper. In the era of LLM, GPT-3 can be outdated. There are amount of generative LMs, which are redesigned and well-trained to make them much more lightweight and powerful than GPT-3.\n\nWe chose two models that are comparable, in terms of the release date, size of training data, number of pretraining steps and size (as described in Section 3). As an example, you suggest to include Llama3 in the comparison, but this model is trained on 1000x bigger dataset (approximately), for much more steps and the smallest Llama3 is more than 5x larger than the largest DeBERTa; we are not sure how this additional comparison would help to prove/disprove our claim that MLMs are capable of in-context learning. Note that we are not claiming that the 4-year-old DeBERTa model is the state of the art (even though it is somewhat intriguing that DeBERTa matches Llama3 on tasks such as OpenBookQA or Winogrande).\n____\n> The authors need to prove that either DeBERTa is the best masked LM or provide more elements using other masked LMs.\n\nOur main claim is that MLMs can function as in-context learners, for this, it is enough to show that one such model exhibits these abilities.  We do not claim that DeBERTa is the best masked language model. There are technical reasons for choosing DeBERTa over other models, we describe them in Related Work as well as in Section 2. In short, we need a model that is capable of processing long inputs (> 512 tokens), that is large (>1B parameters) and that is primarily trained on English (to fairly compare against GPT-3). As far as we know, DeBERTa is the only model that satisfies this.\n____\n> As I explained the weaknesses, authors are encouraged to answer the questions I made in the weaknesses section. For details: ...\n\nWe hope that we addressed these questions with the comments above.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper seeks to demonstrate that MLMs can be capable in-context learnings, comparable with their autoregressive counterparts. The paper proposes a simple (if somewhat inefficient) technique for sampling autoregressively from DeBERTa. Experiments with DeBERTa and a matched autoregressive LM (GPT-3) are conducted on a suite of in-context learning evaluations. Surprisingly, the MLM (DeBERTa) performs well on in-context learning tasks and in some cases outperforms GPT-3. The majority of reviewers found the results of this paper both surprising and valuable, indicating that the demonstration of ICL ability in MLMs (showing advantages over autoregressive models on some tasks) might point to new training objectives and learning strategies for LLMs more generally. Some concerns were raised about the focus on DeBERTa (rather than other MLMs) and whether DeBERTa had unfair access to right-hand context in experiments -- these concerns were all adequately addressed in rebuttal.
Decision: Accept (poster)