Exploiting the Potential of Seq2Seq Models as Robust Few-Shot Learners
Authors: Jihyeon Lee
Dain Kim
Doohae Jung
Boseop Kim
Kyoung-Woon On
Venue: COLM (2024)
Keywords: Encoder-Decoder Model, In-context Learning, Few-shot Learning
TL;DR: Seq2seq models, with objective-aligned prompting and fusion-based methods, exhibit promising few-shot learning capabilities across diverse tasks, surpassing larger decoder-only models.

Abstract: In-context learning, which offers substantial advantages over fine-tuning, is predominantly observed in decoder-only models, while encoder-decoder (i.e., seq2seq) models excel in methods that rely on weight updates. Recently, a few studies have demonstrated the feasibility of few-shot learning with seq2seq models; however, this has been limited to tasks that align well with the seq2seq architecture, such as summarization and translation. Inspired by these initial studies, we provide a first-ever extensive experiment comparing the in-context few-shot learning capabilities of decoder-only and encoder-decoder models on a broad range of tasks. Furthermore, we propose two methods to more effectively elicit in-context learning ability in seq2seq models: objective-aligned prompting and a fusion-based approach. Remarkably, our approach outperforms a decoder-only model that is six times larger and exhibits significant performance improvements compared to conventional seq2seq models across a variety of settings. We posit that, with the right configuration and prompt design, seq2seq models can be highly effective few-shot learners for a wide spectrum of applications.

Published as a conference paper at COLM 2024
Exploiting the Potential of Seq2Seq Models as Robust Few- Shot Learners
Jihyeon Lee" '22 Dain Kim"t, Doohae Jung"' Boseop Kim', Kyoung-Woon On'
Kakao Corp., NVIDIA, Kyunghee University
j jihyeonlenvidia com, dannykm@khu. ac. kr, (wavy. jung, mat. kllld, off off]ekakaocorp.ccm
Abstract
In-context learning, which offers substantial advantages over fine-tuning, is predominantly observed in decoder-only models, while encoder-decoder (i.e., seq2seq) models excel in methods that rely on weight updates. Re- cently, few studies have demonstrated the feasibility of few-shot learning with seq2seq models; however, this has been limited to tasks that align well with the seq2seq architecture, such as summarization and translation. In- spired by these initial studies, provide a first-ever extensive experiment comparing the in-context few-shot learning capabilities of decoder-only and encoder-decoder models broad range of tasks. Furthermore, we propose two methods to more effectively elicit in-context learning ability in seq2seq models: objective-aligned prompting and fusion-based approach. Remarkably, our approach outperforms a decoder-only model that is six times larger and exhibits significant performance improvements compared to conventional seq2seq models across variety of settings. We posit that, with the right configuration and prompt design, seq2seq models can be highly effective few-shot learners for wide spectrum of applications.
Introduction
Recent studies have demonstrated that large language models can possess entirely different competencies, referred to as emergent abilities (Brown et al. 2020 Chowdhery et al. 2022 Rae et al. 2021 Wei et al. 2022b). The concept emergent abilities in Large Language Models (LLMs), mitiillyy introduced by Wei et al. (2022b), posits that increasing the model size and dataset can lead to the sudden emergence of abilities such as in-context learning, complex reasoning, and common-sense reasoning. In-context learning, in particular, one of the distinct characteristics of LLMs, serves as a key metric for assessing their effectiveness.
In-context learning refers to the ability of the model to perform tasks by leveraging con- textual information provided through prompts, without the need for additional training. Specifically, in the case of in-context few-shot learning, the model can generate suitable outputs for the desired target input by using a small number of examples. This offers step beyond traditional frameworks, which typically require extensive training data and fine-tuning of the model. Instead, it presents a new paradigm where a single model can effortlessly perform new tasks without necessitating a separate training process.
The capability of in-context learning has been predominantly explored in decoder-only models, as the rapid evolution of pretrained models has focused on these unidirectional architectures. However, the lequence-to-sqqeee
(seq2seq) architecture, despite its sig- nificant advantage of encoding contexts without sacrificing bidirectionality, has not been extensively investigated regarding its potential for in-context learning capabilities (Sanh
ett al. 2022 Tay et al. 2022 Soltan et al. 2022 Patel et al. 2022 Wang et al.) 2023).
Sanh et al. 2022 trained an encoder-decoder model using multitask prompts in supervised manner, but this approach only enabled zero-shot generalization to new tasks, (ooltnn et al.
* Indicates equal contribution
+Work done at Kakao Corp., correspondence to jihyeol@@nvidi..cmm and dannykm@khu.ac.kr

---

Published as a conference paper at COLM 2024
primarily demonstrated the in-context abilities of seq2seq models for generative tasks such as summarization and translation, which are tasks inherently well-suited for seq2seq models. They also reported performance natural language understanding (NLU) benchmarks but only in a zero-shot scenario, making it difficult to confirm their in-context learning proficiency across wide range of tasks.
2022)
Encoder Input
Decoder Output
Thesis: 'varry Potte" based an thus story. Answer, Fasse <ls> Theele: Bite Airgeratd
mry Answer
Encoder Input
Decoder Output
Thesk Yiarry Polle laved a stary. Answee
Fitagenal singer.
Encoder Input (vanilla)
Decoder (Vonilla)
Thesks: 'Harry Potre" 's based an thus story. Answvor, Faise <ls> The <is>
Thests Chinaaua
Input
Decoder Output (wl sentinel token)
Thesic: hiarry Potte" le bosed ana true story. Answer Fasse <ig>
Thesss Chiuahue te Mesican deg. Answer: thee <is>>
Fltgeratd
sngee
Encoder Zaput (wr mode tog)
Decodor Output tag)
Thasks Potter
chusso True
Thasks The Chiuuahue is Marican dop Anowan Tris <aa> Thasks Ele Flzgaaakd was jazz siger Answer emasis
Figure 1: Different prompting strategies for in-context learning. (a) A target input can be placed to the encoder side, concatenated with few-shot examples, or decoder side standalone. (b) Examples of pretraining objective aligned prompt. In alignment with T5, the sentinel token is attached to the target output. In alignment with UL2, the mode tag is added as prefix to the encoder input.
Motivated by the current landscape, for the first time, we thoroughly investigate the zero- shot to few-shot performance of seq2seq models across a wide range of evaluation sets. Our findings demonstrate that seq2seq models also serve as robust few-shot learners not only for generation tasks but also for understanding tasks, even outperforming their larger decoder-only counterparts [Zhang et al. 2022).
We conduct various experiments on how structure the prompts in seq2seq models. While Raffel et al. (2020); Tay et al. (2022) popularized the usage of sentinel tokens during the pretraining stage of seq2seq models to optimize the denoising objective, the proper application of the sentinel tokens during the inference stage (i.e., in-context learning) is less well-established, and is often not even mentioned in most studies (Sanh et al. 2022 Tay et al. 2022). We find that aligning prompts with the pretraining objective yields up +20.5% performance improvement on the SuperGLUE (Wang et al. 2019) benchmark.
Furthermore, we propose two fusion-based approaches that enhance the few-shot learning capability of encoder-decoder architecture models. The fundamental idea is to indepen- dently process each of the few-shot examples through an encoder and then merge these representations for decoding. This fusion-based approach offers several advantages. Firstly, addresses the restricted maximum sequence length commonly found in seq2seq mod- els (Raffel et al. 2020 Tay et al., 2022 Lester et al. 20021,, which is shorter than decoder-only models, by leveraging encoding parallelization. Secondly, we achieve more efficient en- coding of multiple examples by avoiding unnecessary bidirectional attention calculations between them. Our approach demonstrates significant performance improvements in seq2seq settings, surpassing the OPT 66B model (Zhang et al., 2022) across various tasks. Despite the absence complete attention across the contexts, our approach consistently outperforms traditional methods. Moreover, our methodologies effectively eliminate the permutation bias commonly observed in few-shot learning scenarios.
Overall, our work reveals the few-shot capability of seq2seq models, which has been undervalued compared to their zero-shot and fine-tuned counterparts. To summarize, our key contributions are: 1) we develop an in-context evaluation toolkit for seq2seq models
2

---

Published as a conference paper at COLM 2024
Model
encoder
decoder
(1/5/10-a) 10/oo))
(1/5/10-shot)
T5
65.53/ /9.54/99.999
44.95/47.37 /47.666
T5-LM
61.72/ 58.77 098877/59.7 59.74
51.76/53.85
53/85554.8668
TO
64.35/62 67.5/67.46 /4.5//6.45//63 45.6442
$2.47/53.97 /53.36
UL2
60.05/ 60.02
Table 1: Ablation on the placement of target input. The encoder places the target input on the encoder side, whereas the decoder places it on the decoder side. Bold denotes the best score within each model and for the specific number of shots. The complete results are reported in Appendix A
Model
waiilla
w/ sentinel
w/ mode tag
(0/1/5/10shott)
(0/1/5/10-shol)
T5
51.7/546/522/52.1
52.9/65.5/59
/9.45/5995//99
T5-LM
56.0/50.45566655.9
59.5/61.7 /5.1..7558.//9. 72.2/64.4/6..5/66 /8.3/
UL2
52.4/50.144..4443
58.5 65.2//1.//
Table 2: Ablation on the usage of sentinel tokens and mode tags. Figure (bb depicts the structures of the vanilla, w/ sentinel, and w/ mode tag types. For w/ mode setting, the sentinel token is also utilized. Bold denotes the best score within each model and for the specific number of shots. Due to space constraints, scores are expressed up the first decimal place. The complete results are reported in Appendix B
and conduct extensive experiments to investigate the performance of seq2seq models in zero-shot to few-shot scenarios using fair criteria, 2) we exploit the potential of few-shot learning in encoder-decoder models by exploring prompting strategies and fusion-based approaches, and 3) we experimentally demonstrate that the seq2seq model can outperform the decoder-only model with times larger parameters across diverse datasets. In Section we discuss the impact of these unprecedented findings on the evolution of LLMs.
Exploring prompting strategies for seq2seq models in in-context learning
We begin by demonstrating the different prompting strategies suitable for in-context learning of encoder-decoder models. Since decoder-only models share a unified left-to-right attention architecture (Radford & Narasimhan, 2018), we can naturally feed the demonstrations and target input in a sequential manner and perform generation to obtain the target output. For instance, in a 5-shot English-French translation task, the input sequence is formed by concatenating five English-to-Frende demonstrations, followed by target English input, with the model expected to generate the corresponding French translation the target output.
However, encoder-decoder models process inputs and outputs independently from the en- coder and decoder, respectively, which can result in varying compositions of demonstrations, target input, and target output, as shown in Figure (a). The target input can be utilized either as an encoder input, resulting in the decoder generating only the target output, or as decoder input, enabling the decoder to directly generate the answer conditioned on the target input. We evaluate this ablation on the SuperGLUE dataset, with 1, 5, and 10-shot settings using four representative encoder-decoder models. Table Table shows that positioning the target input on the encoder side results enhanced performance for all model types, with performance gains of up to +2..5%p. We interpret these results follows: During the pretraining stage, the model generates output conditioned on bidirectionally encoded infor- mation from the encoder, instead of relying the preceding textual information from the decoder. As a result, placing the target input on the encoder side shows better performance.
Additionally, recent models include T5 (Raffel et al. 2020) and T5 variants (Lester et al.) 2021 Sanh et al.] 2022 Tay et al. 2022 Patel et al. 2022) suggest different training objectives for language modeling. Raffel et al. (2020), Lester et al. (2021); Sanh et al. (2022) employ
3

---

Published as a conference paper at COLM 2024
denoising objectives to learn better bidirectional representations and utilize sentinel tokens to replace consecutive spans of tokens. TTy et al. (2022) proposed the model UL2, which additionally introduces the concept of mode switching with extra paradigm tags (ie,, (NIG),)I,,) that help the model learn suitable representations for given task.
model
We argue that, even when performing downstream tasks, it is crucial to design the prompt to resemble the pretraining scheme to achieve optimal performance. For instance, T5 manipulates inputs by masking consecutive tokens of the original text with a single sentinel token and then generates the target sequence by conditioning on that sentinel token. Thus, unlike decoder-only models that simply list examples as inputs, it is a more natural setting to prepend a sentinel token to the target for in-context learning using 15. As depicted in Figure [b.),, we conduct experiments by incorporating sentinel token and a mode tag into the vanilla prompt format. Since the mode tag is exclusively used in UT.2, it is not applied to the remaining models. We assess the performance on the SuperGLUE datasets with various number of few-shot examples. In accordance with the pretraining methodologies of the models, we position the sentinel token at the end of the input and the model tag at the beginning. The results are presented in Table 2 Following the pretraining objective yields maximum performance gain of up to +13%p. As expected, aligning with the pretraining objective produces the most favorable results, with a significant discrepancy in scores. It is noteworthy that even T5-LM (Lester et al. (202T) and TO (Sanh et al. 2022), which have subsequent training phase without using sentinel token, demonstrate the positive impact of adding the sentinel token to the prompt. Throughout the following sections, we employ the optimal objective-alignee prompting strategies that we identified in this section, the default configuration in all experiments.
3 Fusion-based approaches for few-shot learning
In this section, we address another factor that impairs the performance of few-shot learning and propose two fusion-based approaches to mitigate this problem. 15-family models (Raffel ee al., 22020] Lester et al., 2021 Sanh et al. 2022; Tay et al. 2022) utilize relative position encoding (RPE) (Shaw et al. 2018), thereby enabling to process long input sequences as long as computing memory allows. However, shown in Press et al. (2022), the extrapolation ability of T5-style RPE diminishes when the input length exceeds twice the maximum pretrained sequence length. That is, encoder-decoder models with shorter training sequence lengths of 512 (Raffel et al. 2020 Tay et al. 2022), compared to decoder- only models (Zhang et al. 2022 Brown et al. 2020 Black et al. 2022], often exhibit limited performance when dealing with long-context tew-shot learning scenarios. Additionally, there is a challenge in terms of computational cost, which escalates quadratically as the number of shots increases. Another challenge to address is the presence of permutation bias. The order of demonstrations affects the positional embedding of tokens, resulting in different predictions even when the examples are identical but the order varies.
To address these problems, we propose a fusion-based approach, in which each demon- stration is independently processed by the encoder. Afterwards, merge them together at a specific stage. We hypothesize that the encoding of relations between demonstrations does not significantly impact in-context learning performance. We categorize this approach into two styles: early-fusion and late-fusion, differentiated by the stage at which the fusion occurs. These concepts are inspired by retrieval-based generation models. Lewis et al. (2020) propose Retrieval-Augmentte
Generation (RAG), where each retrieved document is integrated with the query and passed through the model independently, and the final output probability is calculated as a weighted sum of the probabilities of each output. -eecoer (Fid), introduced by [zacard & Grave (2021), differs from RAG in that the inputs passed through the encoder are concatenated betore being sent to the decoder without any weighting. This allows the model to perform fusion at the decoder level. We apply these ideas to few-shot learning of seq2seq models by treating each fhot example as a retrieved document. Therefore, we use the term late-fusion to refer to the RAG-style method that combines the decoder outputs, whereas the term early-fusion refers to the FiD-style method that combines the encoder outputs. The formulas are described below. The traditional seq2seq in-context learning, which we refer to as the original method,
4

---

Published as a conference paper at COLM 2024
Example Questton
Example Question
Enceder
Deceder
Dsa mpple Question
Decoder
Example Question
Enceder
Decoder
Exa mample Questten
Example Question
Encoder
Uecoder
earky-fusion
(b) late-fusion
Figure 2: An overview of the proposed approaches. Each case is presenting 3-shot setting.
constructs the encoder input by concatenating the target input with the few-shot examples
z to generate the target output y in the decoder (Raffel et al. 2020: Tay al. 2022). The probability of the target output for original method can be expressed
N
Porigin x,) ~
(E x), V1:i 1)
where fenc and fae are encoder and decoder, respectively, and denotes the number of shots. Here, y; is the i-th target token and V/:i-1 1 is the partial output. In the late-fusion approach, as depicted in Figure (b), j-th example 2j is concatenated to x and each combination is utilized separately as the model input. At each token generation step i, the information from k instances that have passed through the entire model is aggregated follows:
N k
Plate (yl x,)
ITE faee (yilsene (1ii-)
Unlike the original RAG model, in our few-shot settings, examples are obtained without going through the retrieval process. Therefore, consider all examples to be equally retrieved, and no weighting is applied to individual examples.
Instead of combining the decoder outputs, early-fusion approach merges the information from the encoder output and passes to the decoder, as shown in Figure (a). All the last hidden states of the encoder, h1, It...., where 11; fenc (2z, x), are concatenated sequentially and used for cross-attention,
N
Pearty(y) (y)x,, xe fac [w h1a,
These fusion-based approaches successfully mitigate the extrapolation problem by main- taining shorter length of the encoder input. Moreover, inference time be reduced in proportion to the number of examples k through batch processing. In the following sections, we experimentally demonstrate the capability of the seq2seq model as robust few-shot learner by integrating the prompting strategies proposed in Section 2 with the fusion approaches presented in this section.
Experimental setup
We provide detailed description of our experimental setup in the following paragraphs. To ensure fair evaluation, we employ identical prompt structures, evaluation tasks, scoring methods, prompt templates, and few-shot demonstrations across all baseline models.
4.1 Baseline models
To test the impacts of our proposed methods on seq2seq models, we employ T5 and its variants, including T5-LM, TO, and UL2, as baseline models. Among seq2seq baseline models, TO is the only model fine-tuned with plenty of zero-shot multitask prompts, and UL2 contains 20B parameters while others contain 11B. For the comparison with decoder- only models, adopt the following decoder baselines: OPT (Zhang et al.) 2022) and BLÓOM (Workshop) 2023), with parameter sizes ranging from 7B to 66B. We provide detailed information about our baseline models in Appendix

---

Published as a conference paper at COLM 2024
Model
Shot
KTE
CB
ANLI RI
ANLI R2
ANLI R3
WSC Winogrande
COPA
StoryCloze HellaSwag"
WiC
average
5 59.35
51.43
33.66
33.92
34.22
36.54
65.02
75.60
70.83
46.32
50.63
50.68
BLOOM-7B
10 57.11
48.21
33.84
33.44
33.95
36.54
64.61
73.00
71.47
46.04
50.03
50.02
gpt3
57.47
57.14
33.40
33.64
34.62 62
36.54
64.75
78.80
71.96
46.62
50.03
51.36
5 50.47
33.57
34.24
33.16
34.05
36.54
68.68
86,00
79,02
52.76
52.01
50.95
OPT-13B
10 49.17
42.14
34.04
33.52
35.00
36.54
68.29
86.40
79.70
52.34
52.79
51.83
gpt3
49.39
40.71
35.04
32.90
35.70
36.54
68.59
87.40
80.17
52.86
52.45
51.98
5
63.61
38.21
30.75
33.64
32.53
38.65
69.49
84.80
78.87
54.82
52.63
52.55
OPT-30B
10 61.66 41.07
31.53
30,60
34.15
36.35
70.89
84.60
79.58
55.26
51,22
52.45
8pL3
62.67
58.57
31.78
31.56
32.72
36.54
70.75
86.60
80.57
55.70
52.23
54.52
5
66.14 49.29
32,83
33.56
34.23
36.35
20.13
88.40
50.72
56.72
51.32
54.52
OPT-66B
10 63.68
56.79
32.50
38.88
34.45
36.54
70,31
40
81.53
56.62
51.54
53.11
gpt3
68.23
60.36
33.90
34.52
34.83
36.54
71.19
89.80
82.30
57.10
52.29
56.46
5 55.02 53.93
33.70
35.00
35.62
39.04
64.64
82 80
77.71
45.32
50,50
52.12
T5-110
10
56.10
51.43
38.66
33.88
35.80
.88
64.66
82.00
77.67
44.64
52.10
51.62
gpL3
50.32
38.57
33.74
33.26
33.98
40.38
64.77
81.20
71.50
43.10
49.94
49.16
73.93
38.18
84,20
45.90
58.11
15-11B-wnly
10
63.39
27.86 86
41.18
38.32
39.50 585
65.19
62.23
84.80
77.56
45.80
50.00
58.72
gpt3
64.04
75.00
42.08
37.94
68.27
85.40
77.57
46.06
50.16
58.98
5 62.74
70.36
39.88
38.12
39.27
61.92
62.00
83,80
77.56
46.02
50.03
57.43
15-11B-late
10
73.93
40.16
38.32
45.78
58.01
8pt3 63.90 70.36
40.64
38.06
39.63
69.42
62.15
84.80
77
45.88
50.22
58.39
Table 3: Comparison of our approach with various decoder models using minimal tem- plates proposed by Gao et al. (2021). Bold denotes the best average score, and underline denotes the best score within each task. T5 -early and -late demonstrate superiority over decoder models in tasks such as CB and WSC, while exhibiting limitations in tasks such as Winogrande and Hellaswag, Tasks denoted with a star (*) are evaluated with 1K samples due to their size. The row labeled "gpt3" refers to the optimal number of shots for each task suggested by Brown et al. (2020). Detailed configurations are reported in Table
(a) T5
(b) T5-LM
(c) UL.2
(d) TO
Figure 3: Comparison of in-context learning ability among seq2seq models by increasing the number of shots. We experiment on the same tasks as presented in Table 3 and report the average accuracy and standard deviation results for four seq2seq baseline models.
4.2 Evaluation tasks and scoring methods
In Table and 2 we evaluate the models on eight subtasks from SuperGLUE to evaluate
language understanding ability. For comprehensive evaluation across various types of tasks, we additionally assess the baselines on 11 tasks selected by Sanh et al. five subtasks from SuperGLUE, Hellaswag (Zellers al. 2019) , ANLI Nie et al. 2020),
(2022) including
Winogrande (Sakaguchi et al. 2019), and StoryCloze (Mostalazadeh et al. 2016). We report accuracy as the metric and the model selects the option with the lowest cross-entropy loss among all the available multiple-choice options. Any form of normalization with respect to the target output is not applied. Please refer to Appendix H for justification regarding normalization, and to Appendix detailed explanations of each benchmark dataset.
We adopt two following generation tasks for the comprehensive evaluation: XSum (Narayan) et al.) 2018) and WebNLG (Moryossef et al. 2019), which are summarization and data-to- text tasks, respectively. All the samples are generated with greedy decoding, and the ROUGE (Ln 2004) metric is used for the evaluation of generation tasks.

---

Published as a conference paper at COLM 2024
4.3 Prompts and few-shot demonstrations
We employ prompt templates offered by Language Model Evaluation Harness Gao et al. 2021), a widely adopted few-shot evaluation toolkit for decoder-only models. For each generation task, we adopt single template offered by PromptSource (Bach et al. 2022). We conduct five experiments with different random seeds and present the averaged scores for Tan Figure 3
For the other evaluations, we conduct single experiment. For detailed
information about prompts and few-shot demonstrations, please refer to Appendix
5 Evaluation results
5.1 Seq2seq models perform better than decoder-only models
In Table 3 we evaluate both encoder-decoder and decoder-only models on broad range of NLU benchmarks. We select T5 as the primary baseline model to apply our approaches because it is the most commonly used seq2seq pretrained model. Furthermore, T5-LM and TO are models that have undergone further training with different objectives, making unfair to compare them against other pretrained models. As shown in Table 3 T5 with both early- and late-fusion approaches achieve superior performance over OPT--66B model, which is six times larger in parameter size, with maximal margin of 3.6%p in average. Even with the best-shot configuration found by GPT-3 (Brown et al.) 2020), our methods consistently exhibit superior performance compared to the decoder-only baselines in most cases, with sigmificant margin. Note that all the T5 variants reported Table B already incorporate the objective-aligned prompt designs suggested in Section 2
Complete results for the remaining seq2seq baselines (i.e., T5-LM, TO, and UL2) can be found in Appendix Additionally, we present the results of our method in comparison to other well-known decoder-only LLMs (i.e., GPT-3, PaLM, and GPT-NeoX) in Appendix C
5.2 Robustness of our approach regardless of the baseline models
In Figure we examine the in-context learning abilities of different seq2seq baseline models and observe trend that varies with the number of shots. Our approach demonstrates its effectiveness by consistently improving the performance when applied to all models. The black lines, which represent the original method, predominantly exhibit decline score in the few-shot setting, when compared to zero-shot or one-shot inference.
In contrast, the results obtained by applying our approach show a completely different tendency. The red and green lines, representing the application of early-fusion and late-fusion, respectively, exhibit an upward curve, indicating that the few-shot learning performance outperforms the 1-shot performance. The only exception is UL2 where the few-shot perfor- mance remains equal to the 1-shot performance.
Surprisingly, despite TO being fine-tuned with zero-shot prompts, both of our approaches result in higher performance with the GPT-3 best-shot setting compared to the zero-shot result of the original TO model. Considering that all four models were trained differently, this result reinforces the general applicability of our approach to seq2seq models, irrespective of how they were previously trained.
Additionally, we also observe that early-fusion consistently outperforms late-fusion by small margin. Generally, the pretraining objective of encoder-decoder models is designed such that the decoder aggregates the outputs of the encoder. This process maximizes the joint probability of the input text sequence for the decoder, achieved through the encoder- decoder a attention module. In this context, early-fusion implicitly selects examples that assist in resolving the test query by fully leveraging the encoder-decoder attention module. On the other hand, late-fusion does not differentiate whether individual examples aid in solving the test query; it simply aggregates all responses. Thus, we posit that the capability of the early-fusion to selectively prioritize certain examples, an ability absent in the late-fusion, is major factor contributing to its superior performance. For detailed scores for each setting, please refer to Appendix D
7

---

Published as a conference paper at COLM 2024
1-shot
5-shot
1-shot
32-shot
Model
Model
(R1/R2/RL)
(R1/R2/RL)
(R1/R2/RL)
(RI/R2/RL)
T5"
13.72/22.4/111.97
7.57 /0.66/6.344
T5*
18.63/7.98/16222
0.25/0.13/0.244
T5
25.12/8.69/20.72
26.39/ /8.89/991....
T5
39.13/23.44932.61
31.40/22.63// /3/04.09
T5-early 15-late
30.31/ /0.31/111.5/ 11.55/ 11.55//2.5111 25.10 29.89/ 11//9/1139///// /1.29//7///
T5-early
49.47 29.1//9.1//// 40.84
T5-late
48.47 18.47/28.54/400
Table 4: Evaluation on XSum dataset. The asterisk(*) on the right side of the T5 denotes the case where the sentinel tokens are not used during inference time. RI, R2, and RL denotes ROUGE-1,22 respectively.
Table 5: Evaluation on WebNLG dataset. The asterisk(") on the right side of the T5 denotes the case where the sentinel tokens are not used during inference time. R1, R2, and RL denotes ROUGE-1,21 respectively.
OPT-13B
T5-original
T5-early T5-late
average
52.16
60.57
68.00
67.50
std
2.02
4.51
0.00
0.00
Table 6: Evaluation of permutation bias for each method. We experiment on an identical of 5-shot demonstrations, with only a different order. Bold represents highest score for the average and lowest value for the standard deviation.
5.3 Validation of our approach in the generation tasks
In this subsection, we verify the robustness of our method by evaluating it on several generation tasks. As mentioned in Section 1 generation tasks, such as summarization and translation, align well with the objective of the encoder-decoder architecture and have been proven be successful with seq2seq models. However, there is still lack of organized evaluation of the few-shot learning ability on those tasks. We thus further evaluate our model on XSum and WebNLG datasets, in both one-shot and few-shot learning settings. Though studies from Tay et al. (2022) and (ooltan et al. (2022) report the few-shot results of seq2seq models for XSum dataset, those results only encompass one-shot learning. Consequently, to the best of our knowledge, this is the first work that explores the results of "few"-shol learning for seq2seq models on the generation tasks.
The application of objective-aligned prompting and fusion-based approaches shows remarkable improvement in the summarization task compared to the original T5 model, as shown in Table 4 A similar trend holds in WebNLG, a data-to-tex" task, shown in Table This implies that our proposed approach not only serves as a robust few-shot learner for understanding tasks but also for generation tasks. The analysis comparing decoder-only models in generative tasks can be found in Appendix E
5.4 Analysis of the permutation bias
One challenge of in-context few-shot learning is the potential variation in model predictions due to the order in which demonstrations fed to the model. The late-fusion approach eliminates the so-called permutation bias problem, as it operates by fusing the probabilities of decoder outputs regardless of the order in which the inputs are fed. To verify the effectiveness of our methods in reducing permutation bias, we conduct experiments by reordering the few-shot examples. We select four tasks, CB, COPA, WSC, and WiC, from each of the four task taxonomies introduced by Synah et al. (2022): : natural language inference, sentence completion, coreference resolution, and word sense, to cover a diverse range of task types. We examine the permutation bias for 5-shot setting and randomly sample 50 test sets from each task using the same random seed. In Table we report the average and standard deviation for all 120 possible permutations with 5-shot examples for each task. Surprisingly, both the early-fusion and late-fusion methods demonstrate zero standard deviations for all tasks, thanks to the elimination of order relation on the encoder side. In the early-fusion method, there is a subtle variation in probability caused by the relative positional bias when the order of the examples is fused. However, this slight difference
8

---

Published as a conference paper at COLM 2024
does not have any impact on the metric score. In contrast, the decoder-only baseline and the original encoder-decoder baseline show higher standard deviations with respect to the permutation of examples.
Related work
From UniLM (Dong al., 2019 to BART (Lewis et al. 2019), 15, UL2, and AlexaTM (Soltan et al. 2022), encoder-decoder architecture models have continuously advanced and set new records by introducing novel denoising pretraining objectives. They excelled in tasks that require comprehensive understanding of the input context and generate output based on it, such as translation, summarization, and other sequence-to-sequene tasks. However, this was particularly evident during the fine-tuning process. The seq2seq models have had limitations handling sequences long enough, and was only after the T5 series of papers that they were able to address this to some extent by incorporating relative position encoding. As a result, in-context few-shot learning, which necessitates a long encoder input length, has received relatively less attention. As consequence, despite the recent explosive growth of LLMs, the size of encoder-decodee than 20 billion parameters (lay et al., 2022 Soltan et al. 2022). This stands in contrast to decoder-only models, which have reached the scale of hundreds of billions (Brown et al., 2020 Chowdhery et al. 2022 Zeng et al. 2022).
models has remained stagnant, with less
In-context learning has emerged as an alternative to the fine-tuning paradigm (Devlin et al. 2018 Radford & Narasimhan 2018), which incurs expensive training costs, especially in the case of very large language models (Brown et al. 2020). According to BBronn et al. (2020), pretrained decoder-only LLMs achieve proficiency various tasks simply by prompting few examples as input, without requiring any parameter updating. Unlike the many
advancements of large decoder-only models (Rae et al. 2021; Zhang et al., 2022 Touvron iet al. 2023 Hao et al., 2022), the mainstream approach for encoder-decoder LLMs to adapt to a specific task remained supervised instruction-tuning (Sanh et al., 2022 Wei
et al. 2022a; Longpre et al., 2023). Recently, few studies attempted to explore in-context
learnung; UL2 and Alexa'TM reported zero-shot results on the SuperGLUE dataset, and
TO utilized multitask prompted training to enhance zero-shot performance. Some of the
studies employed techniques that emulate decoder-only models. Patel et al. (2022) utilized decoder-only-style sequential autoregressive prompting. [aa al. (2022) mixed causal language modeling with denoising objectives. However, these approaches are restricted to particular setups and there has been lack of structured results regarding few-shot learning.
7 Discussion and conclusion
In this work, we demonstrate that the seq2seq model with proper adaptation enables few- shot learning across broad range of tasks. To the best of our knowledge, no previous research has fully exploited the potential of seq2seq models as robust few-shot learners, language understanding tasks. Our
outperforming decoder-only models, particularly alignment of prompts with pretraining remarkable results are attributed to two factors: 1) the objectives and 2) the use of fusion-based methods that independently process examples, compensating for the objective incompatibility and structural shortcomings of seq2seq models, respectively. Through the carefully designed experiments, we verify that our approach exhibits a significant performance advantage under diverse conditions, including varying the number of shots, different baseline models, and a range of tasks, even with the complete removal of permutation bias. Another major contribution of our work is the provision of unified in-context learning evaluation toolkit for seq2seq models. We have conducted controlled experiments to systematically compare and analyze the ability of existing seq2seq models - a process not previously undertaken. We plan to release the toolkit. Importantly, our findings rediscover the strengths of the encoder-decoder architectures and shed new light on their potential as conversational agents (e.g,, GPT- 4 (OpenAI, 2023)), which have been underestimated. We believe that the evolution of seq2seq models is still in progress, and our approach offers a substantial contribution toward maximizing their in-context learning capability.

---

Published as a conference paper at COLM 2024
References
Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Fries, Maged Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike Tian-jian Jiang, and Alexander Rush. PromptSource: An integrated development environment and repository for natural language prompts. In Proc. the Annual Meeting of the Association for Compulational Linguistics (ACL), 2022.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-20B: An open-source autoregressive language model. In Proc. the Annual Meeting of the Association for Computational Linguistics (ACL), 2022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXio, 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Isvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben- ton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanu- malayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Oliveira Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. ArXiv, 2022.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, ArXiv, 2018.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2019.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. ArXio, 2020.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Lau- rence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 2021.
Yaru Hao, Yutao Sun, Li Dong, Zhixiong Han, Yuxian Gu, and Furu Wei. Structured prompting: Scaling in-context learning to 1,000 examples. ArXiv, 2022.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Proc. of the Anmual Conference of the European Chapter of the Association for Computational Linguistics (FACCL, 2021.
10

---

Published as a conference paper at COLM 2024
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proc. of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-to- sequence pre-training for natural language generation, translation, and comprehension. ArXio, 2019.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. detrieval-augmented generation for knowledge-intensive nlp tasks. In Proc. the Advances in Neural Information Processing Systems (NeuriPS), 2020.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries, In Text Summa- rization Branches Out, 2004.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, 2019.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning. ArXiv, 2023.
Amit Moryossef, Yoav Goldberg, and Ido Dagan. Step-by-step: Separating planning from realization in neural data-to-text generation. ArXiv, 2019.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proc. of The Annual Conference of the North American Chapter of the Association for Computationall Linguistics (NAACL), 2016
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! Topic-aware convolutional neural networks for extreme summarization. In Proc. of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial NLI: A new benchmark for natural language understanding. In Proc. the Anmual Meeting of the Association for Computational Linguistics (ACL), 2020.
OpenAL. Gpt-4 technical report. ArXiv, 2023.
Ajay Patel, Bryan Li, Mohammad Sadegh Rasooli, Noah Constant, Colin Raffel, and Chris Callison-Burch. Bidirectional language models are also few-shot learners. ArXiv, 2022.
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In Proc. the International Conference on Learning Representations (ICLR), 2022.
Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training. ArXiv, 2018
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John F. J. Mellor, Irina Higgins, Antonia Creswell, Nathan McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Pa- ganini, L. Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Gan-Baptiste
11

---

Published as a conference paper at COLM 2024
Lespiau, Maria Tsimpoukelli, N. K. Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pa- jarskas, Tobias Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' A'Atume Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew G. Johnson, Blake A. Hechtman, Laura Weidinger, lason Gabriel, William S. Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem W. Ayoub, Jeff Stanway, L. L. Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher. ArXiv, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. In Journal of Machine Learning Research (JMLR), 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. ArXic, 2019.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Ur- mish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexan- der M Rush. Multitask prompted training enables zero-shot task generalization. In Proc. the International Conference on Learning Representations (ICLR), 2022.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proc. of The Anmual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2018
Saleh Soltan, Shankar Ananthakrishnan, Jack FitzGerald, Rahul Gupta, Wael Hamza, Haidar Khan, Charith Peris, Stephen Rawls, Andy Rosenbaum, Anna Rumshisky, Chan- dana Satya Prakash, Mukund Sridhar, Fabian Triefenbach, Apurv Verma, Gokhan Tur, and Prem Natarajan. Alexatm 20b: Few-shot learning using large-scale multilingual seq2seq model. ArXiv, 2022.
Yi Tay, Mostafa Dehghani, Vinh Quang Tran, Xavier García, Dara Bahri, Tal Schuster, Huaixiu Zheng, Neil Houlsby, and Donald Metzler. Unifying language learning paradigms. ArXiv, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXio, 2023.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general- purpose language understanding systems. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2019.
Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Junnan Li, and Steven C. H. Hoi. Codet5+: Open code large language models for code understanding and generation. ArXiv, 2023.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In Proc. the International Conference on Learning Representations (ICLR), 2022a.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. In Transactions Machine Learning Research (TMLR), 2022b.
12

---

Published as a conference paper at COLM 2024
BigScience Workshop. Bloom: A 176b-parameter open-access multilingual language model. ArXiv, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Proc. the Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong, and Jie Tang. Glm-130b: An open bilingual pre-trained model. ArXiv, 2022.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. ArXio, 2022.
13

---

Published as a conference paper at COLM 2024
Appendix
We complement our paper with additional experimental results and miscellaneous details throughout this material. Section A provides the complete experimental results for Table regarding the proper placement of the target input in the encoder-decoder model. Sec- tion B exhibits the impact of objective-aligned prompting for in-context learning, which complements Table 2 Moreover, we report the few-shot learning performance of other Section [ presents detailed numerical values utilized for reporting Figure 3 In Section E. we compare our methods with decoder-only models on generative tasks. Section offers detailed infor- mation regarding the baseline models used in our experiments. Section GeroonGGdeee demonstrates examples of the prompts utilized in the generation tasks. Section []povvidss an explanation of how normalization is considered when measuring few-shot scores, Lastly, Section gives explanations for each of the benchmark datasets.
decoder-only LLMs that are not featured in the main paper, in Section C
A Detailed results for Table
Table T verifies that the placement of a target input greatly impacts the in-context learning performance of seq2seq models. Table provides detailed results of whether it is advan- tageous to have a target input in the encoder or the decoder, based on each task and each number of examples. In most cases, the target input given to the encoder produces signif- icantly better results than that of the decoder. Although there are a few instances where the scores are improved when the input is given to the decoder for specific combinations of models and tasks (e.g., UL2 with RTE task, T5-LM with WSC task), these are small exceptions that make it difficult identify trends.
B Detailed results for Table 2
In T present the complete results that complement Table 2 Without using sentinel tokens, 15 shows consistent scores across five out of eight tasks, regardless of the variation in the number of shots. However, this issue is resolved when sentinel tokens used. Since TO is further trained with the zero-shot prompts, we can observe that without the use sentinel tokens, the score is initially higher at zero-shot results but significantly decreases the number of demonstrations increases. On the other hand, when sentinel tokens are employed during inference, even though the highest scores are still observed in zero-shot scenarios, the performance is almost maintained as the number of demonstrations increases. Based on these observations, we conclude that aligning the prompt structure with the pretraining objective generally helps the seq2seq model better understand few-shot examples. Note that the scores reported in Table are not the ones that applied our proposed methodologies like early- or late-fusion.
C Comparing with other decoder LLMs
While numerous LLMs have their own benchmark evaluation scores reported, it is not guaranteed that these models are evaluated using the same baseline. Despite this potential discrepancy, to provide a point of reference, we present the results of our method alongside those of other well-known decoder-only LLMs in Table 10 including GPT-3, PaLM (Chowd. hery et al. 2022) and GPT-NeoX (Black et al. 2022). All the records of decoder LLMs in Table are from the paper. Compared average score. Although there is only a small overlap in tasks, T5-early beats GPT-NeoX on average and demonstrates much better performance in ANLI tasks compared to GPT-3. This indicates that our model performs well considering its size, as the throughput of the seq2seq model is comparable to that of a half-sized decoder-only model, as stated by Tay et al. (2022). In this regard, it is unfortunate that the absence of a larger encoder-decoder model hinders us from making an equivalent comparison with decoder models larger than 100B.
P-L.M-8B, T5 with early-fusion achieves higher
14

---

Published as a conference paper at COLM 2024
Placement Model
Shot
RTE
COPA
CB
WIC
WSC
BoolQ"
MultiRC
RECORD"
average
1
60.65
84.00
82.14
50.00
39.42
86.10
43.00
78.90
65.53
15
5
52.71
84.00
67.86
50.00
38.46
61.50
42.70
79.10
59.54
10
52.71
84.00
51.79
54.08
36.54
61.50
56.60
76.50
59.09
1 64.02
83.00
52.57
50.00
36.54
74.30
46.20
85,50
61.72
T5-LM
5
47.29 29
78.00
57.14
50.00
44.23
61.50
46.90
83.10
58.77
10
47.29
79.00
53.57
50.00
60.58
61.50
43.60
82.40
59.74
encoder
1 74.73
82.00
58.93
50.16
35,58
74.30
58.20
80,90
64.35
TO
5
71.84
80.00
69.64
56.11
52.88
70.10
57.60
81.40
67.45
10
47.29
82.00
51.79
56.90
64.42
70.40
57.20
77.40
63.42
1 54.15
86.00
42.86
51.57
36.54
78.20
42.70
88.40
60.05
UL2
5
49.10
81.00
50.00
50.00
36.54
63.60
54.30
86.30
58.85
10
47.29
84.00
50.00
50.00
36.54
68.70
57.30
86.30
60.02
1 52.71
60.00
41.07
48.12
36.54
58.50
47.60
15.10
44.95
5
52.71
77.00
41.07
46.00
38.00
58.80
51.00
14.40
47.37
52.71 10
73.00
41.07
50.00
38.00
57.50
52.30
16.30
47.61
51.99
74.00
46.43
50.00
36.54
63.00
42.60
51.76
T5-LM
5 47.29
72.00
48.21
47.00
62.00
61.50
42.70
50.10
53.85
10
47.29
75.00
51.79
46.00
62.00
61.50
42.70
51.20
54.68
decoder
53.43
77.00
48.21
48.59
36.54
61.80
48.00
46.20
52.47
TO
5
48.74
74.00
53.57
53.29
40.38
62.40
52.60
46.80
53.97
10
46.93
75.00
48.21
52.82
42.31
61.30
53.90
46.40
53.36
1
62.09
61.00
12.50
50.00
41.35
63.40
43.30
78.00
51.46
UL2
5
58.12
76.00
50.00
50.00
36.54
61.50
44.60
82,50
57.41
10
53.07
80.00
64.29
50.00
36.54
61.40
45.90
81.90
59.14
Table 7: Detailed results for Table 1. The asterisk(*) on the right side of the task indicates that due to the large size of the test dataset, evaluation is performed on a random sample of 1K instances from the test dataset. Bold denotes the best average score among different placements of the target input for each model and number of shots.
D Additional results for Table and Detailed results for Figure
In order to thoroughly analyze the performance of seq2seq baselines, we report the additive results evaluated on 15, T5-LM, TO, and UL2 for Table i Table Although compare various decoder-only models with T5 (with our methodologies applied), other encoder- decoder models such T5-LM and UT.2 also achieve significant scores that approximate those of the OPT-3UB model, which is much larger.
Table 11 is also the supplementary table for Figure which depicts the average and standard deviation results for four seq2seq baselines for each methodology. The "std" values we report do not represent the standard deviation of scores across different tasks, but are calculated by averaging the standard deviation values across all tasks, with five seeds evaluated on each task. By applying our approaches, the average scores consistently increase for all models, while simultaneously reducing the standard deviation.
E
Comparison with decoder-only models on generative tasks
The pretraining objectives of existing seq2seq models aren't ideally configured for free response generation; their focus was more on improving downstream task capabilities through span corruption and local reconstruction objectives, rather than on language mod- eling objectives. Consequently, for generative tasks, they might be less effective compared to decoder-only models. To illustrate this, we conducted comparisons with decoder-only baselines on representative generation tasks, XSum and WebNLG, with the results detailed in Table 12 and Table 13 respectively. We highlight the scores of both our model and the one exhibiting the best performance. Our best model, T5-early, outperforms OPT-13B in terms of ROUGE-2 and OPT-30B in terms of ROUGE-L on the XSum dataset, despite having only 11B in size. However, shows slightly lower overall performance compared to OPT-66B. For the WebNLG dataset, model generally scores lower than decoder-only models. Notably,
15

---

Published as a conference paper at COLM 2024
the significant difference in 5-shot performance between T5* and proposed T5-early model indicates that our method meaningfully enhances the generation performance of the seq2seq model, irrespective of the pretrained language model's performance. With the improvements in the seq2seq model's pretraining objectives, we expect a considerable boost in performance.
Training details for baseline models
Seq2seq models that we select as our baseline are trained in various manners. T5 is pre- trained only with the denoising objective, utilizing number of sentinel tokens that indicate where be denoised. However, as the released version of T5 fine-tuned model, while T5.1.1 is trained solely on the C4 dataset with 1T tokens, we opt to utilize T5.1.1 as our T5 baseline. T5-LM is an additionally trained T5 model with a causal language modeling objective. TO is variant of T5-LM, which is fine-tuned with plenty of prompts for various tasks. trained roughly 12B additional tokens compared to T5-LM. While While|Sanh et al. (2022) present three types of T5-architecture models (TO, TO+, and TO++) based variations of fine-tuning datasets, we choose TO to ensure that the datasets used in our experiments have not been previously encountered during the training process. Sentinel tokens were not used for further training both for T5-LM and TO. Finally, UL2 is a variant model of T5, pretrained with three types of denoising objectives: R- S-, and X-denoising, All three objectives utilized the sentinel token, with "R" and "S" respectively corresponding to denoising and causal language modeling objectives. "X" represents the extreme version of both objectives. Similar to 15, it used the C4 dataset to train about trillion tokens.
All four sea2seq baseline models are basically pretrained with sequence lengths of 512 (Rffel et al., 2020 Tay et al. 2022) or 1,024 (Sanh et al. 2022), which account for a quarter or half of the typical decoder LLMs (Brown et al., 2020 Zhang et al. 2022; Rae et al. 2021). Therefore, the seq2seq model faces disadvantage when the number of examples used for few-shot evaluation increases, as lacks exposure to long-length inputs curing training.
According to Zhang et al. 22022), the OPT models are trained on approximately up to 300B tokens, including the Pile (Gao et al. 2020) and the datasets used for RoBERTa (Liu et al. 2019) pretraining. In our experiments, 30B, and 66B), which are up to 6 times larger than the T5 baseline model. Additionally, to compare with decoder models smaller than 13B, we also evaluate the BLOOM-7B model, which is pretrained on approximately 340 billion tokens from the ROOTS corpus.
evaluate three different sizes of OPT models (13B,
G Detailed information about prompts and few-shot demonstrations
For the natural language generation tasks in Table and Table 5
we adopt one of the prompts from the PromptSource (Bach et al. 2022) since Language Model Evaluation Harness does not provide templates for XSum and WebNLG. We manually select one of the prompts from the PromptSource that best describes the task. We provide one-shot prompting samples utilizing the selected template in Figure
With respect to the few-shot demonstrations, we distinguish between two settings where the demonstrations used for each prediction are maintained the same, referred to the "fixed" setting, or randomly sampled at each time, referred to as the "non-fixed" setting. For the generation tasks, we conduct experiments with "non-fixed" setting, otherwise utilize "fixed" setting.
H Normalization to the scores
we do not apply any normalization to the output losses, with respect to the token or string length of the output sequences. In the case of HellaSwag, applying normalization often leads to higher score. However, it may not hold true for other tasks. To ensure consistent evaluation across all datasets, we decide to adopt a unified approach without normalization.
As mentioned in Section 4.2
16

---

Published as a conference paper at COLM 2024
Prompt
Model
Shot
RTE
CB
WSC
COPA
WiC
Boolq*
Multirc*
ReCoRD*
average
52.71
41.07
36.54
59.00
50.00
61.30
42.70
70.10
51.70
52.71
39.29
36.54
79.00
50.00
61.50
42.70
75.10
54.60
5
52.71
41.07
36.54
61.00
50.00
61.50
42.70
71.90
52.18
10
52.71
41.07
36.54
63.00
50.00
61.50
42.70
69.20
52.09
0
52.71
26.79
63.46
71.00
50.31
40.80
57.30
85.60
56.00
47.29
17.86
36.54
63.00
44.83
61.60
49.60
82.20
50.36
T5-LM
5
46.57
35.71
63.46
60.00
50.00
61.30
57 40
78.20
56.61
47.29
41.07
63.46
60.00
50.00
61.50
57.30
74.50
56.89
vanilla
83.75
76.79
73.08
76.00
50.47
73.00
72.70
80.60
73.30
74.73
55.36
68.27
78.00
50.00
65.70
66 60
80.10
67.34
TO
47.29
53.57
63.46
75.00
50,00
56.80
57.30
79.50
60.37
47.29
50.00
70.00
50.00
57.30
77.60
56.77
52.71
21.43
35.58
71.00
48.75
60.60
43.20
85.90
52.39
UL2
52.71
8.93
37.50
64.00
49.84
60.10
42.70
50.10
52.71
8.93
50.31
50.50
43.10
10
52.71
8.93
36.54
59.00
49.84
61.40
42.70
83.30
49.30
33,93
41.35
44.67
71.60
44.90
52.89
60.65
82.14
39.42
84.00
50.00
86.10
43.00
78.90
65.53
T5
52.71
67.88
38.46
84.00
50.00
61.50
42.70
79.10
59.54
10
52.71
51.79
36.54
83.00
54.08
61.50
56.60
76.50
59.09
62.45
48.21
36.54
71.00
50.00
75.20
46.10
86.40
59.49
64.62
53.57
36.54
83.00
50.00
74.30
46.20
85.50
61.72
T5-LM
5
47.29
57.14
44.23
78,00
50.00
61.50
46.90
85.10
58.77
10
47.29
53.57
60.58
79.00
50.00
61.50
43.60
82.40
59.74
w/ sentinel
81.95
78.57
53.85
82.00
55.64
79.40
65.20
80.80
72.18
74.73
58.93
35.58
82.00
50.16
74.30
58.20
80,90
64.35
5
71.84
69.64
52.88
80.00
56.11
70.10
57 60
81.50
67.46
10
47.29
51.79
64.42
82.00
56.90
70.40
57.20
77.40
63.42
0
51.62
33.93
53.85
69.00
51.41
76.60
49.70
84.60
58.84
54.15
42.86
36.54
86.00
51.57
78.20
42.70
88.40
60.05
UL2
49.10
10
50.00
36.54
81.00
50.00
63.60
54.30
86.30
58.85 85
47.29
50.00
36.54
84.00
50.00
68.70
57.30
0
53.79
8.93
47.12
87.00
53.13
80.30
50.40
87.20
58.48
48.01
44.64
46.15
86.00
54.70
88.40
42.90
86.90
62.21
w/ mode
U1.2
48.38
50.00
83.00
50.47
86.30
61.17
47 47.29
50.00
50.00
57.30
Table 8: Detailed results for Table 2 The asterisk(*) on the right side of the task indicates
that due to the large size of the test dataset, evaluation is performed on random sample le of 1K instances from the test dataset. For w/ mode setting, the sentinel token is also utilized. Bold denotes the best average score among vanilla, w/ sentinel, and w/ mode approaches, for the same model and the same number of shots.
RTE
CB
ANLII
ANLI2
ANLBB
WSC
Winogrande
COPA
StoryCloze
HellaSwag
WIC
shots
32
50
32
32
20
32
Table 9: The number of shots with the highest score reported in GPT-3. For each task, the length of concatenated examples approximates the sequence length of the pretrained GPT-3 model. Consequently, conducting few-shot learning with such settings is disadvantageous for seq2seq models, which are pretrained on shorter sequence lengths.
Additionally, in Brown et al. 2020 a pre-normalization step conducted where calibration is applied to each calculation of scores. With the same context mentioned in the paragraph above, we do not implement any preceding calibration to calculate the output scores.
17

---

Published as a conference paper at COLM 2024
Model
RTE
ALNII
ANLIZ
ANII
WSC
Winogande
COPA
StoryCloze
HellaSwag
WIC
average
PaLM-BB
56.7
32.7
81.5
52.4
Pal.M-540BB
81.2
89.3
36.9
56.1
51.2
89.5
95.0
89.0
83.8
64.6
76.5
GeT-NeoX 208
52.2
33.1
31.6
38.5
68.3
52.8
(93.4)
GPT3 1258
729
821
36.8
34.0
40.
75.0
27.7
92.0
877
79.3
55.3
66.6
15varly 118
61.0
77.
42.1
38.3
40.0
68.3
62.5
63.8
25.3
50.4
50,2
59.3(50.3)
Table 10: Comparison with other decoder LLMs for various NLU tasks. The scores for PaLM and GPT-NeoX are 5-shot results. And the scores for GPT3 and T5-early are the highest scores among the reported ones for each task. The average scores inside the parentheses indicate the average scores of tasks where the GPT-NeoX report. Note that normalization is applied to the scores for PaI.M and GPT3, which might be advantageous compared to when it is not applied.
Model
Method
Shot
RT1
ANLIL
ANLI2
ANLB
WSC
Winogrande
COPA
StaryClore
HellaSwagg
WIC
average
sid
originsf
47.65
34.06
77,50
75.34
50.38
69:97
51.17
1.48
47:22
32.96
74.47
50.86
2.10
8p3
47,08
32.42
61.04
77,54
1.63
TS-LM
5
53.88 03.96
56.07
35.62
54.84
36.95
52.50
61.93
82.20
75.53
50.52
69.97
53.64
2.07
anrlyfision
10
$3.29
53.21
35.82
54 56
36-47
$2.96
61.86
61.50
75.00
50.00
50113
53.12
1.88
gp3
52.20
53.21
$5.82
34.46
36.00
60.19
61.83
$7.90
25.25
30.44
50.51
$3.45
1.30
54.22
56.79
35.00
54.45
36.93
90.38
61.16
61.00
75.51
50.30
50.34
52,89
1.61
latesfatton
10
53.14
36.76
61:17
61.20
75.29
50.18
52.92
1.91
8p43
53.29
53.96
36.67
61.32
75,02
1.3L
20
41110
3.80
origina!
53.02
35.62
39.04
45.32
50.50
2775
10
56.10
33.8
$3.80
37.88
77.67
51.62
2.42
gp33
50.32
38.57
33.74
53.36
$3.98
40.35
81.20
71.50
43.10
49.54
49.16
1.99
15
63.61
23.93
40.00
58.15
30.55
62.69
62.46
64.20
77.64
45.00
50.0
58.11
2.00
anlyyfision
10
63.99
27.86
41
38.32
39.56
65.19
62.23
84.80
77.56
45.50
50.00
58.72
1.47
64.01
75.00
42.08
57.4
39.95
68.27
62.32
85.40
77.57
46.06
50.16
58.96
0.97
62.74
58.12
62.00
57.43
fustion
63.32
39.57
62.13
77.31
sp33
38.06
62.00
62.15
$1,80
77.27
15,58
50.22
58.39
1.32
81,95
76.57
10 40
42.42
53.85
60.95
75.95
50.36
55.64
60.65
0.15
71.05
72.90
37 20
38.75
01.73
75
56.86
3.74
originsl
5
67
144
76.07
59.4
56.28
38.33
63.19
61.56
81.50
18
49.06
54.30
58.00
2.73
10
63. 57
60.36
$6.06
3430
35.12
62.88
60.36
81.50
71.28
47.92
52.96
54.27
2.72
8p43
2729
63.21
33.30
53288
33.15
63.85
60.17
79.20
61.96
46.30
50.63
52.61
1.93
TO
5
81
76.43
44.36
50 36
42,60
63.65
61.86
82,20
25.25
50.40
55.27
61.00
2.28
antlyfissin
10
79,78
68.21
64.26
56
11.77
61.23
61.86
62,20
75.05
50.30
55.64
60:15
2.13
gp3
81
70.71
44.38
30 90
41.90
68.08
62.15
63,90
74.86
30.10
56.21
61.10
1.52
30 26
62.12
latesfvston
80.22
62.16
50.00
sp43
1.92
53.79
8.43
33.10
33.00
31.92
07.12
64.17
78.01
$2,46
50.09
0.05
54.87
49.29
34.6
59.7%
33.32
47.50
63.96
85.00
78.51
$6.56
62.20
58.59
2.19
origins!
5
49.03
50.00
33.52
58.52
33.50
36.54
66.35
84.20
77.66
55.96
50.09
51.83
1.72
10
47.51
50.00
33.30
53.30
32.97
36.51
64.67
84.20
76.77
54.50
50.00
51.26
1.41
spd
47.25
50.00
33.50
33:30
33.12
36.54
6384
82.00
63.00
51.64
50.13
49.49
0.53
U12
5
$5.61
47.50
30,88
32.48
32.60
47.88
45:35
$1,80
75.20
$7.12
52.91
53.59
1.25
anllyfissio
10
$4.95
47.50
54.12
31.2
32.23
49.21
65.02
65,00
75:17
36.96
52.79
53.57
1.18
8p43
54.37
51.52
31.58
39.02
68.13
86,00
36.92
53.41
52.78
32.62
32.90
56,80
52.67
1.44
Lile-fusion
53.36
31.94
52.64
1.89
8p3
53.07
31. 53
32
36,90
Table 11: Detailed results for Figure 3 The asterisk(") on the right side of the task indicates that due to the large size of the test dataset, evaluation is performed on a random sample of 1K instances from the test dataset. The underlined values represent the highest scores within each model for each task, while the bold values represent the settings where the average scores for all the tasks are highest within each model.
I Detailed explanations about each benchmark dataset
As outlined in Section 4.2
we adopted the SuperGlue benchmark along with four additional datasets to evaluate natural language understanding ability, and XSum and WebNLG for the generation task. The following provides detailed description of each dataset.
BoolQ (Boolean Question)): BoolQ is dataset comprising natural yes/no questions derived from Google searches, with each question paired with a relevant Wikipedia passage.
18

---

Published as a conference paper at COLM 2024
1-shot
5-shot
1-shot
32-shot
Model
Model
(R1/R2/RL)
(KI/R2/RL)
(R1/R2/RL)
(RI/R2/RL)
15*
13.72/22.6/111.97
7.57/0.66/ 211.66/6,33
15*
18.63/7.98/ /2.5//778//1..5 16.22
0.25/0.13/0.24
T5
25.12/8.69/20.72
T5
39.13/23.44/32.61
41,40/22.63 63/34.09
T5-carly
30.31/11.55/25.1
T5-early
49.47
BLOOM-7B
21.50/4.75/16.33
21.96/5.06/11
BLOOM--B
03.77/88.4/55119
68.07/4335 /56.1
OPT-13B
28.37 /9.37/9.93/22.46
31.32/11.52/24.72
OPT-13B
55.03/33.04/455 /45.53
66.26/43 0466/43.44/555 55.85
ОРT-30B
27.61/10.06/22.222
31.54/12.06/25.19
OPT-30B
05.00/35.355 447.8
63.57/43.07 55..2
OPT-66B
29.31/10.64/2345
22.52/12.86/26.19
OPT-66B
61.44/37.5455055
68.77 /45.77 57.75
Table 12: Evaluation on XSum dataset. The asterisk(*) on the right side of the T5 denotes the case where the sentinel tokens are not used during inference time. R1, R2, and RL denotes ROUGE-1,2,L, respectively.
Table 13: Evaluation on WebNLG dataset. The asterisk(*) on the right side of the T5 denotes the case where the sentinel tokens are not used during inference time. R1, R2, and RL denotes ROUGE-1,2,L, respectively.
CB (ComnitmentBank); is a textual entailment task in which models must judge if given sentence entails, contradicts, or is neutral to a hypothesis. This task tests model's ability understand implications and contradictions in texts.
COPA (Choice of Plausible Alternatives): COPA causality detection task where models are required to determine either the cause or effect from two possible options for given statement.
MultiRC (Multi-Sentence Reading Comprehension): MultiRC involves answering questions about paragraph, with each question having multiple correct answers. It assesses the model's ability to understand and integrate information from multiple sentences.
ReCoRD (Reading Comprehension with Commonsense Reasoning Dataset): ReCoRD is multiple-choice QA task that challenges models to predict masked- out entity from a list of possible entities in the provided passage.
RTE (Recognizing Textual Entailment): RTE requires models to determine whether one sentence logically follows from another.
WiC (Word-in-Context) WiC is a binary classification task for word sense disam- biguation, requiring models determine if a polysemous word is used in the same sense across two different sentences.
WSC (Winograd Schema Challenge): WSC is a coreference resolution task that involves selecting the correct referent of the pronoun in a sentence, requiring commonsense reasoning and everyday knowledge.
HellaSwag: HellaSwag is a benchmark dataset for testing commonsense NLI that is particularly challenging for state-of-the-art models, yet its questions are trivial for humans.
ANLI (Adversarial Natural Language Inference): ANLI is a challenging NLI bench- mark comprising series of progressively harder rounds in which models must identify the relationship (entailment, contradiction, or neutral) between pairs of sentences.
WinoGrande:
WinoGrande is large-scale dataset inspired by the original WSC design, but adjusted to improve both the scale and the hardness of the dataset.
StoryCloze:
StoryCloze requires models read a four-sentence story and then select the correct ending from two options.
WebNLG: WebNLG corpus consists of sets of RDF triplets describing facts and the corresponding facts form of natural language text. It challenges models to generate coherent and fluent text paragraphs from triplets.
XSum (Extreme Summarization): XSum is a dataset designed for evaluating auto- matic text summarization. requires models to create single-sentence summary of news article.
19

---

Published as a conference paper at COLM 2024
Template for XSum: "DOC write summary of above"
Input:
The full cost of damage in Newton Stewart, one of the areas worst affected, is still being assessed. Repair work is ongoing in Hawick and many roads in Peeblesshire remain badly affected by standing water. (ellipsis)
"Obviously it is heart-breaking for people who have been forced out of their homes and the impact businesses." He said it was important that "immediate steps" were taken to protect the areas most...
Write a summary of the text above Clean-up operations are continuing across the Scottish Borders and Dumfries and Galloway after flooding caused by Storm Frank.
[FULL ARTICLE OF TEST INPUT]
Write a summary of the text above
Output:
[MODEL GENERATED SUMMARY]
Template for WebNLG: "Explicit Graph Description"
Input:
Take the following graph comprising triple sets, where each element of triple is separated by "I" and each triple set by ",": Aarhus Airport cityServed "Aarhus, Denmark".
Make a verbalization of the triple set into plain text.
The Aarhus is the airport of Aarhus, Denmark.
Take the following graph comprising triple sets, where each element of triple is separated by "[" and each triple set by ".":
[TRIPLET SET OF TEST INPUT]
Make verbalization of the triple set into plain text.
Output: [MODEL GENERATED VERBALIZATION]
Figure 4: Selected PromptSource templates for evaluating the XSum and WebNLG tasks. Both examples describe the prompt for one-shot learning scenarios.
20


Reviews:
[{"review_id": "w1HJEPMPcv", "reviewer": "Reviewer_Xrdw", "summary": "The paper presents a study on the in-context learning capabilities of seq2seq models (e.g., T5, UL2, etc.). The authors delve into various prompting strategies for seq2seq models and introduce two distinct fusion-based approaches for few-shot learning, drawing from previous work such as RAG and FiD. Finally, the paper compares the permutation bias between fusion-based seq2seq and decoder-only models.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "We appreciate the reviewer's feedback on our work. Here’s our response to your inquiries.\n\n> What is the novelty presented in the paper? It seems that the most relevant results are those using fusion-based approaches.\n> \n\nWe acknowledge that the fusion-based approach adapts previous work with slight modifications. Our contribution is rediscovering seq2seq models as few-shot learners, focusing on applying straightforward and universally applicable methods that align with their structure and objectives. The effectiveness of our objective-aligned prompting and fusion-based approach, while straightforward, is attributed to their simple yet carefully crafted design, and has been validated through extensive experiments for generalizability. Further details on objective-aligned prompting are in the next response.\n\n---\n\n> According to table 1. Are seq2seq models few shot learners if the target input is placed in the encoder?\n> \n\nDeciding where to position the target input in in-context learning is fundamental and precedes other prompting strategies. Existing research has generally overlooked this, making its consideration a prerequisite in our experiments. The results in Table 1 underscore the importance of prompt structuring for in-context learning in seq2seq models. Therefore, we positioned this component at the outset of our experimental approach, allowing all subsequent experiments to be based on these results.\n\n---\n\n> Is there any reason the authors did not include llama2,3 results in their comparison?\n> \n\nWe excluded comparisons with models like Llama2 and Llama3 because comparable encoder-decoder models are unavailable. Demonstrating superiority over state-of-the-art decoder-only models like Llama2 is challenging, as it requires fair comparisons between similarly trained encoder-decoder and decoder-only models, which are rare. Recent research has focused on decoder-only models, often overlooking seq2seq models. Additionally, Llama3 was released after our submission, preventing direct comparison. Thus, our paper shows the seq2seq model’s in-context learning capabilities by comparing them with contemporaneous decoder-only models like OPT and Bloom. \n\n---\n\n> Minor issues\n> \n\nFor clarity on Figure 3's deltas, see the standard deviation in Table 11's last column, Appendix D.\n\nWe will improve Table 1 and Figure 2's readability in the camera-ready version. Thank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper studies an exciting topic: the difference in performance between encoder-decoder and decoder-only models.\n- The paper shows a good number of ablations.\n- The paper is generally well-written and organized but could be improved slightly more for better readability.", "reasons_to_reject": "Main:\n- Novelty: The paper's novelty regarding the fusion-based approach is not too great. This seems more like an extension of the application of previous work, such as RAG and FiD.\n- The authors stay in the abstract: \"Our approach outperforms a decoder-only model that is six times larger and exhibits significant performance improvements compared to conventional seq2seq models.\" This statement is incorrect since you are not comparing it with llama2,3, recent decoder-only architectures that perform better than Bloom or OPT. Also, the subtitle in section 5.1 highlights this without further clarifying why not consider the most recent models.\n- Table 1 is not very informative in terms of few-shot capabilities. Even though we see a significant performance improvement when the target input is in the encoder, I do not see any tendency for more samples to help performance. The authors boldly put these numbers, but this does not agree with the paper's story.\n\nMinor:\n- Table 1 should mention the dataset in which ablations are done. This would improve the readability of the paper.\n- Figure 2 could be improved a lot. Adding the terms from the equation on page 5 will help readers understand easily and faster.", "questions_to_authors": "Questions:\n- Is there any reason the authors did not include llama2,3 results in their comparison? I would like an explanation, if any since we have seen advancements in the performance of LLMs. Compare your numbers with  BLOOM; OPT does not make the paper's story look as good as it could be.\n- According to table 1. Are seq2seq models few shot learners if the target input is placed in the encoder? A clarification of these experiments would be ideal in the paper. \n- What is the novelty presented in the paper? It seems that the most relevant results are those using fusion-based approaches (since these are the only ones where I could see a tendency of more shots helping performance). However, this is more like an extension of previous work.\n- Could you share the dev standard numbers for Figure 3? I would like to see the deltas more clearly.", "extra_scores": {}}, {"review_id": "DlzPzKtgWm", "reviewer": "Reviewer_KQR9", "summary": "First, the paper conducted the first-ever extensive experiment comparing the in-context few-shot learning capabilities of decoder-only and encoder-decoder models on a broad range of tasks. Second, the paper proposes two methods to more effectively elicit in-context learning ability in seq2seq models: objective-aligned prompting and a fusion-based approach.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and helpful feedback. In the following, let us try to address your questions.\n\n> The scientific innovation of the proposed methods is very limited. The idea of early-fusion, late-fusion, and object-aligned prompting is quite straightforward and simple; however, the authors tried to make it look more complicated.\n> \n\nOur primary contribution is to rediscovery of inherent capabilities of seq2seq models as few-shot learners, rather than introducing entirely novel methodologies. Therefore, we focus on applying straightforward and universally applicable methods that align with the natural structure and obejctives of seq2seq models. The effectiveness of our objective-aligned prompting and fusion-based approach, while seemingly straightforward, is attributed to their simple yet carefully crafted design, and has been validated through extensive experiments for generalizability within the seq2seq framework. \n\nWe acknowledge that the manuscript may overcomplicate these methods. We appreciate this observation and will simplify our descriptions in the camera-ready version.\n\n---\n\n> Section 5.1 states that seq2seq models perform better than decoder-only models, which is a very big claim. Do you feel you have enough data points to validate this?\n> \n\nWe utilized the T5-11B model and compared its performance with that of the OPT-66B model, providing preliminary evidence that seq2seq models can, in certain contexts, outperform decoder-only models. However, we acknowledge that our claim could be seen as broad based on the limited data from these two models alone.\n\nGiven the complexity of fair and comprehensive comparisons, which would require equating training datasets, model sizes, and hyperparameters across diverse pretrained models, our study's findings should be seen as initial rather than conclusive. In practice, comparable encoder-decoder and decoder-only models trained under perfectly matched conditions are exceedingly rare. Additionally, the focus in recent research has predominantly been on enhancing decoder-only models with high-quality data to develop advanced skills, leaving seq2seq models less represented in recent in-context learning studies.\n\nTherefore, while our paper presents evidence supporting the competitive in-context learning capabilities of seq2seq models, such as T5, against contemporaneous decoder-only models like OPT, GPT-NeoX, Palm, and Bloom, these findings are specific to the conditions and datasets used.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper conducted the first-ever extensive experiment comparing the in-context few-shot learning capabilities of decoder-only and encoder-decoder models on a broad range of tasks.", "reasons_to_reject": "This paper is more like an internal technical report rather than a research paper since the scientific innovation of the proposed methods is very limited. The paper presents many factual results from experiments, but the discussion of these results is shallow.\n\nThe idea of early-fusion and late-fusion is quite straightforward and simple; however, the authors tried to make it look more complicated. The objective-aligned prompting is also quite straightforward.\"", "questions_to_authors": "Section 5.1 states that seq2seq models perform better than decoder-only models, which is a very big claim. Do you feel you have enough data points to validate this?", "extra_scores": {}}, {"review_id": "Uy0G7YQPE0", "reviewer": "Reviewer_84MH", "summary": "This paper studies the topic of few-shot in-context learning for\nencoder-decoder language models (e.g., T5).  Its main contribution is empirical\nresults detailing the impacts of : 1.  different ways of formulating the\nprompt, and 2. approaches for dealing with context limitations.\n\nOn the first topic, the paper investigates whether it is better to include the\ntarget example in the encoder or decoder, as well as the impact of making the\nprompt resemble the masked language modeling objective.  The results show\nconsistent benefit (across 4 models; T5, T5-LM, T0, and UL2) to placing the\ntarget in the encoder, and using mask tokens in the prompt.  Interestingly, the\nbenefit of the mask tokens is present even for models that are subsequently\ntrained on maskless instructions (e.g., T0).\n\nOn the second topic, the paper investigates the impact of fusion-in-decoder-\nand RAG-style approaches for encoding the in-context examples. These\napproaches have the benefits of: 1. allowing the model to leverage more\nexamples than would effectively fit in its context window, and 2. permutation\ninvariance.  The results establish consistent benefit to both approaches over\nnaive concatenation of examples in the source prompt, with the\nfusion-in-decoder approach slightly edging out the RAG approach.\n\nAnalyses are performed on benchmarks including SuperGLUE, HellaSwag, and XSum.\nModels evaluated include:\n\n- Encoder-decoders: T5, T5-lm, T0, and UL2\n- LLMs: OPT, BLOOM, PALM and GPT3.\n\nAs stated in the previous paragraphs, the key results are that in-context\nlearning consistently works best when:\n\n- The input example is included in the encoder,\n- The prompt uses <mask> tokens to closer resemble the LM's pretraining\n  objective, and\n- RAG/Fusion-in-decoder are used to perform inference.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive opinion and constructive feedback. We will add the two missing references to the camera-ready version, and discuss their relevance to our work.\n\n> I'm somewhat critical of the claim that the approaches explored are being \"proposed\", as opposed to preexisting.\n> \n\nWe acknowledge that our methods, rooted in prior work, make the term \"proposed\" ambiguous. Our contribution is in demonstrating the effectiveness of these techniques in seq2seq in-context learning, which had not been explored.\n\nSeq2seq models offer various structuring options beyond inserting mask tokens. Our findings show that aligning prompts with the model’s pretraining objectives is most effective, an aspect often overlooked and not disclosed in previous studies.\n\nIn the fusion-based approach, we leveraged the structural advantage of seq2seq models for parallel encoding. By adopting methods from FiD and RAG and applying multiple encoders instead of traditional retrieval and scoring modules in few-shot scenarios, we achieved significant performance gains. We hope this innovation is recognized.\n\nWe will clearly articulate the lineage of the methods in the revised manuscript.\n\n---\n\n> The results are somewhat overstated.\n> \n\nWe acknowledge that our claim could seem broad based on limited data from two models alone. \n\nIn practice, comparable encoder-decoder and decoder-only models trained under perfectly matched conditions are exceedingly rare. Recent research has focused on enhancing decoder-only models with high-quality data, often neglecting seq2seq models in in-context learning.\n\nTherefore, while our paper presents evidence supporting the competitive in-context learning capabilities of seq2seq models, such as T5, against contemporaneous decoder-only models like OPT and Bloom, these findings are specific to the conditions and datasets used. \n\nWe will revise our paper to incorporate your feedback. Thank you.\n\n---\n\n> “In the early-fusion method, there is a subtle variation in probability caused by the relative positional bias.” Could you provide a explanation?\n> \n\nIn the early fusion method, each of the examples is processed through the encoder, and their hidden states are concatenated. This is followed by cross-attention with queries from the decoder block. The order of concatenation affects their relative distance to the queries, leading to differences in the relative position bias and resulting in subtle variations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The empirical results will be useful for other practicioners investigating\n  few-shot in-context learning with encoder-decoder LMs. Consistent benefits to\n  including <mask> tokens in the prompt, and RAG/fusion-in-decoder were\n  observed across 4 different models; this provides convincing evidence that\n  future practicioners should adopt these approaches as a standard.\n- The topic of few-shot in-context learning with encoder-decoder LMs has been\n  overlooked, and this paper includes some compelling results that suggest that\n  this line of research merits further investigation. In particular, I think\n  that the following benefits of RAG/fusion-in-decoder are compelling: a)\n  improving the efficiency of inference, and b) eliminating sensitivity to the\n  ordering of the few-shot examples", "reasons_to_reject": "- I'm somewhat critical of the claim that the approaches explored are being\n  \"proposed\", as opposed to preexisting. \"Objective-aligned prompting\" (aka\n  putting mask tokens in the input) is the default approach used  in prior\n  work on finetuning-based few-shot learning with encoder-only LMs [1], and has\n  also been used in prior work on few-shot learning with encoder-decoder LMs\n  [2]. Similarly the fusion-based approaches are just direct applications of\n  fusion-in-decoder and RAG.\n- In my opinion, the results are somewhat overstated. The paper uses the\n  comparison of their best T5-11B-based approach to OPT-66B to justify the\n  claim that \"seq2seq models perform better than decoder-only models\", however,\n  the comparison to PALM 8B in Table 8B in the Appendix seems to indicate that\n  there is no clearly better approach, and I suspect that this claim would be\n  further undermined had the paper benchmarked against Llama or Mistral.\n  Similarly, in the conclusion this paper refers to its results as \"remarkable\"\n  when in my opinion a phrase like \"encouraging\" would be more appropriate.\n\n[1] https://aclanthology.org/2021.eacl-main.20/\n[2] Patel, et al., 2022", "questions_to_authors": "Questions:\n\nIn Section 5.4 you state:\n\n> In the early-fusion method, there is a subtle variation in probability caused by the relative\npositional bias when the order of the examples is fused.\n\ncould you provide a formal explanation of this?\n\nMissing Citations:\n\n- This is concurrent work but very similar: https://arxiv.org/abs/2308.07922\n- I'd also recommend citing the Shick and Schutze paper above and discussing a bit the parallels to works on finetuning-based approaches for few-shot learning w/ encoder-only models from a few years ago.", "extra_scores": {}}, {"review_id": "2Q4HsngeoP", "reviewer": "Reviewer_TDzM", "summary": "This paper explores the potential of sequence-to-sequence (seq2seq) models for few-shot in-context learning. While few-shot in-context learning has proven effective in decoder-only neural language models, its application in seq2seq models remains underexplored. The authors conducted ablation studies using various prompting techniques, including sentinel tokens and mode tags, yielding several key findings:\n\n- Incorporating the target input within the encoder significantly improves results, likely due to the bidirectional nature of seq2seq encoders.\n- The paper proposes two fusion methods: a late-fusion reminiscent of Retrieval-Augmented Generation (RAG) and an early-fusion that concatenates encoder hidden states from previous prompts as input for the decoder. The early-fusion method consistently outperforms the late-fusion across various metrics. The fact that seq2seq decoders cherry-pick where to attend on the encoder side might explain why this is happening. Both fusion methods are shown to be more effective than the vanilla approach.\n- Impressively, the few-shot seq2seq models demonstrated superior performance to decoder-only models six times their size, indicating a promising direction for future research.\n\nAlthough the individual contributions may not appear novel when scrutinized in isolation, the collective advancements position this paper as a significant milestone. The clear writing, robust experimental setup, and promising results collectively suggest that seq2seq models deserve renewed attention in the wake of decoder-only models' popularity.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer TDzM for your positive opinion. We appreciate your recognition of our paper’s clarity, robust experimental setup, and promising results. Below are our thoughts to further address your questions.\n\n> I am really curious how your method would work on zero and few shot translation. Do you think there is a way to use your current setting to do that?\n> \n\nCertainly, the same method can be applied to translation tasks. However, our fusion-based approach is not applicable in a zero-shot setting, making it more effective in a few-shot setting. We will conduct additional experiments to include this aspect in the camera-ready version. Thank you for the suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The clear writing, robust experimental setup, and promising results collectively suggest that seq2seq models deserve renewed attention in the wake of decoder-only models' popularity.", "reasons_to_reject": "NA", "questions_to_authors": "I am really curious how your method would work on zero and few shot translation. Do you think there is a way to use your current setting to do that?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies in-context learning (ICL) in sequence-to-sequence models. ICL remains under-explored and poorly understood for sequence to sequence architectures compared to decoder-only models. The study is deemed valuable by the reviewers. The authors are highly encouraged to incorporate the reviewers' feedback to improve the paper.
Decision: Accept