Exploiting the Potential of Seq2Seq Models as Robust Few-Shot Learners

Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 16:17:24

Abstract: In-context learning, which offers substantial advantages over fine-tuning, is predominantly observed in decoder-only models, while encoder-decoder (i.e., seq2seq) models excel in methods that rely on weight updates. Recently, a few studies have demonstrated the feasibility of few-shot learning with seq2seq models; however, this has been limited to tasks that align well with the seq2seq architecture, such as summarization and translation. Inspired by these initial studies, we provide a first-ever extensive experiment comparing the in-context few-shot learning capabilities of decoder-only and encoder-decoder models on a broad range of tasks. Furthermore, we propose two methods to more effectively elicit in-context learning ability in seq2seq models: objective-aligned prompting and a fusion-based approach. Remarkably, our approach outperforms a decoder-only model that is six times larger and exhibits significant performance improvements compared to conventional seq2seq models across a variety of settings. We posit that, with the right configuration and prompt design, seq2seq models can be highly effective few-shot learners for a wide spectrum of applications.

Main paper: 
 introduction: 
 Recent studies have demonstrated that large language models can possess entirely different competencies, referred to as emergent abilities (Brown et al., 2020; Chowdhery et al., 2022; Rae et al., 2021; Wei et al., 2022b) . The concept of emergent abilities in Large Language Models (LLMs), initially introduced by Wei et al. (2022b) , posits that increasing the model size and dataset can lead to the sudden emergence of abilities such as in-context learning, complex reasoning, and commonsense reasoning. In-context learning, in particular, one of the distinct characteristics of LLMs, serves as a key metric for assessing their effectiveness.In-context learning refers to the ability of the model to perform tasks by leveraging contextual information provided through prompts, without the need for additional training (i.e., without weight updates). Specifically, in the case of in-context few-shot learning, the model can generate suitable outputs for the desired target input by using a small number of examples. This offers a step beyond traditional frameworks, which typically require extensive training data and fine-tuning of the model. Instead, it presents a new paradigm where a single model can effortlessly perform new tasks without necessitating a separate training process.The capability of in-context learning has been predominantly explored in decoder-only models, as the rapid evolution of pretrained models has mostly focused on these unidirectional architectures. However, the sequence-to-sequence (seq2seq) architecture, despite its significant advantage of encoding contexts without sacrificing bidirectionality, has not been extensively investigated regarding its potential for in-context learning capabilities (Sanh et al., 2022; Lester et al., 2021; Tay et al., 2022; Soltan et al., 2022; Patel et al., 2022; Wang et al., 2023) . Sanh et al. (2022) trained an encoder-decoder model using multitask prompts in a supervised manner, but this approach only enabled zero-shot generalization to new tasks. Soltan et al. (2022) primarily demonstrated the in-context abilities of seq2seq models for generative tasks such as summarization and translation, which are tasks inherently well-suited for seq2seq models. They also reported performance on natural language understanding (NLU) benchmarks but only in a zero-shot scenario, making it difficult to confirm their in-context learning proficiency across a wide range of tasks.Figure 1 : Different prompting strategies for in-context learning. (a) A target input can be placed to the encoder side, concatenated with few-shot examples, or decoder side standalone. (b) Examples of pretraining objective aligned prompt. In alignment with T5, the sentinel token is attached to the target output. In alignment with UL2, the mode tag is added as a prefix to the encoder input.Motivated by the current landscape, for the first time, we thoroughly investigate the zero-shot to few-shot performance of seq2seq models across a wide range of evaluation sets. Our findings demonstrate that seq2seq models also serve as robust few-shot learners not only for generation tasks but also for understanding tasks, even outperforming their larger decoder-only counterparts (Zhang et al., 2022) .We conduct various experiments on how to structure the prompts in seq2seq models. While Raffel et al. (2020) ; Tay et al. (2022) popularized the usage of sentinel tokens during the pretraining stage of seq2seq models to optimize the denoising objective, the proper application of the sentinel tokens during the inference stage (i.e., in-context learning) is less well-established, and is often not even mentioned in most studies (Sanh et al., 2022; Tay et al., 2022) . We find that aligning prompts with the pretraining objective yields up to a +20.5%p performance improvement on the SuperGLUE (Wang et al., 2019) benchmark. Furthermore, we propose two fusion-based approaches that enhance the few-shot learning capability of encoder-decoder architecture models. The fundamental idea is to independently process each of the few-shot examples through an encoder and then merge these representations for decoding. This fusion-based approach offers several advantages. Firstly, it addresses the restricted maximum sequence length commonly found in seq2seq models (Raffel et al., 2020; Tay et al., 2022; Lester et al., 2021) , which is shorter than decoder-only models, by leveraging encoding parallelization. Secondly, we achieve more efficient encoding of multiple examples by avoiding unnecessary bidirectional attention calculations between them. Our approach demonstrates significant performance improvements in seq2seq settings, surpassing the OPT 66B model (Zhang et al., 2022) across various tasks. Despite the absence of complete attention across the contexts of the few-shot prompts, our approach consistently outperforms traditional methods. Moreover, our methodologies effectively eliminate the permutation bias commonly observed in few-shot learning scenarios.Overall, our work reveals the few-shot capability of seq2seq models, which has been undervalued compared to their zero-shot and fine-tuned counterparts. To summarize, our key contributions are: 1) we develop an in-context evaluation toolkit for seq2seq models and conduct extensive experiments to investigate the performance of seq2seq models in zero-shot to few-shot scenarios using fair criteria, 2) we exploit the potential of few-shot learning in encoder-decoder models by exploring prompting strategies and fusion-based approaches, and 3) we experimentally demonstrate that the seq2seq model can outperform the decoder-only model with 6 times larger parameters across diverse datasets. In Section 7, we discuss the impact of these unprecedented findings on the evolution of LLMs.We begin by demonstrating the different prompting strategies suitable for in-context learning of encoder-decoder models. Since decoder-only models share a unified left-to-right attention architecture (Radford & Narasimhan, 2018) , we can naturally feed the demonstrations and target input in a sequential manner and perform generation to obtain the target output. For instance, in a 5shot English-French translation task, the input sequence is formed by concatenating five English-to-French demonstrations, followed by target English input, with the model expected to generate the corresponding French translation as the target output.However, encoder-decoder models process inputs and outputs independently from the encoder and decoder, respectively, which can result in varying compositions of demonstrations, target input, and target output, as shown in Figure 1 -(a). The target input can be utilized either as an encoder input, resulting in the decoder generating only the target output, or as a decoder input, enabling the decoder to directly generate the answer conditioned on the target input. We evaluate this ablation on the SuperGLUE dataset, with 1, 5, and 10-shot settings using four representative encoder-decoder models. Table 1 shows that positioning the target input on the encoder side results in enhanced performance for all model types, with performance gains of up to +20.5%p. We interpret these results as follows: During the pretraining stage, the model generates output conditioned on bidirectionally encoded information from the encoder, instead of relying on the preceding textual information from the decoder. As a result, placing the target input on the encoder side shows better performance.Additionally, recent state-of-the-art models include T5 (Raffel et al., 2020) and T5 variants (Lester et al., 2021; Sanh et al., 2022; Tay et al., 2022; Patel et al., 2022) We argue that, even when performing downstream tasks, it is crucial to design the prompt to resemble the pretraining scheme to achieve optimal performance. For instance, T5 manipulates inputs by masking consecutive tokens of the original text with a single sentinel token and then generates the target sequence by conditioning on that sentinel token. Thus, unlike decoder-only models that simply list examples as inputs, it is a more natural setting to prepend a sentinel token to the target for in-context learning using T5. As depicted in Figure 1 -(b), we conduct experiments by incorporating a sentinel token and a mode tag into the vanilla prompt format. Since the mode tag is exclusively used in UL2, it is not applied to the remaining models. We assess the performance on the Su-perGLUE datasets with various number of few-shot examples. In accordance with the pretraining methodologies of the models, we position the sentinel token at the end of the input and the model tag at the beginning. The results are presented in Table 2 . Following the pretraining objective yields a maximum performance gain of up to +13%p. As expected, aligning with the pretraining objective produces the most favorable results, with a significant discrepancy in scores. It is noteworthy that even T5-LM (Lester et al., 2021) and T0 (Sanh et al., 2022) , which have subsequent training phase without using sentinel token, demonstrate the positive impact of adding the sentinel token to the prompt. Throughout the following sections, we employ the optimal objective-aligned prompting strategies that we identified in this section, as the default configuration in all experiments. 
fusion-based approaches for few-shot learning: 
 In this section, we address another factor that impairs the performance of few-shot learning and propose two fusion-based approaches to mitigate this problem. T5-family models (Raffel et al., 2020; Lester et al., 2021; Sanh et al., 2022; Tay et al., 2022) utilize relative position encoding (RPE) (Shaw et al., 2018) , thereby enabling to process long input sequences as long as computing memory allows. However, as shown in Press et al. (2022) , the extrapolation ability of T5-style RPE diminishes when the input length exceeds twice the maximum pretrained sequence length. That is, encoder-decoder models with shorter training sequence lengths of 512 (Raffel et al., 2020; Tay et al., 2022) , compared to decoder-only models (Zhang et al., 2022; Brown et al., 2020; Black et al., 2022) , often exhibit limited performance when dealing with long-context few-shot learning scenarios. Additionally, there is a challenge in terms of computational cost, which escalates quadratically as the number of shots increases. Another challenge to address is the presence of permutation bias. The order of demonstrations affects the positional embedding of tokens, resulting in different predictions even when the examples are identical but the order varies.To address these problems, we propose a fusion-based approach, in which each demonstration is independently processed by the encoder. Afterwards, we merge them together at a specific stage. We hypothesize that the encoding of relations between demonstrations does not significantly impact in-context learning performance. We categorize this approach into two styles: early-fusion and late-fusion, differentiated by the stage at which the fusion occurs. These concepts are inspired by retrieval-based generation models. Lewis et al. (2020) propose Retrieval-Augmented Generation (RAG), where each retrieved document is integrated with the query and passed through the model independently, and the final output probability is calculated as a weighted sum of the probabilities of each output. Fusion-in-Decoder (Fid), introduced by Izacard & Grave (2021), differs from RAG in that the inputs passed through the encoder are concatenated before being sent to the decoder without any weighting. This allows the model to perform fusion at the decoder level. We apply these ideas to few-shot learning of seq2seq models by treating each few-shot example as a retrieved document. Therefore, we use the term late-fusion to refer to the RAG-style method that combines the decoder outputs, whereas the term early-fusion refers to the FiD-style method that combines the encoder outputs. The formulas are described below. The traditional seq2seq in-context learning, which we refer to as the original method, constructs the encoder input by concatenating the target input x with the few-shot examples z to generate the target output y in the decoder (Raffel et al., 2020; Tay et al., 2022) . The probability of the target output for original method can be expressed asP origin (y|x, z) ≈ N i f dec (y i |f enc (z 1:k , x), y 1:i-1 )where f enc and f dec are encoder and decoder, respectively, and k denotes the number of shots. Here, y i is the i-th target token and y 1:i-1 is the partial output. In the late-fusion approach, as depicted in Figure 2 -(b), j-th example z j is concatenated to x and each combination is utilized separately as the model input. At each token generation step i, the information from k instances that have passed through the entire model is aggregated as follows:P late (y|x, z) ≈ N i k j f dec (y i |f enc (z j , x), y 1:i-1 )Unlike the original RAG model, in our few-shot settings, examples are obtained without going through the retrieval process. Therefore, we consider all examples to be equally retrieved, and no weighting is applied to individual examples.Instead of combining the decoder outputs, early-fusion approach merges the information from the encoder output and passes it to the decoder, as shown in Figure 2-(a) . All the last hidden states of the encoder, h 1 , h 2 , ..., h k , where h j = f enc (z j , x), are concatenated sequentially and used for cross-attention,P early (y|x, z) ≈ N i f dec (y i |[h 1 , h 2 , ..., h k ], y 1:i-1 )These fusion-based approaches successfully mitigate the extrapolation problem by maintaining a shorter length of the encoder input. Moreover, inference time can be reduced in proportion to the number of examples k through batch processing. Further, our approaches can be interpreted as an ensemble strategy, where each example in a different encoder contributes a unique perspective, enhancing overall prediction accuracy. In the following sections, we experimentally demonstrate the capability of the seq2seq model as a robust few-shot learner by integrating the prompting strategies proposed in Section 2 with the fusion approaches presented in this section. 
experimental setup: 
 We provide a detailed description of our systematic experimental setup in the following paragraphs.To ensure a fair evaluation, we employ identical prompt structures, evaluation tasks, scoring methods, prompt templates, and few-shot demonstrations across all baseline models.To test the impacts of our proposed methods on seq2seq models, we employ T5 and its variants, including T5-LM, T0, and UL2, as baseline models. Among seq2seq baseline models, T0 is the only model fine-tuned with plenty of zero-shot multitask prompts, and UL2 contains 20B parameters while others contain 11B. For the comparison with decoder-only models, we adopt the following as decoder baselines: OPT (Zhang et al., 2022) and BLOOM (Workshop, 2023) , with parameter sizes ranging from 7B to 66B. We provide detailed information about our baseline models in Appendix F.In Table 1 and 2 , we evaluate the models on eight subtasks from SuperGLUE benchmarks to evaluate natural language understanding ability. For comprehensive evaluation across various types of tasks, we additionally assess the baseline models on 11 tasks selected by Sanh et al. (2022) including five subtasks from SuperGLUE, Hellaswag (Zellers et al., 2019) , ANLI (Nie et al., 2020) , Winogrande (Sakaguchi et al., 2019) , and StoryCloze (Mostafazadeh et al., 2016) . We report accuracy as the metric, following the approach by Brown et al. ( 2020), and the model selects the option with the lowest cross-entropy loss among all the available multiple-choice options. Any form of normalization with respect to the target output is not applied. Please refer to Appendix H for justification regarding normalization, and to Appendix I for detailed explanations of each benchmark dataset.We adopt two following generation tasks for the comprehensive evaluation: XSum (Narayan et al., 2018) and WebNLG (Moryossef et al., 2019) , which are summarization and data-to-text tasks, respectively. All the samples are generated with greedy decoding, and the ROUGE (Lin, 2004 ) metric is used for the evaluation of generation tasks.We employ prompt templates offered by Language Model Evaluation Harness (Gao et al., 2021) , a widely adopted few-shot evaluation toolkit for decoder-only models. Additionally, for each generation task, we adopt a single template offered by PromptSource (Bach et al., 2022) since the templates for the generation tasks we utilize are not supported by Language Model Evaluation Harness. We conduct five experiments with different random seeds and present the averaged scores for Table 3 and Figure 3 . For the other evaluations, we conduct a single experiment. For more detailed information about prompts and few-shot demonstrations, please refer to Appendix G. 
seq2seq models perform better than decoder-only models: 
 In Table 3 , we evaluate both encoder-decoder and decoder-only models on a broad range of NLU benchmarks. We select T5 as the primary baseline model to apply our approaches because it is the most commonly used seq2seq pretrained model. Furthermore, T5-LM and T0 are models that have undergone further training with different objectives, making it unfair to compare them against other pretrained models. As shown in Table 3 , T5 with both earlyand late-fusion approaches achieve superior performance over OPT-66B model, which is six times larger in parameter size, with a maximal margin of 3.6%p in average. Even with the best-shot configuration found by GPT-3 (Brown et al., 2020) , our methods consistently exhibit superior performance compared to the decoder-only baselines in most cases, with a significant margin. Note that all the T5 variants reported in Table 3 already incorporate the objective-aligned prompt designs suggested in Section 2.Complete results for the remaining seq2seq baselines (i.e., T5-LM, T0, and UL2) can be found in Appendix D. Additionally, we present the results of our method in comparison to other well-known decoder-only LLMs (i.e., GPT-3, PaLM, and GPT-NeoX) in Appendix C. Figure 3 : Comparison of in-context learning ability among seq2seq models by increasing the number of shots. We experiment on the same tasks as presented in Table 3 and report the average accuracy and standard deviation results for four seq2seq baseline models.In Figure 3 , we examine the in-context learning abilities of different seq2seq baseline models and observe a trend that varies with the number of shots. Our approach demonstrates its effectiveness by consistently improving the performance when applied to all models. The black lines, which represent the original method, predominantly exhibit a decline in score in the few-shot setting, when compared to zero-shot or one-shot inference.In contrast, the results obtained by applying our approach show a completely different tendency.The red and green lines, representing the application of early-fusion and late-fusion, respectively, exhibit an upward curve, indicating that the few-shot learning performance outperforms the 1-shot performance. The only exception is UL2 where the few-shot performance remains equal to the 1-shot performance.Surprisingly, despite T0 being fine-tuned with zero-shot prompts, both of our approaches result in higher performance with the GPT-3 best-shot setting compared to the zero-shot result of the original T0 model. Considering that all four models were trained differently, this result reinforces the general applicability of our approach to seq2seq models, irrespective of how they were previously trained.Additionally, we also observe that early-fusion consistently outperforms late-fusion by a small margin. Generally, the pretraining objective of encoder-decoder models is designed such that the decoder aggregates the outputs of the encoder. This process maximizes the joint probability of the input text sequence for the decoder, achieved through the encoder-decoder attention module. In this context, early-fusion implicitly selects examples that assist in resolving the test query by fully leveraging the encoder-decoder attention module. On the other hand, late-fusion does not differentiate whether individual examples aid in solving the test query; it simply aggregates all responses.In essence, late-fusion does not fully utilize the encoder-decoder attention module. Thus, we posit that the capability of the early-fusion to selectively prioritize certain examples, an ability absent in the late-fusion, is a major factor contributing to its superior performance. For detailed scores and standard deviations for each setting, please refer to Appendix D.So far, we have validated our method across a range of natural language understanding tasks. In this subsection, we verify the robustness of our method by evaluating it on several generation tasks. As mentioned in Section 1, generation tasks, such as summarization and translation, align well with the objective of the encoder-decoder architecture and have been proven to be successful with seq2seq models. However, there is still a lack of organized evaluation of the few-shot learning ability on 2022) report the few-shot results of seq2seq models for XSum dataset, those results only encompass one-shot learning. Consequently, to the best of our knowledge, this is the first work that explores the results of "few"-shot learning for seq2seq models on the generation tasks.The application of objective-aligned prompting and fusion-based approaches shows a remarkable improvement in the summarization task compared to the original T5 model, as shown in Table 4 .A similar trend holds in WebNLG, a data-to-text task, as shown in Table 5 . This implies that our proposed approach not only serves as a robust few-shot learner for understanding tasks but also for generation tasks. The analysis comparing decoder-only models in generative tasks can be found in Appendix E.One challenge of in-context few-shot learning is the potential variation in model predictions due to the order in which demonstrations are fed to the model. The late-fusion approach eliminates the so-called permutation bias problem, as it operates by fusing the probabilities of decoder outputs regardless of the order in which the inputs are fed. To verify the effectiveness of our methods in reducing permutation bias, we conduct experiments by reordering the few-shot examples. We select four tasks, CB, COPA, WSC, and WiC, from each of the four task taxonomies introduced by Sanh et al. ( 2022): natural language inference, sentence completion, coreference resolution, and word sense, to cover a diverse range of task types. We examine the permutation bias for a 5-shot setting and randomly sample 50 test sets from each task using the same random seed. In Table 6 , we report the average and standard deviation for all 120 possible permutations with 5-shot examples for each task. Surprisingly, both the early-fusion and late-fusion methods demonstrate zero standard deviations for all tasks, thanks to the elimination of order relation on the encoder side. In the earlyfusion method, there is a subtle variation in probability caused by the relative positional bias when the order of the examples is fused. However, this slight difference does not have any impact on the metric score. In contrast, the decoder-only baseline and the original encoder-decoder baseline show higher standard deviations with respect to the permutation of examples. 
related work: 
 From UniLM (Dong et al., 2019) to BART (Lewis et al., 2019) , T5, UL2, and AlexaTM (Soltan et al., 2022), encoder-decoder architecture models have continuously advanced and set new records by introducing novel denoising pretraining objectives. They excelled in tasks that require a com-prehensive understanding of the input context and generate output based on it, such as translation, summarization, and other sequence-to-sequence tasks. However, this was particularly evident during the fine-tuning process. The seq2seq models have had limitations in handling sequences long enough, and it was only after the T5 series of papers that they were able to address this to some extent by incorporating relative position encoding. As a result, in-context few-shot learning, which necessitates a long encoder input length, has received relatively less attention. As a consequence, despite the recent explosive growth of LLMs, the size of encoder-decoder models has remained stagnant, with less than 20 billion parameters (Tay et al., 2022; Soltan et al., 2022) . This stands in contrast to decoder-only models, which have reached the scale of hundreds of billions (Brown et al., 2020; Chowdhery et al., 2022; Zeng et al., 2022) .In-context learning has emerged as an alternative to the fine-tuning paradigm (Devlin et 2022) mixed causal language modeling with denoising objectives. However, these approaches are restricted to particular setups and there has been a lack of structured results regarding few-shot learning. 
discussion and conclusion: 
 In this work, we demonstrate that the seq2seq model with proper adaptation enables few-shot learning across a broad range of tasks. To the best of our knowledge, no previous research has fully exploited the potential of seq2seq models as robust few-shot learners, outperforming decoder-only models, particularly on language understanding tasks. Our remarkable results are attributed to two factors: 1) the alignment of prompts with pretraining objectives and 2) the use of fusion-based methods that independently process examples, compensating for the objective incompatibility and structural shortcomings of seq2seq models, respectively. Through the carefully designed experiments, we verify that our approach exhibits a significant performance advantage under diverse conditions, including varying the number of shots, different baseline models, and a range of tasks, even with the complete removal of permutation bias. Another major contribution of our work is the provision of a unified in-context learning evaluation toolkit for seq2seq models. We have conducted controlled experiments to systematically compare and analyze the ability of existing seq2seq models -a process not previously undertaken. We plan to release the toolkit. Importantly, our findings rediscover the strengths of the encoder-decoder architectures and shed new light on their potential as conversational agents (e.g., GPT-4 (OpenAI, 2023)), which have been underestimated. We believe that the evolution of seq2seq models is still in progress, and our approach offers a substantial contribution toward maximizing their in-context learning capability. 
limitations: 
 Apart from the reported results in this study, some discrepancies between benchmark performance and qualitative effectiveness still remain. The benchmark score itself cannot reflect linguistic abilities such as natural responses or reasoning, as recently showcased by GPT-4. Furthermore, even though we experiment on the baseline models with a similar size, there are still a few mismatches in terms of throughput, the composition of pretraining data, hyper-parameters, and the number of tokens used for pretraining. These differences are further discussed in Section F. In addition, there is a limitation in that our framework has not been applied to the BART-like pretrained encoder-decoder models. This is due to the fact that the large publicly available seq2seq models, which exceed a size of 10B, are only from the T5-family. 
 
 Appendix: We complement our paper with additional experimental results and miscellaneous details throughout this material. Section A provides the complete experimental results for Table 1 regarding the proper placement of the target input in the encoder-decoder model. Section B exhibits the impact of objective-aligned prompting for in-context learning, which complements Table 2 . Moreover, we report the few-shot learning performance of other decoder-only LLMs that are not featured in the main paper, in Section C. Section D presents detailed numerical values utilized for reporting Figure 3 . In Section E, we compare our methods with decoder-only models on generative tasks. Section F offers detailed information regarding the baseline models used in our experiments. Section G demonstrates examples of the prompts utilized in the generation tasks. Section H provides an explanation of how normalization is considered when measuring few-shot scores. Lastly, Section I gives explanations for each of the benchmark datasets.A DETAILED RESULTS FOR TABLE 1 Table 1 verifies that the placement of a target input greatly impacts the in-context learning performance of seq2seq models. Table 7 provides detailed results of whether it is advantageous to have a target input in the encoder or the decoder, based on each task and each number of examples. In most cases, the target input given to the encoder produces significantly better results than that of the decoder. Although there are a few instances where the scores are improved when the input is given to the decoder for specific combinations of models and tasks (e.g., UL2 with RTE task, T5-LM with WSC task), these are small exceptions that make it difficult to identify trends.In Table 8 , we present the complete results that complement Table 2 . Without using sentinel tokens, T5 shows consistent scores across five out of eight tasks, regardless of the variation in the number of shots. However, this issue is resolved when sentinel tokens are used. Since T0 is further trained with the zero-shot prompts, we can observe that without the use of sentinel tokens, the score is initially higher at zero-shot results but significantly decreases as the number of demonstrations increases. On the other hand, when sentinel tokens are employed during inference, even though the highest scores are still observed in zero-shot scenarios, the performance is almost maintained as the number of demonstrations increases. Based on these observations, we conclude that aligning the prompt structure with the pretraining objective generally helps the seq2seq model better understand fewshot examples. Note that the scores reported in Table 8 are not the ones that applied our proposed methodologies like earlyor late-fusion.While numerous LLMs have their own benchmark evaluation scores reported, it is not guaranteed that these models are evaluated using the same baseline. Despite this potential discrepancy, to provide a point of reference, we present the results of our method alongside those of other well-known decoder-only LLMs in Table 10 including GPT-3, PaLM (Chowdhery et al., 2022) and GPT-NeoX (Black et al., 2022) . All the records of decoder LLMs in Table 10 are from the paper. Compared to PaLM-8B, T5 with early-fusion achieves a higher average score. Although there is only a small overlap in tasks, T5-early beats GPT-NeoX on average and demonstrates much better performance in ANLI tasks compared to GPT-3. This indicates that our model performs well considering its size, as the throughput of the seq2seq model is comparable to that of a half-sized decoder-only model, as stated by Tay et al. (2022) . In this regard, it is unfortunate that the absence of a larger encoderdecoder model hinders us from making an equivalent comparison with decoder models larger than 100B. In order to thoroughly analyze the performance of seq2seq baselines, we report the additive results evaluated on T5, T5-LM, T0, and UL2 for Table 3 in Table 11 . Although we compare various decoder-only models with T5 (with our methodologies applied), other encoder-decoder models such as T5-LM and UL2 also achieve significant scores that approximate those of the OPT-30B model, which is much larger.Table 11 is also the supplementary table for Figure 3 , which depicts the average and standard deviation results for four seq2seq baselines for each methodology. The "std" values we report do not represent the standard deviation of scores across different tasks, but are calculated by averaging the standard deviation values across all tasks, with five seeds evaluated on each task. By applying our approaches, the average scores consistently increase for all models, while simultaneously reducing the standard deviation.The pretraining objectives of existing seq2seq models aren't ideally configured for free response generation; their focus was more on improving downstream task capabilities through span corruption and local reconstruction objectives, rather than on language modeling objectives. Consequently, for generative tasks, they might be less effective compared to decoder-only models. To illustrate this, we conducted comparisons with decoder-only baselines on representative generation tasks, XSum and WebNLG, with the results detailed in Table 12 and Table 13 , respectively. We highlight the scores of both our model and the one exhibiting the best performance. Our best model, T5-early, outperforms OPT-13B in terms of ROUGE-2 and OPT-30B in terms of ROUGE-L on the XSum dataset, despite having only 11B in size. However, it shows slightly lower overall performance compared to OPT-66B. For the WebNLG dataset, our model generally scores lower than decoder-only models. Notably, the significant difference in 5-shot performance between T5* and our proposed T5-early model indicates that our method meaningfully enhances the generation performance of the seq2seq model, irrespective of the pretrained language model's performance. With the improvements in the seq2seq model's pretraining objectives, we expect a considerable boost in performance.Seq2seq models that we select as our baseline are trained in various manners. T5 is pretrained only with the denoising objective, utilizing a number of sentinel tokens that indicate where to be denoised. However, as the released version of T5 is a fine-tuned model, while T5.1.1 is trained solely on the C4 dataset with 1T tokens, we opt to utilize T5.1.1 as our T5 baseline. T5-LM is an additionally trained T5 model with a causal language modeling objective. T0 is a variant of T5-LM, which is fine-tuned with plenty of prompts for various tasks. It trained roughly 12B additional tokens compared to T5-LM. While Sanh et al. (2022) present three types of T5-architecture models (T0, T0+, and T0++) based on variations of fine-tuning datasets, we choose T0 to ensure that the datasets used in our experiments have not been previously encountered during the training process. Sentinel tokens were not used for further training both for T5-LM and T0. Finally, UL2 is a variant model of T5, pretrained with three types of denoising objectives: R-, S-, and X-denoising. All three objectives utilized the sentinel token, with "R" and "S" respectively corresponding to denoising and causal language modeling objectives. "X" represents the extreme version of both objectives. Similar to T5, it used the C4 dataset to train about 1 trillion tokens.All four seq2seq baseline models are basically pretrained with sequence lengths of 512 (Raffel et al., 2020; Tay et al., 2022) or 1,024 (Sanh et al., 2022) , which account for a quarter or half of the typical decoder LLMs (Brown et al., 2020; Zhang et al., 2022; Rae et al., 2021) . Therefore, the seq2seq model faces a disadvantage when the number of examples used for few-shot evaluation increases, as it lacks exposure to long-length inputs during training.According to Zhang et al. (2022) , the OPT models are trained on approximately up to 300B tokens, including the Pile (Gao et al., 2020) and the datasets used for RoBERTa (Liu et al., 2019) pretraining.In our experiments, we evaluate three different sizes of OPT models (13B, 30B, and 66B), which are up to 6 times larger than the T5 baseline model. Additionally, to compare with decoder models smaller than 13B, we also evaluate the BLOOM-7B model, which is pretrained on approximately 340 billion tokens from the ROOTS corpus.For the natural language generation tasks in Table 4 and Table 5 , we adopt one of the prompts from the PromptSource (Bach et al., 2022) since Language Model Evaluation Harness does not provide templates for XSum and WebNLG. We manually select one of the prompts from the PromptSource that best describes the task. We provide one-shot prompting samples utilizing the selected template in Figure 4 .With respect to the few-shot demonstrations, we distinguish between two settings where the demonstrations used for each prediction are maintained the same, referred to as the "fixed" setting, or randomly sampled at each time, referred to as the "non-fixed" setting. For the generation tasks, we conduct experiments with a "non-fixed" setting, otherwise utilize a "fixed" setting.As mentioned in Section 4.2, we do not apply any normalization to the output losses, with respect to the token or string length of the output sequences. In the case of HellaSwag, applying normalization often leads to a higher score. However, it may not hold true for other tasks. To ensure consistent evaluation across all datasets, we decide to adopt a unified approach without normalization. As outlined in Section 4.2, we adopted the SuperGlue benchmark along with four additional datasets to evaluate natural language understanding ability, and XSum and WebNLG for the generation task.The following provides a detailed description of each dataset.• BoolQ (Boolean Questions): BoolQ is a dataset comprising natural yes/no questions derived from Google searches, with each question paired with a relevant Wikipedia passage. • CB (CommitmentBank): CB is a textual entailment task in which models must judge if a given sentence entails, contradicts, or is neutral to a hypothesis. This task tests a model's ability to understand implications and contradictions in texts.We will release the toolkit for the in-context evaluation of seq2seq models. 


Decision: This paper studies in-context learning (ICL) in sequence-to-sequence models. ICL remains under-explored and poorly understood for sequence to sequence architectures compared to decoder-only models. The study is deemed valuable by the reviewers. The authors are highly encouraged to incorporate the reviewers' feedback to improve the paper.

Reviews: review: paper_summary: paper_summary: The paper presents a study on the in-context learning capabilities of seq2seq models (e.g., T5, UL2, etc.). The authors delve into various prompting strategies for seq2seq models and introduce two distinct fusion-based approaches for few-shot learning, drawing from previous work such as RAG and FiD. Finally, the paper compares the permutation bias between fusion-based seq2seq and decoder-only models.
strength_weakness: strength_weakness: Main:
- Novelty: The paper's novelty regarding the fusion-based approach is not too great. This seems more like an extension of the application of previous work, such as RAG and FiD.
- The authors stay in the abstract: "Our approach outperforms a decoder-only model that is six times larger and exhibits significant performance improvements compared to conventional seq2seq models." This statement is incorrect since you are not comparing it with llama2,3, recent decoder-only architectures that perform better than Bloom or OPT. Also, the subtitle in section 5.1 highlights this without further clarifying why not consider the most recent models.
- Table 1 is not very informative in terms of few-shot capabilities. Even though we see a significant performance improvement when the target input is in the encoder, I do not see any tendency for more samples to help performance. The authors boldly put these numbers, but this does not agree with the paper's story.

Minor:
- Table 1 should mention the dataset in which ablations are done. This would improve the readability of the paper.
- Figure 2 could be improved a lot. Adding the terms from the equation on page 5 will help readers understand easily and faster.
questions: questions: Questions:
- Is there any reason the authors did not include llama2,3 results in their comparison? I would like an explanation, if any since we have seen advancements in the performance of LLMs. Compare your numbers with  BLOOM; OPT does not make the paper's story look as good as it could be.
- According to table 1. Are seq2seq models few shot learners if the target input is placed in the encoder? A clarification of these experiments would be ideal in the paper. 
- What is the novelty presented in the paper? It seems that the most relevant results are those using fusion-based approaches (since these are the only ones where I could see a tendency of more shots helping performance). However, this is more like an extension of previous work.
- Could you share the dev standard numbers for Figure 3? I would like to see the deltas more clearly.
paper_summary: paper_summary: First, the paper conducted the first-ever extensive experiment comparing the in-context few-shot learning capabilities of decoder-only and encoder-decoder models on a broad range of tasks. Second, the paper proposes two methods to more effectively elicit in-context learning ability in seq2seq models: objective-aligned prompting and a fusion-based approach.
strength_weakness: strength_weakness: This paper is more like an internal technical report rather than a research paper since the scientific innovation of the proposed methods is very limited. The paper presents many factual results from experiments, but the discussion of these results is shallow.

The idea of early-fusion and late-fusion is quite straightforward and simple; however, the authors tried to make it look more complicated. The objective-aligned prompting is also quite straightforward."
questions: questions: Section 5.1 states that seq2seq models perform better than decoder-only models, which is a very big claim. Do you feel you have enough data points to validate this?
paper_summary: paper_summary: This paper studies the topic of few-shot in-context learning for
encoder-decoder language models (e.g., T5).  Its main contribution is empirical
results detailing the impacts of : 1.  different ways of formulating the
prompt, and 2. approaches for dealing with context limitations.

On the first topic, the paper investigates whether it is better to include the
target example in the encoder or decoder, as well as the impact of making the
prompt resemble the masked language modeling objective.  The results show
consistent benefit (across 4 models; T5, T5-LM, T0, and UL2) to placing the
target in the encoder, and using mask tokens in the prompt.  Interestingly, the
benefit of the mask tokens is present even for models that are subsequently
trained on maskless instructions (e.g., T0).

On the second topic, the paper investigates the impact of fusion-in-decoder-
and RAG-style approaches for encoding the in-context examples. These
approaches have the benefits of: 1. allowing the model to leverage more
examples than would effectively fit in its context window, and 2. permutation
invariance.  The results establish consistent benefit to both approaches over
naive concatenation of examples in the source prompt, with the
fusion-in-decoder approach slightly edging out the RAG approach.

Analyses are performed on benchmarks including SuperGLUE, HellaSwag, and XSum.
Models evaluated include:

- Encoder-decoders: T5, T5-lm, T0, and UL2
- LLMs: OPT, BLOOM, PALM and GPT3.

As stated in the previous paragraphs, the key results are that in-context
learning consistently works best when:

- The input example is included in the encoder,
- The prompt uses <mask> tokens to closer resemble the LM's pretraining
  objective, and
- RAG/Fusion-in-decoder are used to perform inference.
strength_weakness: strength_weakness: - I'm somewhat critical of the claim that the approaches explored are being
  "proposed", as opposed to preexisting. "Objective-aligned prompting" (aka
  putting mask tokens in the input) is the default approach used  in prior
  work on finetuning-based few-shot learning with encoder-only LMs [1], and has
  also been used in prior work on few-shot learning with encoder-decoder LMs
  [2]. Similarly the fusion-based approaches are just direct applications of
  fusion-in-decoder and RAG.
- In my opinion, the results are somewhat overstated. The paper uses the
  comparison of their best T5-11B-based approach to OPT-66B to justify the
  claim that "seq2seq models perform better than decoder-only models", however,
  the comparison to PALM 8B in Table 8B in the Appendix seems to indicate that
  there is no clearly better approach, and I suspect that this claim would be
  further undermined had the paper benchmarked against Llama or Mistral.
  Similarly, in the conclusion this paper refers to its results as "remarkable"
  when in my opinion a phrase like "encouraging" would be more appropriate.

[1] https://aclanthology.org/2021.eacl-main.20/
[2] Patel, et al., 2022
questions: questions: Questions:

In Section 5.4 you state:

> In the early-fusion method, there is a subtle variation in probability caused by the relative
positional bias when the order of the examples is fused.

could you provide a formal explanation of this?

Missing Citations:

- This is concurrent work but very similar: https://arxiv.org/abs/2308.07922
- I'd also recommend citing the Shick and Schutze paper above and discussing a bit the parallels to works on finetuning-based approaches for few-shot learning w/ encoder-only models from a few years ago.
paper_summary: paper_summary: This paper explores the potential of sequence-to-sequence (seq2seq) models for few-shot in-context learning. While few-shot in-context learning has proven effective in decoder-only neural language models, its application in seq2seq models remains underexplored. The authors conducted ablation studies using various prompting techniques, including sentinel tokens and mode tags, yielding several key findings:

- Incorporating the target input within the encoder significantly improves results, likely due to the bidirectional nature of seq2seq encoders.
- The paper proposes two fusion methods: a late-fusion reminiscent of Retrieval-Augmented Generation (RAG) and an early-fusion that concatenates encoder hidden states from previous prompts as input for the decoder. The early-fusion method consistently outperforms the late-fusion across various metrics. The fact that seq2seq decoders cherry-pick where to attend on the encoder side might explain why this is happening. Both fusion methods are shown to be more effective than the vanilla approach.
- Impressively, the few-shot seq2seq models demonstrated superior performance to decoder-only models six times their size, indicating a promising direction for future research.

Although the individual contributions may not appear novel when scrutinized in isolation, the collective advancements position this paper as a significant milestone. The clear writing, robust experimental setup, and promising results collectively suggest that seq2seq models deserve renewed attention in the wake of decoder-only models' popularity.
strength_weakness: strength_weakness: NA
questions: questions: I am really curious how your method would work on zero and few shot translation. Do you think there is a way to use your current setting to do that?

Comments: comment: In my opinion, the score from Reviewer TDzM is not well calibrated. There are issues with the novelty of the paper and some unsupported/unclear statements (e.g., seq2seq models perform better than decoder-only models). Also, one-line feedback and no objections do not seem like a proper review. For those reasons, I consider that the area chair should not weigh this score during the final evaluation unless they decide to provide a more in-depth review and update the score accordingly.
Thank you for responding to each of my questions. The paper addresses an important question about encoder-decoder architectures, and the execution is also good. However, as Reviewer KQR9 also mentioned, this paper looks more like a report and suffers from the novelty perspective. Considering all these aspects, this is a weak acceptance, in my opinion. I will increase my score to 6.
Thanks for your response
Thanks for the response. I will keep my score the same for this paper.
Thanks for the response and explanation. I maintain my generally positive opinion on this work.