FiD-ICL: A Fusion-in-Decoder Approach for Efficient In-Context Learning
Year: 2023

Abstract:
Large pre-trained models are capable of fewshot in-context learning (ICL), i.e., performing a new task by prepending a few demonstrations before the test input. However, the concatenated demonstrations are often excessively long and induce additional computation. Inspired by fusion-in-decoder (FiD) models which efficiently aggregate more passages and thus outperforms concatenation-based models in opendomain QA, we hypothesize that similar techniques can be applied to improve the efficiency and end-task performance of ICL. To verify this, we present a comprehensive study on applying three fusion methods—concatenationbased (early fusion), FiD (intermediate), and ensemble-based (late)—to ICL. We adopt a meta-learning setup where a model is first trained to perform ICL on a mixture of tasks using one selected fusion method, then evaluated on held-out tasks for ICL. Results on 11 heldout tasks show that FiD-ICL matches or outperforms the other two fusion methods. Additionally, we show that FiD-ICL (1) is 10x faster at inference time compared to concat-based and ensemble-based ICL, as we can easily precompute the representations of in-context examples and reuse them; (2) enables scaling up to meta-training 3B-sized models, which would fail for concat-based ICL.

heading: 1 Introduction
text: Large pre-trained models demonstrated remarkable performance in learning new language tasks via few-shot fine-tuning (FT)—initializing a model with pre-trained weights and optimizing it based on a few examples (Zhang et al., 2021). FT-based approaches currently achieve state-of-the-art performance (Liu et al., 2022), yet they require backpropagating and computing gradients over the full models, which can be prohibitive under memory and resource constraints.
†Work done while interning at Allen Institute for AI. 1Code: https://github.com/INK-USC/FiD-ICL
An alternative approach to few-shot learning is in-context learning (ICL). By concatenating a few examples and prepending them before the test instance, the model can perform a new task readily (Brown et al., 2020). ICL is more efficient at its “learning” stage, as it only uses one forwardpass and does not require gradients at all. Yet it is less efficient at inference stage, as the concatenated examples can become overly long and induce excessive computational costs. Additionally, ICL performance typically falls short of FT-based methods (Liu et al., 2022).
These limitations and trade-offs between ICL and FT motivate our exploration of methods that are efficient at both few-shot learning and inference time. In particular, we aim to achieve this by exploring different methods that incorporate (or “fuse”) the in-context examples during inference. We draw connections between open-domain QA (Chen and Yih, 2020) and ICL, since both problems task a model with reading long context (multiple retrieved
8158
passages vs. multiple examples) and making a prediction based on the context (answer a relevant question vs. infer about a new input). Further, we draw inspiration from fusion-in-decoder (FiD; Izacard and Grave 2021), a method that can efficiently “aggregate evidence” from many retrieved passages to answer open-domain questions. Given that FiD models significantly outperforms concatenationbased methods for open-domain QA, we hypothesize that FiD can be applied to ICL analogously to improve its efficiency and end-task performance.
To verify this, we present a systematic comparison of three different methods to incorporate incontext examples: concatenation-based, fusion-indecoder, and ensemble-based (Fig. 1). We term them as “fusion” methods, and characterize them as early, intermediate, and late fusion, based on their formulation. We conduct comprehensive experiments with the P3 dataset (Sanh et al., 2022) in a meta-learning setting similar to Min et al. (2022b)– i.e., a model is first trained to perform ICL on a mixture of tasks using one selected fusion method, then evaluated on held-out tasks for ICL.
Our empirical results suggest that, while being significantly more efficient on computation complexity and memory usage, FiD-ICL is comparable to or outperforms the other two fusion methods on the 11 P3 held-out tasks. This observation is consistent across three different model sizes. Notably, given the memory efficiency of FiD-ICL, we are able to meta-train 3B-sized ICL models within an academic budget, which would lead to out-ofmemory errors and fail in the case of concatenationbased ICL. Our best model, FiD-ICL trained from T5-LM-XL (3B), narrows the gap with T-Few (Liu et al., 2022)–a state-of-the-art few-shot FT method– to 3% difference in accuracy.
Moreover, our formulation of FiD-ICL decouples the computation of few-shot examples and the test input, allowing the computation over the fewshot examples to be pre-computed and reused. As a result, FiD-ICL is up to 10x faster at inference time compared to the other two fusion methods. We further support this argument with computation cost analysis and inference speed tests.
However, it is still questionable whether the ICL models we investigate learn effectively from the few-shot examples. We replicate a set of diagnostic experiments in Min et al. (2022c) by perturbing the in-context examples (e.g., using fewer/more shots, replacing correct labels with random labels). We
found that ICL methods, regardless of what fusion method is employed, are still rather insensitive to these perturbations, and do not rely on input-label mapping as much as expected. These observations call for further investigation and efforts to improve the effectiveness of in-context learning.

heading: 2 Related Work
text: Few-shot Fine-tuning. It has been shown that fine-tuning a large pre-trained model with only a few examples yields strong performance on a wide range of NLP tasks (Zhang et al., 2021). The performance can be further improved by incorporating prompts and demonstrations in the input (Schick and Schütze, 2021; Gao et al., 2021). Moreover, parameter-efficient fine-tuning methods can be applied to improve memory and storage costs (Liu et al., 2022). However, these methods are still relatively expensive at training time as they require back-propagating through the full model.
In-Context Learning. In-context learning (ICL) is an alternative approach for few-shot learning by simply concatenating the few-shot examples and using them as a prompt before the actual inference example. Very large pre-trained models, such as GPT-3 (Brown et al., 2020) and PaLM (Chowdhery et al., 2022), are capable of ICL off the shelf (i.e., without any gradient update) and achieve competitive performance. Smaller models can be metatrained to obtain this capability (Chen et al., 2022; Min et al., 2022b). We follow the latter problem setting and focus on smaller models (up to 3B size). One disadvantage of ICL is that the inference cost grows rapidly as the number of few-shot examples increases. Researchers also find that ICL models do not rely on input-label mapping as much as expected, casting doubts on the effectiveness of ICL (Min et al., 2022c).
Zero-shot/Few-shot Task Generalization. Towards the goal of building a generalist NLP system, recent works adopt a meta-learning paradigm (Schmidhuber, 1987) and propose to meta-train a model on a set of given tasks (i.e., meta-train set). The resulting model is expected to solve novel tasks (in a meta-test set) in a zero-shot or few-shot setting. This is made possible by unifying task format with prompts (Zhong et al., 2021; Sanh et al., 2022; Wei et al., 2022), providing task instructions/descriptions (Weller et al., 2020; Mishra et al., 2022; Wang et al., 2022b), and scaling up
and diversifying the meta-train set (Ye et al., 2021; Chung et al., 2022). The meta-training set is typically utilized with multi-task learning (Caruana, 1997) or model-agnostic meta-learning (Finn et al., 2017). In this work, we compare different fusion methods in such meta-learning setting. In particular, we focus on applying fusion-in-decoder technique to ICL and investigate the benefits and limitations of it.

heading: 3 Investigating Fusion Methods for ICL

heading: 3.1 Problem Setting
text: Overview. Our goal is to build models that are capable of few-shot in-context learning without gradient updates when handling an unseen task. Prior work show that such capabilities can be obtained by learning from a collection of seen tasks and training the model on concatenation-based in-context learning (Chen et al., 2022; Min et al., 2022b). In this work, we examine alternative ways to synthesize and incorporate information in multiple shots (i.e., “fusion method”).
Data. We use three non-overlapping sets of tasks, meta-train (Ttrain), meta-valid (Tvalid), and metatest (Ttest). We assume all tasks are in text-to-text format. Each task T in Ttrain contains a set of training examples, i.e., T = {(x, y)}. Tasks in Tvalid and Ttest are few-shot. Each task T in Tvalid or Ttest contains a support set and a query set.2 Models in this study are expected to learn from the k-shot support set {(x(s)i , y(s)i )} without gradient updates, and do inference on the query set{(x(q)i , y(q)i )}. Additionally, we assume all tasks in Tvalid and Ttest can be evaluated with rank classification, with a set of choices C given for each query example (x(q), y(q)). In this case, the model does inference by ranking the probabilities assigned to each choice c ∈ C.
Meta-Training and Inference Procedure. We closely follow the procedure described in MetaICL (Min et al., 2022b). In the meta-training phase, we first sample one task T from Ttrain, then sample k support examples {(x(s)i , y(s)i )} and m query examples {(x(q)i , y(q)i )} from the task. We update the model (using a selected fusion method) to minimize the loss of generating the correct target sequences
2Support/query set are the same as few-shot train/test set. We adopt these terms to distinguish from meta-train/meta-test.
y (q) i . In the meta-test/inference phase, for each unseen task in Ttest, we are given a fixed set of k-shot support examples, and the model is expected to do inference on all query examples {(x(q)i , y(q)i )}.

heading: 3.2 Fusion Methods
text: Previously in Fig. 1 we provide the visualizations of fusion methods that we compare. In this section, we reinstate our motivations and describe them more formally.
Overview. Fusion-in-decoder (Izacard and Grave, 2021) is a competitive method for incorporating multiple retrieved documents for open-domain QA, and it significantly outperforms concatenation-based methods (Lewis et al., 2020b). Bringing these insights to few-shot learning, in-context learning can be viewed as concatenating the raw text of few-shot examples and doing “early fusion”. We investigate whether doing fusion at later stages, such as fusion-in-decoder (i.e., “intermediate fusion”) or ensemble (i.e., “late fusion”) will bring additional benefits.3

heading: Early Fusion: Concatenation-based ICL.
text: This refers to the method of concatenating (x(s)1 , y(s)1 , ..., x(s)k , y(s)k , x(q)) into a long text input and feeding this sequence to a model. The model is expected to generate y(q). Specifically, we compute argmaxc∈C P (c∣x(s)1 , y(s)1 , ..., x(s)k , y(s)k , x(q)). Note that in transformer models the computation cost typically grow quadratically with sequence length (and thus the number of shots).

heading: Intermediate Fusion: Fusion-in-decoder (FiD).
text: In fusion-in-decoder, the support examples(x(s)1 , y(s)1 ), ..., (x(s)k , y(s)k ) and the query x(q) are encoded separately by the same encoder layers in the transformer model. The representations produced by the last encoder layer are then concatenated (i.e., “fused”) and sent to the decoder layers. In this way, the computation cost grows linearly with the number of shots.
Note that our formulation is slightly different from the original fusion-in-decoder models for open-domain QA (ODQA). In ODQA, the question (x(q)) is first concatenated with each retrieved paragraph (x(s)i ) and then encoded separately by
3The terms of early/intermediate/late fusion are inspired by multi-modal literature, but their meanings are slightly different in this work.
the model. For ICL, we decouple the computation of support examples and the query example. In this way, the support examples can be encoded only once and re-used throughout the inference phase. See §5.2 for discussion.
Late Fusion: Ensemble-based ICL. Early fusion and intermediate fusion naturally bring us to the idea of ensemble-based approaches, which are effectively doing “late fusion”. They are previously explored in Min et al. (2022a) for classification tasks and demonstrate competitive performance. We implement this by training one-shot concat-based ICL models and aggregating the k different predictions at inference time. More specifically, we compute argmaxc∈C ∑ki=1 P (c∣x(s)i , y(s)i , x(q)). Theoretically, the cost of ensemble-based ICL grows linearly with the number of shots.
Other Variants. Adapting FiD in open-domain QA for ICL is non-trivial. In the early stages of this work, we also examined two more variants named as FiD-Pairwise and FiD+Ensemble. FiD-Pairwise is closer to the original FiD implementation for open-domain QA. FiD+Ensemble a hybrid method that combines the techniques in FiD and Ensemble. Details are elaborated in Fig 6 and §A.1. The fusion-in-decoder design illustrated in Fig. 1 is the best one in our preliminary study, and therefore we adopt it in the main experiments.

heading: 4 Experiment Settings

heading: 4.1 Data
text: We use Public Pool of Prompts (P3) dataset (Sanh et al., 2022). The dataset includes a collection of diverse NLP tasks with crowd-sourced prompt templates. The tasks are partitioned into a Meta-Train set and a Meta-Test set. In the main experiments
we use all 11 tasks in the meta-test set (Meta-Test11). For analysis experiments, we use a subset of 7 tasks for faster experimentation (Meta-Test-7). We use 16 shots for all few-shot experiments, unless specified otherwise. Additionally, we use 14 BIG-bench tasks (Srivastava et al., 2023) as a MetaValidation set for selecting the best checkpoint. We provide the full list of datasets and more details in Table 4 and §B.

heading: 4.2 Model
text: We limit our scope to encoder-decoder models for our experiments.4 We use T5-LM-Adapt models5 and T0 models (Sanh et al., 2022) as initializations in our experiments. The two model groups have the same model architecture but different weights; T0 is trained to multi-task on the P3 meta-train set using T5-LM-Adapt as initialization. We experiment with models of three different sizes: Base (250M), Large (800M), XL (3B).6

heading: 4.3 Compared Methods
text: The goal of using few-shot ICL methods is to learn from the few-shot examples so that it improves on top of zero-shot performance; further, we aim to close its gap to few-shot fine-tuning, which requires gradient updates. To quantify these, we include zero-shot inference and few-shot fine-tuning in our experiments, in addition to the three fusion methods that we compare. We provide an overview of the training and evaluation procedure of these methods in Table 1.
Zero-shot. We directly evaluate T5-LM-Adapt and T0 models on the Meta-Test, in the zero-shot setting.
Few-shot ICL. We initialize from either T5-LMAdapt or T0, meta-train it with the three fusion methods (concatenation, fusion-in-decoder, ensemble) described in §3.2. We evaluate all saved checkpoints on Meta-Validation, then evaluate the one selected checkpoint on Meta-Test. Unless specified otherwise, we use 16 shots during training and evaluation.
4We elaborate our discussion on encoder-decoder vs. decoder-only models in §A.4.
5 https://huggingface.co/google/t5-xl-lm-adapt
6We replicate the experiment setting in Sanh et al. (2022) and trained our own T0-Base/Large/3B model for this work. Notably, our reproduction of T0-3B outperforms the public checkpoint by a large margin, suggesting that the public T03B checkpoint may be undertrained. See §C.1 for details on training these models.
Few-shot Fine-tuning. For each meta-test task, we fine-tune either T5-LM-Adapt or T0 with the few-shot examples. Apart from simple fine-tuning, we also experiment with the T-Few fine-tuning recipe (Liu et al., 2022), which updates only a small portion of parameters, and includes an unlikelihood loss and a length normalization loss during fine-tuning.

heading: 4.4 Reporting the Results
text: For a high-level comparison across different methods, we report Meta-Test-11 average accuracy. Note that this one number is taking average on three levels: (1) averaging over the 11 held-out tasks; (2) for each task, averaging over all prompts associated with the task (in P3 dataset, each task is accompanied with multiple prompts); (3) for each (task, prompt) pair, averaging over 5 different samples of few-shot examples, to mitigate the influence brought by a specific set of few-shot examples.
We also report detailed per-task performance of Meta-Test-11 for more fine-grained analysis.

heading: 5 Experiment Results

heading: 5.1 Performance on Held-out Tasks
text: Following our experiment settings, we present the results of all compared methods in Fig. 2. We have the following observations. Firstly, we highlight that the efficient design of FiD-ICL and EnsembleICL enables us to train them in a larger scale (e.g., 3B models) on an academic budget. We fail to do so for Concat-ICL as training with a max sequence length of 4096 results in out-of-memory errors.7
74096 tokens = 16 examples × 256 tokens/example. We are able to train 3B models when reducing k to 4, and we include the result in Table 7 for completeness. Our conclusion
Secondly, when comparing the three fusion methods, FiD is comparable or outperforms the other two fusion methods in all three model sizes. Izacard and Grave (2021) attribute the success of FiD to “scaling to large number of contexts” in the encoder and “better aggregating evidence from multiple passages” in the decoder. We conjecture that the same inductive biases are also beneficial for few-shot ICL. Thirdly, our best FiD-ICL model (trained from T5-LM XL) achieves an average accuracy of 60.0% on Meta-Test-11. As a gradient-free method, this leaves a 1.4% gap compared to simple finetuning, and a 3.0% gap to T-Few fine-tuning (using T0-3B). This demonstrates the great potential of gradient-free ICL methods, and we hope future work can further improve ICL to close the gap.

heading: 5.2 Efficiency
text: One major motivation of our work is efficiency—to find a few-shot learning method that is efficient at both few-shot learning and inference time. In this section we estimate and compare the computational cost and inference speed of all methods.
Computation Complexity. Our estimation is based on the following assumptions: (1) the output length lout is much smaller than the input length lin, i.e., lout ≪ lin, so that the cost for an input (lin) and a complete in-context example (lin + lout) are roughly comparable, i.e., lout + lin ≈ lin; (2) training (forward and backward pass) requires 3 times the cost of inference (forward pass) (Liu et al., 2022). We use M1,M2,M3 to represent the baseline cost for one forward pass using a zeroshot model over one example in the encoder selfattention layers, decoder cross-attention layers and
on performance and efficiency remains the same.
decoder self-attention layers, respectively. Computation costs of other methods will be represented in multipliers of M1,M2,M3.
We summarize our estimation in the top section of Table 2. (1) We use “pre-inference cost” in the table to represent the one-time costs. For FiD-ICL, this refers to the cost of pre-computing the representations of examples using the encoder. For few-shot FT, this refers to the cost of applying gradient-based optimization. FiD-ICL has a significantly smaller pre-inference cost compared to few-shot FT. (2) In terms of inference cost, FiDICL is more efficient than the other two fusion methods in all the layers that we list. It uses kM2 more computation in the decoder cross attention layers compared to a zero-shot or fine-tuned model.
Inference Speed. Additionally, we select two tasks (RTE and StoryCloze) in the meta-test set and measure the run time. For few-shot FT, we optimize the model for 300 updates, which is the recommended value in T-Few (Liu et al., 2022). In Table 2, we show that FiD-ICL is up to 10x faster than the other two fusion methods. Moreover, FiDICL, while achieving competitive performance, is faster than few-shot FT when pre-inference and inference time are combined.
Note that inference speed comparison above is dependent on number of test examples. In practice, when the test set is larger, the pre-inference cost will be amortized and FT will become faster when
pre-inference cost and inference cost are summed. The break-even point for FiD-ICL and FT appears at 3.4k test instances for RTE and 5.6k test instances for Story Cloze. Therefore we believe FiDICL is most useful when the test set is small or when fast prototyping is needed.

heading: 6 Analysis
text: In this section we evaluate our ICL models in various scenarios, in hope to better understand their behavior and limitations. In §6.1 we evaluate the models to perform ICL with varying number of shots, when they were originally meta-trained to do 16-shot ICL. In §6.2 we study the influence to performance when the in-context examples are perturbed. In §6.3 we try to understand where ICL methods lie among other recent advances by comparing the performance of different model families.

heading: 6.1 Evaluate with Varying Number of Shots
text: Performance. One advantage of fusion-indecoder models is that they may be trained with a small number of passages (e.g., 5 passages), but evaluated with a larger number of passages (e.g., 100 passages) (Izacard and Grave, 2021). For fewshot learning, this enables flexibility in the number of shots used. We conduct a similar analysis by changing the number of shots available at meta-test time. All our models are originally meta-trained to perform 16-shot in-context learning, and here we evaluate them with {2, 4, 8, 16, 32} shots. Results
are visualized in Fig. 3 and reported in Table 9.
While the performance of fine-tuning method consistently increases when more shots become available, the performance of in-context learning methods is less sensitive to the number of shots. We further look at per-task performance and find two distinctive patterns: (1) On COPA and WSC, the performance gradually improves with more shots, as expected. Interestingly, FiD-ICL outperforms simple fine-tuning on WSC, suggesting that FiDICL is somehow “good at” learning WSC in particular. (2) On NLI tasks such as RTE and CB, performance surprisingly drops with more shots. These two patterns together lead to the unchanging performance on average (in Fig. 3 Left).
These observations suggest that ICL may be more suitable to certain task types than others. This may be relevant to the intrinsic task hardness (Zhao et al., 2022) or the difference between inductive biases exhibited by ICL and FT methods (Chan et al., 2022). One relevant observation is that on RTE, GPT-3 few-shot performance is not always better than zero-shot or one-shot performance (Brown et al. 2020, Appendix H), suggesting that RTE may have some unique characteristics. We leave further investigation as future work.
Inference Speed. Previously in §5.2, our run time analysis has been limited to the case of k = 16. As shown in the Complexity Analysis section in Table 2, efficiency is dependent on the number of in context example k, and the efficiency benefit of FiD is more significant with larger k. To provide a full picture of the efficiency benefits of FiD-ICL with smaller k, we report the run time when k = 2, 4, 8 in Table 3. We observe that FiD-ICL is constantly faster than Concat-ICL and Ensemble-ICL.

heading: 6.2 Perturbation to In-Context Examples
text: Min et al. (2022c) show that ICL models are rather insensitive to perturbations in in-context examples. Even with 100% wrong labels, little performance drop is observed with ICL models.8 This is unexpected as the performance of fine-tuning would be drastically worse when labels are incorrect.
To investigate whether the fusion methods we use in this work help resolve these issues, we conduct a similar ablation study. We compare the performance of the following: (1) No Perturbation; (2) No Input, remove the inputs but keep the la-
8A more recent work (Wei et al., 2023) suggest that extremely large LMs can override semantic priors when these perturbations are applied.
bels; (3) Random Label, randomly select one of the valid options as the output; (4) Wrong Label, randomly select one of the wrong options; (5) No Label, remove the labels but keep the inputs. We examine both large-size (800M) and XL-size (3B) models, selecting the better model between T5-LM and T0 initialization.9 We visualize the results in Fig. 4.
As expected, we observe a clear trend of No Perturbation > Random Label > Wrong Label for the T0-FT method. For ICL methods, performance drops in most cases when No Label perturbation is applied, suggesting that the presence of labels is essential. FiD-ICL suffers from No Label perturbation more than other two methods, suggesting that it may be capturing more information from the labels. However, performance does not change significantly with Random Label or Wrong Label perturbation, suggesting that FiD-ICL also struggle to learn from input-label mapping, despite their improved performance over Concat-ICL. Enabling ICL models to learn effectively and faithfully from examples remains a challenging problem.

heading: 6.3 Comparing with Other Model Families
text: Previously, we limit our scope to encoder-decoder models meta-trained to perform in-context learning. It is also necessary to have contextualized understanding by referencing and comparing with performance of other model families. We plot performance of various models in Fig. 5 and Fig. 7. We hope this can explain where FiD-ICL lies among other recent advances, and partly disentangle factors such as model architecture, training procedure.
Meta-trained vs. Not Meta-trained. In Fig. 5(a) we show the performance of T5-LM models that do not go through any meta-training. We show that in our problem setting, meta-training is crucial for the model to acquire the capability of zeroshot learning or few-shot in-context learning. Surprisingly, T5-LM models demonstrate little zeroshot or few-shot in-context learning capabilities on our meta-test tasks. We further try to quantify the effect of model architecture (encoder-decoder vs. decoder-only) and prompts used (P3 prompts or GPT-3 prompts), which we visualize in Fig. 7 and discuss in §A.2.
9Note that all four perturbations can be applied to ICL models, but only (3)(4) can be applied to FT-based method.
vs. GPT-3 models. We quote the GPT-3 performance (Brown et al., 2020) on Meta-Test-7 tasks in Table 13 and visualize them in Fig. 5(b). Note that the performance is not directly comparable as the number of shots vary from 20 to 70 for GPT-3 models, while our experiments are using 16 shots.10 We would like to highlight that meta-trained encoderdecoder models outperforms off-the-shelf decoder models by a large margin, which aligns with the findings in Sanh et al. (2022); Wang et al. (2022a). Further, few-shot ICL models improves on top of zero-shot methods.

heading: 7 Conclusion
text: Motivated by the train-test efficiency differences between few-shot in-context learning and few-shot fine-tuning, we aim to find a balance and benefit from the strengths of both approaches. Towards this goal, we introduce FiD-ICL, a fusionin-decoder approach for ICL, inspired by fusionin-decoder models for open-domain QA (Izacard and Grave, 2021). With extensive experiments, we show that fusion-in-decoder ICL (intermediate fusion) is more favorable compared to concatenation-
10This comparison is less fair due to differences in model architecture, pre-training procedure, and prompts used. Yet, we think GPT-3 performance provide a reasonable reference.
based ICL (early fusion) and ensemble-based ICL (late fusion), in terms of both performance and computation efficiency. Moreover, fusion-in-decoder ICL partly closes the gap between gradient-free ICL methods and gradient-based fine-tuning methods, highlighting the potential of approximating gradient-based optimization with efficient forwardonly methods (Phang et al., 2022). Future work may build upon our insights to further improve the computation efficiency of few-shot learning. However, similar to the findings in Min et al. (2022c), our analysis on ICL models suggest that they barely learn the input-label mapping from the in-context examples. We also have mixed results when more shots become available for the ICL model. We hope future work can further improve the performance of ICL by enabling it to learn from input-label mapping effectively and faithfully.

heading: Limitations
text: Firstly, following the work of T0 (Sanh et al., 2022), we mainly focus on NLP tasks that can be formulated as rank classification. This covers classification and multiple-choice tasks, but not other task categories such as generation or regression. We hope to extend our training and evaluation to encompass a wider range of task categories, and hope the research community will collaborate in creating resources for such study.
Secondly, though we showed that FiD-ICL outperforms Concat-ICL, we still lack clear understanding on the source of such improvement. We hypothesized that FiD enables the model to learn from in-context examples more effectively, yet our perturbation experiments show that FiD-ICL mod-
els still learn little from input-label mapping (§6.2). Much more work is needed to further understand of the working mechanism of ICL models.
Thirdly, given the complexity of our study, we limit the scope to encoder-decoder models. We made this decision due to the superior performance of encoder-decoder models in task-level generalization (Wang et al., 2022a) and their compatibility with fusion-in-decoder method. Also, our important baselines, T0 (Sanh et al., 2022) and T-Few (Liu et al., 2022), are implemented with the T5 model family. We include more discussion in §A.4.

heading: Acknowledgements
text: We thank Hamish Ivison, Zhaofeng Wu, Ian Magnusson, Dirk Groeneveld for insightful discussions. In addition, we thank anonymous reviewers, members of AllenNLP, USC INK Lab, the USC NLP community for their valuable feedback.

heading: A Additional Experiments and Findings

heading: A.1 Alternative Fusion Methods
text: In addition to the three methods we investigate in the main paper, we experimented with two other methods, which we refer to as FiD-Pairwise and FiD+Ensemble. We illustrate these two methods in Fig. 6. FiD-Pairwise is closer to the original fusionin-decoder model for open-domain QA. In FiDPairwise, the input is appended to each example individually. FiD+Ensemble is a hybrid model that employs the “fusion” operations in both FiD and Ensemble-based ICL. It applies fusion-in-decoder with one example and the input. Then it aggregates the predictions similar to ensemble-based ICL.
We conduct experiments with these two methods with base-size model (250M). We report the performance in Table 8. FiD (with T0 initialization) remains the best method among all compared fusion methods. Given the fact that FiD-Pairwise and FiD+Ensemble are less efficient than the FiD we use in the main paper, we stop investigating them in larger model scales.
A.2 Influence of Using P3 Prompted Data
In Fig. 7(a) we report performance on P3 held-out tasks using P3 prompted data and T5-LM models. Surprisingly we the performance does not grow with model scale. Also it does not grow when more shots become available. For all models the performance is close to majority/random baseline. We were skeptical about these results so we further evaluate public GPT models with the same P3 prompted data, which is reported in Fig. 7(b). The public GPT models include GPT-2 of various sizes, GPT-Neo-2.7B and GPT-J-6B. We still observe similar trends as in Fig. 7(a).
In Fig. 7(b), the main differences between the two groups were the prompt templates used. We
hypothesize that P3 prompts may appear unnatural for GPT models and thus leads to the near random performance. We did some initial experiments with the COPA dataset using GPT-J-6B model, where using GPT-3 prompts yields zero-shot accuracy of 81%, but using P3 prompts gives accuracy ranging from 47% to 54%. We hope future work conducts rigorous comparisons about this.

heading: A.3 T5-LM or T0 as initialization?
text: In Fig. 2, we observe that for large/XL models, T5-LM is a better initialization than T0 for ICL meta-training. Our hypothesis is that T0 training (i.e., training T5-LM to become T0) may cause the model to forget general pre-train knowledge or lose the capabilities in modeling long context. This results in meta-training ICL being less effective. However this observation is dependant on model size. For base-size models, T0 is a better initialization.

heading: A.4 Discussion on Enc-Dec vs. Dec-only Models
text: Prior work suggest that in a similar meta-learning setting, enc-dec models outperform dec-only models (Wang et al. 2022a, Sec 4.2). Another supporting evidence is that a 3B FLAN-T5 (enc-dec) model outperforms 175B OPT-IML (dec-only) on few-shot in-context learning (Longpre et al. 2023, Figure 1). Given the competitive performance of enc-dec models on the problem of interest, we focus on enc-dec models in this work.
To give our best efforts in make fair and comprehensive comparisons between enc-dec and deconly models, we evaluated our meta-test tasks with publicly-available GPT models (Fig. 7, Table 12),
quoted the performance on these tasks using GPT-3 models (Fig. 5, Table 13), and discussed our findings in §A.2.
One may argue that the computation for ICL using decoder-only models is also partly cache-able and thus the conclusions on computation efficiency may vary when decoder-only models are used. We agree with this argument. To account for this, we experimented with a key-value caching mechanism for a dec-only Concat-ICL model.11 Caching enables a 3-5x speed-up, demonstrating that this is a promising direction. We hope a rigorous comparison between enc-dec and dec-only models regarding this matter can be done in future work.

heading: B Data

heading: B.1 Meta-Train and Meta-Test
text: We use the data and prompts provided in P3 (Sanh et al., 2022). We use the meta-train and meta-test partition used to train T0 (as opposed to the ones
11We used the catwalk library: https://github.com/a llenai/catwalk/tree/prefix-caching
use for training T0+ or T0++). We provide the full list of these datasets and their reference in Table 4.
For meta-train, we use all the prompts associated with meta-train tasks. For meta-test, Sanh et al. (2022) provide the list of prompts for evaluation.12 We want to point out a caveat here that this list is only a portion of all prompts associated with meta-test tasks. For example, hellaswag_Topic_of_the_context is a prompt name associated with the HellaSwag dataset, but it is not relevant to the original HellaSwag task, and should not used in evaluation.
We consider ANLI R1/R2/R3 as three separate tasks. Therefore the original meta-test in P3 has 11 tasks (Meta-Test-11).

heading: B.2 Meta-Validation
text: In our preliminary experiments we observe that ICL methods may suffer from meta-overfitting: metatest performance drops when the model is trained for more steps on meta-train. To ensure a fair eval-
12 https://github.com/bigscience-workshop/t-zer
o/blob/master/evaluation/template_list.py
uation set up, we additionally use 14 BIG-bench Task used in Sanh et al. (2022) as meta-validation (listed in Table 4). We use this meta-validation set for selecting intermediate checkpoints saved during meta-training. Apart from this, we do not tune any other hyper-parameters.

heading: B.3 Few-shot Sampling
text: The performance of few-shot learning is highly subject to the sample of few-shot examples. To mitigate its influence in evaluation, our evaluations are based on 5 different few-shot samples. We first obtain the 5 samples of T0 held-out tasks used in Liu et al. (2022).13 We then further sub-sample 16 examples to be our few-shot support set. We report the average of the 5 samples for all few-shot methods.

heading: B.4 Data Sources
text: We obtain all our data from huggingface datasets (Lhoest et al., 2021). In the following we provide the links:
• P3 (meta-train/meta-test): https://huggin gface.co/datasets/bigscience/P3
• BIG Bench (meta-validation): https://hugg ingface.co/datasets/bigbench
The full list of datasets and their citations are in Table 4.

heading: C Training Details

heading: C.1 Training T0-Base/Large/3B
text: Sanh et al. (2022) only provide model checkpoints in sizes of 3B and 11B. For a thorough investigation of different fusion methods, we aim to conduct experiments across different model sizes. Therefore, we replicate training procedure of T0-3B/T0-11B and train our own T0 models. We also largely reference the practice in Lin et al. (2022), in which the authors trains a BART0 model using BART-Large (Lewis et al., 2020a).
Specifically, we sub-sample at most 50k examples for each prompted task, following Lin et al. (2022). We combine all examples as a large dataset for multi-task learning, and do not apply any task sampling re-weighting technique. We list the key hyper-parameters in Table 5.
Sanh et al. (2022) reported an average of 51.0 on P3 held-out tasks. Our re-evaluation of the public
13 https://github.com/r-three/t-few
checkpoint yields the same value of 51.0. Our T0Base achieves 49.1, our T0-Large achieves 52.4, and our replication of T0-3B achieves 57.6. We hypothesize that the publicly released T0-3B may be under-trained, which corroborates with the findings in Lin et al. (2022) and Ivison et al. (2022).
C.2 ICL methods
Hyperparameters are listed in Table 6. Gradient checkpointing is enabled when training ConcatICL-Large, FiD-ICL-3B and Ensemble-ICL-3B models.

heading: C.3 Few-shot Fine-tuning
text: Hyperparameters are listed in Table 5.
C.4 Implementation
Our implementations are based on huggingface transformers (Wolf et al., 2020).

heading: D Extended Related Work

heading: Sparse Attention for In-Context Learning.
text: Concurrent to our work, Ratner et al. (2022) proposed parallel context window (PCW) and Hao et al. (2022) proposed structured prompting for incontext learning. In a broader sense, these two works and our FiD-ICL can be viewed as applying sparse attention mask to the in-context examples. Ratner et al. (2022) and Hao et al. (2022) mainly focus on (1) applying such sparse masks to off-theshelf decoder-only models and (2) incorporating more in-context examples than what one context window can typically fit. Our work differs in (1) problem settings, as we mainly compare different fusion methods in a meta-learning setting; (2) experiment settings, as we fix the number of shots available, and investigate the performance and efficiency of the models. Despite these differences, the shared intuitions and findings invite future research in adopting efficient architectures for improving different aspects of ICL.
Fusion-in-decoder and Hypernetworks. In recent years, hypernetworks (Ha et al., 2017) are explored for various NLP problems (Ivison and Peters, 2022; Karimi Mahabadi et al., 2021), including zero-shot and few-shot task generalization (Ye and Ren, 2021; Phang et al., 2022). We believe the encoder in our fusion-in-decoder approach can be viewed as a hypernetwork. The encoder is effectively generating prefix parameters for the decoder, as demonstrated in Fig. 9. In Table 7 we compare with HyperT5 (Phang et al., 2022), a concurrent work that trains a hypernetwork to produce adaptation parameters.14 Our fusion-in-decoder ICL is comparable with HyperT5.
Related to the concept of hypernetworks, recent work also suggest that in-context learning can be viewed as applying implicit optimization to the model itself (Akyürek et al., 2023; von Oswald et al., 2022; Dai et al., 2022).

heading: E Extended Results
text: • Table 7 reports the per-task performance and average accuracy reported in Fig. 2.
• Table 9 includes the numbers in Fig. 3.
• Table 10 and Table 11 includes the numbers in Fig. 4.
• Table 13 includes the GPT-3 results quoted from the original paper (Brown et al., 2020). They were visualized in Fig. 5 and Fig. 7.
• Table 12 includes the numbers of our evaluation with GPT-style models. They were visualized in in Fig. 7.
• Table 14 includes the performance of notmeta-trained encoder-decoder models, also visualized in Fig. 5.
14Though the performance is not directly comparable (e.g., the in-context examples used are different), we believe they provide reasonable references.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitation section after conclusion.

heading: 3 A2. Did you discuss any potential risks of your work?
text: Limitation section after conclusion.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts?

heading: Left blank.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4, Appendix B
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix B
C 3 Did you run computational experiments?

heading: Left blank.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Table 3 and Table 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Table 3 and Table 4

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4.4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix C
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants?

heading: Left blank.
text: D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Despite these differences, the shared intuitions and findings invite future research in adopting efficient architectures for improving different aspects of ICL. Fusion-in-decoder and Hypernetworks. In recent years, hypernetworks (Ha et al. , 2017) are explored for various NLP problems (Ivison and Peters, 2022; Karimi Mahabadi et al. , 2021), including zero-shot and few-shot task generalization (Ye and Ren, 2021; Phang et al. , 2022). We believe the encoder in our fusion-in-decoder approach can be viewed as a hypernetwork. The encoder is effectively generating prefix parameters for the decoder, as demonstrated in Fig. 9. In Table 7 we compare with HyperT5 (Phang et al. , 2022), a concurrent work that trains a hypernetwork to produce adaptation parameters. 14 Our fusion-in-decoder ICL is comparable with HyperT5. Related to the concept of hypernetworks, recent work also suggest that in-context learning can be viewed as applying implicit optimization to the model itself (Akyürek et al. , 2023; von Oswald et al. , 2022; Dai et al. , 2022)