Zemi: Learning Zero-Shot Semi-Parametric Language Models from Multiple Tasks
Year: 2023

Abstract:
Although large language models have exhibited impressive zero-shot ability, the huge model size generally incurs high cost. Recently, semiparametric language models, which augment a smaller language model with retrieved related background knowledge, alleviate the need for storing everything into the model parameters. Although existing semi-parametric language models have demonstrated promising language modeling capabilities, it remains unclear whether they can exhibit competitive zeroshot abilities as their fully-parametric counterparts. In this work, we introduce Zemi, a semi-parametric language model for zero-shot task generalization. To our best knowledge, this is the first semi-parametric language model that can demonstrate strong zeroshot performance on a wide range of heldout unseen tasks. We train Zemi with semiparametric multitask training, which shows significant improvement compared with the parametric multitask training as proposed by T0 (Sanh et al., 2021). Specifically, during both training and inference, Zemi is equipped with a retrieval system based on the unlabeled pretraining corpus of our backbone model. To address the unique challenges from large-scale retrieval, we further propose a novel retrieval-augmentation fusion module that can effectively incorporate noisy retrieved documents. Finally, we show detailed analysis and ablation studies on the key ingredients towards building effective zero-shot semiparametric language models. Notably, our proposed ZemiLARGE model outperforms T0-3B by 16% across seven diverse evaluation tasks while being 3.8x smaller in scale.1

heading: 1 Introduction
text: Achieving strong generalization ability on unseen tasks while maintaining a reasonably small param-
∗Work was done when interning at Tencent AI Lab. 1Code and data are available for research purpose at https:
//github.com/MikeWangWZHL/Zemi.
eter size is a long-lasting challenge for natural language processing (NLP) models. Although large language models (Brown et al., 2020; Lieber et al., 2021; Rae et al., 2021; Smith et al., 2022; Hoffmann et al., 2022; Zhang et al., 2022; Ouyang et al., 2022; Chowdhery et al., 2022) have shown impressive zero-shot ability on various NLP tasks, the huge model size generally incurs high cost. Alternatively, instead of stuffing everything in the model parameters, recent work on semi-parametric language models (Grave et al., 2016; Khandelwal et al., 2019; Yogatama et al., 2021; Borgeaud et al., 2021; Zhong et al., 2022) demonstrated competitive language modeling performance compared with much larger fully-parametric language models. The intuition is to use a relatively small language model as a reasoning module and augment it with a retriever to retrieve related background knowledge, which effectively alleviates the need for increasing the model capacity to align with the growing data size.
However, what really makes large language models the focus of attention in the past two years is their strong zero-shot in-context learning abilities. Unfortunately, it is still unclear whether semiparametric language models can exhibit similar zero-shot ability on unseen tasks as their fullyparametric counterparts such as T0 (Sanh et al., 2021) and GPT-3 (Brown et al., 2020). Moreover, improvements in language modeling metrics such as perplexity may not guarantee better performance on downstream tasks especially in low-shot settings (Wei et al., 2022). Thus, in this work, we aim to investigate this unexplored research question, can semi-parametric language models exhibit strong zero-shot generalization abilities on various downstream tasks?
To this end, we introduce Zemi, a zero-shot semi-parametric language model. To the best of our knowledge, this is the first semi-parametric language model that shows strong zero-shot perfor-
3978
mance on a wide range of downstream tasks. In order to effectively train Zemi, we propose to extend the multitask prompted training (Sanh et al., 2021) into semi-parametric settings (Section 2.1). Specifically, during both the training and the inference stage, we augment the prompted instances with retrieved plain text documents. To cover a wider range of unseen tasks, instead of retrieving from specific corpora for certain tasks, such as exploiting Wikipedia for open-domain question answering (Lee et al., 2019; Karpukhin et al., 2020; Izacard and Grave, 2020), we retrieve documents from a large-scale task-agnostic corpus, C4 (Raffel et al., 2020) (Section 2.2). Notably, C4 is the unlabeled pre-training corpus of our backbone model (Raffel et al., 2020), which means that every document is seen by the model and we do not require any annotated or curated resources. This guarantees fair comparison with the parametric counterpart T0 (Sanh et al., 2021).
In our preliminary experiments, we find that existing methods (Izacard and Grave, 2020; Brown et al., 2020) for incorporating retrieved text cannot effectively handle the noise inevitably introduced by retrieving from large-scale corpora. To address this challenge, we propose a novel retrievalaugmentation fusion module that can selectively ignore noisy retrieved text. Specifically, we introduce a light-weight perceiver resampler and a gated cross-attention layer (Alayrac et al., 2022) to enforce the model to attend to salient information of each augmentation and gate out noisy ones (Section 2.3).
We train Zemi on eight multiple-choice question
answering (QA) tasks (4.5x fewer than T0) and evaluate on a diverse set of seven unseen tasks from five categories (Section 3.1). In order to investigate the impact of the retrieval-augmentation, we favor knowledge-intensive tasks over extractive tasks.
Experimental results show that Zemi outperforms both parametric and semi-parametric baselines. Notably, ZemiLARGE outperforms T0-3B by 16% across seven evaluation tasks while being 3.8x smaller in scale (Section 3.2). We further conduct extensive analysis on why Zemi works. We show that the source of the improvements comes from the interplay of our proposed retrieval-augmentation fusion architecture along with the semi-parametric multitask training paradigm. Finally, we perform in-depth ablation studies on all aspects of our model design including the gated mechanism.
To sum up, the main contributions of this paper are threefold:
• We introduce Zemi, which is to our knowledge the first semi-parametric model that demonstrates strong zero-shot task generalization ability.
• We propose a novel retrieval-augmentation fusion module which can effectively handle multiple potentially noisy retrieved documents and is essential towards the effectiveness of semi-parametric multitask training.
• We show detailed analysis and ablation studies on why Zemi works which shed light on future work for developing large-scale universal semi-parametric language models with strong zero-shot ability.

heading: 2 Method

heading: 2.1 Semi-parametric multitask training
text: In this section, we introduce how we extend the multitask training paradigm to semi-parametric language models. We follow the overall text-to-text framework proposed by the previous parametric multitask prompted training (Sanh et al., 2021) where each input-output pair of a certain task is converted into a prompted text input and a generated text output via human-written templates (Bach et al., 2022).2 For Zemi, as illustrated in Figure 1, during both training and inference, we augment Zemi with a retrieval system. Instead of using specific corpora for different tasks, such as Wikipedia for open-domain question answering (Chen et al., 2017; Karpukhin et al., 2020; Izacard and Grave, 2020) and textbooks for science question answering (Mihaylov et al., 2018), we retrieve texts from a large-scale task-agnostic corpus, C4 (Raffel et al., 2020) (Section 2.2). Retrieving from a larger corpus brings wider coverage but also more noisy augmentations. To address this problem we further propose a novel semi-parametric architecture for Zemi that specializes in handling a large number of potentially noisy augmentations (Section 2.3). After semi-parametric multitask training, we perform zero-shot evaluation on seven diverse held-out unseen tasks (Section 3).

heading: 2.2 C4 retrieval
text: To build a universal semi-parametric language model that can generalize to various types of NLP tasks, we retrieve documents from Colossal Clean Crawled Corpus (C4) (Raffel et al., 2020). Notably, C4 is the unlabeled pre-training corpus of our backbone model T5 (Raffel et al., 2020), which guarantees fair comparison with non-retrieval methods in our zero-shot evaluation settings. The C4 corpus (750GB in size) contains more than 364 million documents. Performing dense retrieval (Karpukhin et al., 2020) on such a wide-coverage corpus is very expensive. Thus, for efficiency consideration, we perform document-level indexing and retrieval based on BM25 (Robertson et al., 1995) with ElasticSearch (ElasticSearch) and Huggingface Datasets (Lhoest et al., 2021). Despite its simplicity, recent work (Wang et al., 2022a) has demonstrated the effectiveness of using BM25 for retrieving clean training data as augmentations. To
2https://github.com/bigscience-workshop/promptsource.
further improve the retrieval efficiency, we use 5% of the entire C4 corpus, which is still 3x larger than the Wikipedia corpus (Foundation), as our retrieval corpus in our experiments. For each query, we truncate the query length at 20 tokens and truncate each retrieved document at 256 tokens. See details on the query fields for each dataset in Appendix D.

heading: 2.3 Zemi model architecture
text: One major challenge of retrieving from a largescale task-agnostic corpus is that the retrieved augmentations (documents) can be noisy and inaccurately ranked. Examples of good and noisy retrieved documents can be found in Appendix A. To address this problem, intuitively, we want the model to have the following two properties: (1) be able to simultaneously pay attention to multiple retrieved augmentations instead of only the top-1 document. (2) be able to identify salient information from the retrieved augmentations and selectively ignore uninformative ones.
To this end, we propose the Zemi architecture, a semi-parametric language model capable of selectively incorporating multiple potentially noisy retrieved augmentations. The main idea is to jointly train a light-weight retrieval-augmentation fusion module between the encoder and decoder, which contains two major components, a perceiver resampler and a gated cross-attention, which are inspired by recent work on vision-language fusion (Alayrac et al., 2022).
Figure 2 shows an illustration of the Zemi model architecture. We consider a prompted text input I and a few retrieved textual augmentations A1, A2, ..., Ak. Let lI , liA be the length of the prompted input and the ith augmentation. Let d be the hidden dimension of our backbone model. We first independently encode I and A1, A2, ..., Ak with a shared T5 (Raffel et al., 2020) encoder Enc. We then feed the latent representation of the augmentations A1,A2, ...,Ak through the perceiver resampler.
I = Enc(I) (1) Ai = Enc(Ai) (2) A′i = PerceiverResampler(Ai,Q) (3)
where ∀ i ∈ {1, .., k}, I ∈ RlI×d, Ai ∈ RliA×d and A′i ∈ RlQ×d.
As shown on the bottom right of Figure 2, the perceiver resampler is a variant of Perceiver IO (Jaegle et al., 2021), where a cross-attention is
performed between the variable-length latent representation of an augmentation Ai and a fixed-length learnable latent query vector Q. Let lQ be the predefined length of the latent query, which is typically smaller than the original length of an augmentation liA. The output of the perceiver resampler is a compressed fixed-length latent representation of each augmentation. This resampling mechanism not only allows the model to include longer and a larger number of augmentations but also encourages the model to select salient information from the original augmentations. After the resampler, we concatenate the encoded augmentations A′ = [A′i, ..,A′k] and perform gated cross-attention with the encoded prompted input I. As shown in the top right of Figure 2, the gated cross-attention layer contains two Tanh gates controlling the information flow from the cross-attention layer and the feed-forward layer before the addition with the skip connections, i.e., the original encoded input I. Finally, the hidden states from the gated crossattention module H is fed into the T5 decoder Dec
to generate the output sequence O.
H = GatedCrossAttn([A′i, ..,A′k], I) (4) O = Dec(H) (5)
where [A′i, ..,A′k] ∈ R(k×lQ)×d and H ∈ RlI×d. Following (Alayrac et al., 2022), we initialize
the parameter of the Tanh gate to be 0, allowing the forward pass of the prompted input through the pre-trained T5 encoder-decoder to be intact at the beginning of the training process. With the gated mechanism, the model can learn to gate out noisy augmentations and rely more on the skip connections during semi-parametric multitask training.

heading: 3 Experiments

heading: 3.1 Experimental setup
text: Following (Sanh et al., 2021) we partition various types of NLP tasks into two groups, training tasks and held-out unseen tasks. In this work, we are particularly interested in investigating the impact of the retrieval augmentation Thus, when choosing the training and evaluation tasks, we favor knowledge-intensive tasks over extractive tasks such as summarization, where most knowledge for solving the problem is already self-contained in
the input. Furthermore, we avoid including large datasets, such as DBPedia (Lehmann et al., 2015) (630K instances) and QQP (Shankar Iyer, 2017) (400K instances), due to limited computational resources.
Training Tasks We use a subset of T0’s (Sanh et al., 2021) training mixture for our semiparametric multitask prompted training. Specifically, our training mixture contains eight multiple-choice QA datasets, including, CommonsenseQA (Rajani et al., 2019), CosmosQA (Huang et al., 2019), DREAM (Sun et al., 2019), QASC (Khot et al., 2020), QUARTZ (Tafjord et al., 2019), SciQ (Johannes Welbl, 2017), Social IQa (Sap et al., 2019), and WIQA (Tandon et al., 2019). We choose the subset in multiple-choice QA tasks because they are diverse in domains and overall task formats. Ablation studies on including more types of training tasks can be found in Section 3.4.
Evaluation Tasks For evaluation tasks, we consider seven datasets from five diverse categories following the task taxonomy of T0, including, two sentence completion tasks, COPA (Roemmele et al., 2011) and HellaSwag (HSwag) (Zellers et al., 2019), two multiple-choice QA tasks, OpenbookQA (OBQA) (Mihaylov et al., 2018) and Piqa (Bisk et al., 2020), one word sense disambiguation task, WiC (Pilehvar and CamachoCollados, 2018), one sentiment task, Rotten Tomatoes (RT) (Pang and Lee, 2005), and one natural language inference task, CB (De Marneffe et al., 2019). All scores are reported on the validation set of each dataset. The detailed prompt templates used for training and evaluation can be found in Appendix C.
Prompts We use PromptSource (Bach et al., 2022) with Huggingface Datasets (Lhoest et al., 2021) to construct prompted inputs for each training and evaluation instance. During training, we randomly select two templates for each dataset. During evaluation, we follow the exact evaluation procedure as in T0 (Sanh et al., 2021) and report the mean accuracy across all available templates. All scores are reported on the validation set of each dataset. The detailed templates used for training and evaluation can be found in Appendix C.
Model We consider two variants of Zemi with a different pre-trained backbone, i.e., T5-base and T5-large (Raffel et al., 2020). Following T0 (Sanh et al., 2021), we use the language modeling adapted3 checkpoint, which is trained for an additional 100k steps on a language modeling objective. By default, we use five retrieved passages as augmentations for each instance. More implementation details can be found in Appendix B

heading: 3.2 Main results
text: We aim to explore whether Zemi can exhibit competitive zero-shot performance against larger stateof-the-art language models. We compare Zemi with both parametric (T0 (Sanh et al., 2021), BART0 (Lin et al., 2022), GPT-3 (Brown et al., 2020)) and semi-parametric (ReCross (Lin et al., 2022)) models on seven zero-shot tasks. Table 1 shows the mean zero-shot accuracy across all templates for each task. The last two columns of Table 1 show the averaged performance across different sets of tasks, where Avg7 is averaged across all seven tasks, and Avg5 considers five tasks excluding RT and COPA due to their unavailable baseline
3https://huggingface.co/google/t5-base-lm-adapt.
results. For BART0, ReCross, and GPT-3, we copy the
reported scores directly from their original papers. For the missing score of RT on GPT-3, we run the original text completion API 4 to get the generated outputs which is then mapped to the most similar answer choice using SentenceBert (Reimers and Gurevych, 2019). For T0 models, there are some tasks such as OBQA and Piqa that are not evaluated in the original paper (Sanh et al., 2021), and some tasks such as CB and WiC are evaluated with slightly different templates. Thus, for fair comparison, we re-evaluate all seven tasks on T0-3B and T0-11B using the official implementation and checkpoints5 with the exact same set of templates as our model. See details on the templates used for each task in Appendix C.
Result Table 1 shows that ZemiBASE outperforms previous retrieval-based method, ReCross, on the average of five tasks (Avg5) while being 2x smaller in scale. Notably, ZemiLARGE, significantly outperforms T0-3B on seven evaluation tasks (Avg7) by 16% with 3.8x fewer parameters. This shows that Zemi scales up well with larger backbone models. We also observe that although trained with 4.5x fewer training tasks (8 v.s. 36), Zemi effectively achieves state-of-the-art zero-shot performance. In Section 3.4, we show that adding more tasks into multitask training does not necessarily improve the performance. And the training mixture with multiple-choice QA tasks seems to be highly effective in generalizing to various kinds of unseen tasks.

heading: 3.3 Analysis: semi-parametric v.s. parametric
text: In order to further analyse the source of the strong performance of ZemiLARGE, we compare ZemiLARGE with a baseline (No Aug) trained with parametric multitask training on the same set of training tasks and with the same backbone model, T5-Large (Raffel et al., 2020). To show the impact of our newly proposed retrieval-augmentation fusion module, we further compare ZemiLARGE against two semi-parametric baselines with a different fusion method for incorporating the retrieved augmentations (Concat and FiD). In Table 2, we show that the source of benefit comes from the interplay of the pro-
4For consistency with other results, we report the RT result from the original “davinci” model.
5https://github.com/bigscience-workshop/t-zero.
posed retrieval-augmentation fusion and the semi-parametric multitask training.
Specifically, for Concat, we directly concatenate all retrieved augmentations with the prompted input text. The concatenated input is then truncated to the maximum acceptable length of 1024 tokens and fed to our backbone model. For FiD, we implement the model following (Izacard and Grave, 2020) where we first independently encode each pair of retrieved augmentation and the prompted input text. Then we concatenate the encoder outputs and feed them to the decoder. Note that we keep everything else identical except the retrieval-augmentation fusion module for ZemiLARGE, Concat and FiD.
Zemi architecture improves zero-shot task generalization. In Table 2, we first notice that the semi-parametric setting in itself does not necessarily bring consistent positive gains compared with the No Aug baseline, as shown in the results of Concat and FiD. This can be explained by the fact that the retrieved documents are not always highly correlates with the task of interest, as shown in the examples in Figure 3. The fact that FiD performs better than Concat further verifies this hypothesis, since FiD preserves more input text information in the encoding step and only do fusion with all the retrieved augmentations in the decoder, whereas Concat perform unified self-attention on all augmentations concatenated directly to the input.
On the other hand, with the proposed retrieval-augmentation fusion module that contains the explicit resampling and gating mechanism, ZemiLARGE was able to achieve the best performance on six out of seven tasks, and brings a overall gain of +5% against the No Aug baseline. This result shows that the retrieval-augmentation fusion module in Zemi can effectively enable the model to leverage potentially noisy retrieved augmentations during semi-parametric multitask training, which
brings significant improvement in zero-shot task generalization. In abaltion study 3.4, we further verify that the gated cross-attention is an important factor contributing to the effectiveness of the Zemi architecture.

heading: 3.4 Analysis: ablation studies
text: In this section, we continue investigating why Zemi works by conducting comprehensive ablation studies on different aspects of the model design. As shown in Table 3, we consider the following five categories of ablated settings on ZemiBASE 6:
(i) Tanh gate. We replace the gated crossattention module with vanilla cross-attention in the ablated version. Specifically, we remove the two Tanh gates as shown in Figure 2. We find that removing Tanh gate hurts the zero-shot performance. Note that the Tanh gate is also the main difference between Zemi and FiD (Izacard and Grave, 2020).
(ii) Number of augmentations. We ablate on the number of augmentations. Note that for settings with 20 and 30 augmentations, in order to reduce the computation complexity, we propose another variant of ZemiBASE where we encode augmentations with a separate frozen augmentation encoder. We find that increasing the number of augmentations from single to multiple (five) improves the performance. However, further increasing the number to 10 starts to hurt the performance, which again indicates that the noise starts to overwhelm the useful signals introduced by the retrieval. We also observe that the performance with 30 augmentations outperforms 20 augmentations, we hypothesis that this is due to inaccurate retrieval ranking that leads to some more informative documents being ranked lower. We show an example of this
6We ablate on ZemiBASE instead of ZemiLARGE mainly to reduce the computation overheads of a large amount of experiments.
case in Figure 11. Nevertheless, the fact that we are able to achieve positive gain with as many as 30 augmentations shows the robustness of our model to very noisy augmentations.
(iii) Perceiver resampler latent size. We ablate on the size of the latent query vector in the perceiver resampler. Note that here the latent size is different from the hidden state size of the backbone model. The trade-off of the size of the latent query vector is that, a larger latent size preserves more information from the original augmentation but also includes more noise. A larger latent size can also increase the computational complexity. We find that Zemi is relatively robust to the change of the latent size and achieves the best performance with a latent size of 64.
(iv) Per augmentation length. We investigate the impact of different ways of constructing augmentations from the retrieved documents. Specifically, we increase the maximum length of each augmentation from 256 to 512 and fit two retrieved documents into one augmentation. We keep the number of augmentations the same as default, i.e., 5. We then compare this ablated setting with the 10-augmentation variant in (ii). We find that with the same set of retrieved documents, augmenting the model with longer but fewer augmentations generally outperforms using a larger number of shorter augmentations.
(v) Training mixture. We investigate the impact of adding new types of training tasks to the original training mixture. We dub the models trained with this new training mixture as No Aug+ and Zemi+. Specifically, apart from the eight multiple-choice QA tasks, we further include four more tasks: one closed-book QA task WikiQA (Yang et al., 2015), one topic classification task TREC (Li and Roth, 2002), one sentence completion task COPA (Roemmele et al., 2011), and one sentiment task Rotten
Tomatoes (RT) (Pang and Lee, 2005) 7. We find that adding new types of tasks does not necessarily increase the performance. Although trained with only 8 tasks (v.s. 36 tasks) we are able to achieve state-of-the-art performance (Section 3.2), which shows that the multiple-choice QA mixture is highly effective for generalizing to a wide range of held-out unseen tasks.

heading: 3.5 Analysis: computation overheads
text: There are two main computation overheads compared with the fully-parametric counterpart, i.e., the No Aug baseline. First, retrieving from a large-scale corpus can be time-consuming. As mentioned in Section 2.2, we apply document-level retrieval with BM25 and truncation on the query to reduce the retrieval time. We also perform the retrieval offline to avoid repeated time commitment. As a result, indexing 5% of the C4 corpus takes 1 hour. Offline retrieval for the entire training and evaluation mixture takes 11 hours, which is approximately 0.28 seconds per instance. Furthermore, we measure the computation overhead on inference which is caused by the additional retrieved inputs as well as a small amount of newly introduced parameters (+4.6%). The average computation overhead across all evaluation datasets during inference is around
7We follow T0 to move tasks that are originally in the evaluation split, i.e. COPA and RT, into the training split in this ablated setting.
4x compared with the No Aug baseline. Notably, Table 2 shows that ZemiBASE achieves competitive performance with T0-3B while being 15x smaller in scale, indicating that the benefit of the retrieval augmentation overwhelms the computation overhead.

heading: 4 Related Work

heading: 4.1 Semi-parametric models
text: Semi-parametric models (Sun et al., 2021; Verga et al., 2021; Chen et al., 2017; Lee et al., 2019; Guu et al., 2020; Wang et al., 2019; Karpukhin et al., 2020; Yang et al., 2019; Lewis et al., 2020; Izacard and Grave, 2020), which augmenting a parametric neural network with external knowledge bases or text corpora, have been widely applied to knowledge-intensive NLP tasks such as opendomain question answering. Recent advancements in semi-parametric language models (Khandelwal et al., 2019; Yogatama et al., 2021; Borgeaud et al., 2021; Zhong et al., 2022) have demonstrated improved language modeling performance with a relatively small language model and a retrieval system based on a large-scale corpus. Although the aforementioned semi-parametric language models have shown competitive performance on language modeling, compared with fully-parametric counterparts such as GPT-3 (Brown et al., 2020), it is unclear whether the superiority generally holds on down-
stream tasks. While concurrent work (Izacard et al., 2022) showed initial success in few-shot settings relying on Fusion-in-Decoder (FiD) (Izacard and Grave, 2020) framework, this work focus on the more challenging zero-shot settings (Sanh et al., 2021; Zhou et al., 2022; Gu et al., 2022). Furthermore, instead of reusing FiD framework as in (Izacard et al., 2022), we show that our newly proposed fusion module is more effective than FiD due to the gated mechanism, which is inspired by Highway Networks (Srivastava et al., 2015; Chai et al., 2020), Gated Convolution (Dauphin et al., 2017) and Vision-Language Fusion(Alayrac et al., 2022).

heading: 4.2 Massive multitask prompted training
text: Based on the assumption that the reasonable zeroshot ability of large language models may come from implicit multitask learning during pretraining, recent studies (Sanh et al., 2021; Wei et al., 2021; Ye et al., 2021; Wang et al., 2022b) have demonstrated that explicitly training a language model on a mixture of diverse tasks can effectively improve its zero-shot performance on unseen tasks. In this work, we extend T0’s multitask prompted training to a semi-parametric setting, where we further augment the training and evaluation instances with retrieved documents. Notably, our work is distinguished from previous work ReCross (Lin et al., 2022), which uses upstream training data for augmentation, in twofold. First, we retrieve documents from a much larger task-agnostic corpus instead of clean upstream training instances. Second, in addition to directly concatenating the augmentation with the input just as FiD (Izacard and Grave, 2020), we further propose a novel retrieval-augmentation fusion module to handle retrieval noise.

heading: 4.3 Fusion of retrieved augmentations
text: In this work, the main challenge of designing the semi-parametric language model architecture is how to effectively leverage potentially noisy retrieved documents. Existing methods on incorporating external texts fall in two categories, direct concatenation (Lin et al., 2022; Brown et al., 2020; Liu et al., 2021; Lewis et al., 2020; Wang et al., 2022a) and cross-attention (Izacard and Grave, 2020; Prabhumoye et al., 2021; Borgeaud et al., 2021). However, we find that prior work lacks an explicit design for preventing the model from attending to noisy augmentations. Inspired by recent visual lan-
guage models (Alayrac et al., 2022; Yu et al., 2022; Li et al., 2022; Jiang et al., 2022), we find that we can actually borrow ideas from vision-language fusion for text-text fusion. We identify two key differences from Flamingo architecture: first, we use a much smaller encoder-decoder model that is jointly trained with the newly initialized layers instead of frozen layers. Second, instead of inserting the gated cross-attention module into a large frozen language model (Hoffmann et al., 2022), we add only one layer of gated cross-attention on top of the encoder to alleviate the need for extensive pre-training.

heading: 5 Conclusion
text: In this work, for the first time, we show that semi-parametric language models have the potential to exhibit strong zero-shot task generalization ability by introducing Zemi. Through extensive analysis and ablation study, we further demonstrate that the interplay of the proposed retrieval-augmentation fusion and the semiparametric multitask training is essential towards Zemi’s empirical success. Notably, our proposed ZemiLARGE model outperforms T0-3B by 16% across seven diverse evaluation tasks while being 3.8x smaller in scale.

heading: 6 Limitation
text: In Section 3.2, we show that our training mixture with multiple-choice QA tasks, although small, is highly effective for multitask training. However, it is still unclear why multiple-choice QA tasks are particularly effective. Identifying the key factors towards positive or negative transfer from different tasks in the multitask training mixture would greatly help improve zero-shot task generalization. Future work includes investigating why certain mixtures are more effective than others and expanding the evaluation set to a wider range of tasks. Computation overhead is another noticeable limitation of semi-parametric models which is discussed in detail in Section 3.5. Moreover, future work on developing more efficient and accurate retrieval methods for retrieving from large-scale task-agnostic corpus can definitely improve semiparametric language models.

heading: Acknowledgements
text: We would like to express our gratitude to the anonymous reviewers for their insightful comments and
suggestions. We would also like to thank our colleagues and fellow interns at Tencent AI Lab for their valuable internal discussions and feedback, as well as the students from Blender Lab at the University of Illinois Urbana-Champaign for their insightful feedback.

heading: A Qualitative analysis of the retrieved documents
text: Here we visualize one good and one noisy example of the retrieved documents for each evaluation task. A full list of examples for each training and evaluation task can be found in the supplementary material under the “visualization” folder. As shown in Figure 4, 5, 6, 7, 9, 8, and 10, the retrieved augmentations can contain highly correlated information that can be directly helpful for solving a certain task, however, they can also be very noisy. As mentioned in Section 2.2, the retrieved documents can also be inaccurately ranked, for example in Figure 11, we show that the 21th ranked retrieval result can contain more correlated information than the top ranked ones. Furthermore, as shown in the noisy example of Figure 7, for some tasks such as sentiment analysis, even though the retrieved document is highly correlated with the input text, i.e., with a high BM25 score, the content can steer the prediction into a wrong direction. These observations motivate us to propose the augmentation fusion module with a gated mechanism.
B Implementation details
We use T5-base and T5-large as backbone model for ZemiBASE and ZemiLARGE, respectively. We follow (Alayrac et al., 2022) to implement the perceiver resampler and the gated cross-attention. For both ZemiBASE and ZemiLARGE, unless otherwise specified, we use one layer of gated cross-attention and one layer of perceiver resampler with a latent size of 64. A comprehensive ablation study on the impact of different aspects of our model design such as the Tanh Gate can be found in Section 3.4. All models are trained on the same training mixture as mentioned in Section 3.1 for ten epochs with a batch size of 32 and a learning rate of 1e-4. We report results from the checkpoint that achieved the best overall performance across all tasks. All experiments are done on eight NVIDIA-V100 32GB GPUs.

heading: C Full list of tasks and templates
text: Following T0 (Sanh et al., 2021), we use tasks from Hugginface Datasets (Lhoest et al., 2021) and templates from PromptSource (Bach et al., 2022) marked as “original task" and with “choices_in_prompt". Specifically, for tasks in the training mixture, we randomly sample two
templates per task for semi-parametric multitask prompted training. For tasks in the held-out evaluation mixture, we use all available templates. Table 4, and 5 shows the full list of templates we used for each task during multitask training and zero-shot evaluation.

heading: D Retrieval query key for each task
text: In order to retrieve most relevant documents for each instance, we specify a certain field for each dataset which will be served as the query to the retrieval system. For example, for most multiplechoice QA tasks, we use the “question” string as our query. Table 6 shows a full list of field names we use as retrieval query keys for each dataset. Note that the field name shown in the table is what appears to be in the corresponding Huggingface Dataset (Lhoest et al., 2021).

heading: E Broader impact
text: One major benefit of developing powerful semiparametric language models is that we can reduce the negative environmental impact from training huge parametric models. However, since the backbone language model is pretrained on massive internet-scale text data, there might be unexpected output that can have potential negative impact on the society, such as bias against people of a certain gender, race or sexuality. We are fully aware of the risks of potential misuses and will actively work with the community to improve the responsibility of large NLP models.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 6
3 A2. Did you discuss any potential risks of your work? Appendix F Broader Impact
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Footnote 1. See supplementary material
3 B1. Did you cite the creators of artifacts you used? Reference. All datasets and models used in this paper are cited.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Reference and footnotes. All the datasets and models used and created by this work are publicly avaliable for research purpose.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 3.1
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 3.1
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? See README of the supplementary code.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3.1
C 3 Did you run computational experiments? Section 3
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 3, Appendix B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 3, Appendix B
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3, Appendix B, C, D
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Future work includes investigating why certain mixtures are more effective than others and expanding the evaluation set to a wider range of tasks. Computation overhead is another noticeable limitation of semi-parametric models which is discussed in detail in Section 3. 5. Moreover, future work on developing more efficient and accurate retrieval methods for retrieving from large-scale task-agnostic corpus can definitely improve semiparametric language models