HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisation
Year: 2023

Abstract:
Recent NLP models have shown the remarkable ability to effectively generalise ‘zero-shot’ to new tasks using only natural language instructions as guidance. However, many of these approaches suffer from high computational costs due to their reliance on concatenating lengthy instructions with every input example, resulting in costly reprocessing of the instruction. To avoid this, we introduce Hypernetworks for INstruction Tuning (HINT), which convert task instructions and examples into parameter-efficient modules inserted into an underlying model using a pretrained text encoder, eliminating the need to include instructions in the model input. The hypernetwork in HINT also produces an encoded instruction, which we concatenate with encoded inputs during decoding to further improve performance. HINT models outperform strong state-of-theart baselines by over 10% when controlling for compute (measured in FLOPs). By converting instructions into modules, HINT models can effectively disregard the length of instructions and few-shot example inputs in terms of compute usage. As a result, HINT can enhance its performance by up to 25% by incorporating additional few-shot data, while utilizing only up to 5% more compute. This combines the strengths of parameter-efficient fine-tuning and in-context learning. We release our code publicly1.

heading: 1 Introduction
text: Large pretrained language models have demonstrated a striking ability to perform new tasks through the use of in-context examples or instructions alone (Brown et al., 2020), or after training on input instances augmented with instructions (Weller et al., 2020; Mishra et al., 2022; Sanh et al., 2022; Wei et al., 2022; Chung et al., 2022; Wang et al., 2022b). This ability allows a single model to
1Our code is available at: https://github.com/allenai/hyper-task-descriptions.
adapt to many tasks where training data is difficult to collect or task-specific fine-tuning is impractical (i.e., ‘zero-shot’ settings): models trained on instructions need only a single instruction to achieve non-trivial performance on the task at hand. The most common method to achieve this zero-shot ability is to meta-train the model with task instructions concatenated with every input, allowing the model to learn to associate instructions with tasks. While empirically highly successful, this is inefficient and requires reprocessing lengthy task instructions and any additional task data (e.g., few-shot examples) with every input example.
In this paper, we introduce Hypernetworks2 for Instruction Tuning (HINT), which directly 2Hypernetworks are neural networks trained to generate neural networks (Ha et al., 2017).
11272
generate task-specific parameter-efficient modules given only an instruction, combining the benefits of instruction-based learning with parameterefficient modules. HINT models convert instructions and other task data (e.g., few-shot examples) into efficient modules within a pretrained language model, enabling cheaper inference and better compute scaling with few-shot data for an underlying instruction-based meta-learning approach. Additionally, fusing hypernetwork-encoded instructions with the encoded input at the underlying model decoder greatly improves the performance while using minimal extra compute. An important benefit of HINT is that it processes instructions and other task information only once, making the compute used by our method almost independent of the amount of task data available, unlike both regular finetuning and input concatenation-based approaches (see Figure 3).
We find that our hypernetwork-based approach (‘HINT’), is able to achieve similar performance to baselines that receive the full instruction with every input example while using significantly less compute (as measured by FLOPs), due to the greatly reduced input length. When controlling for inference budget, we find that HINT models outperform strong baselines in zero- and few-shot settings. This validates our assumption that we can significantly reduce inference costs by avoiding reprocessing the instruction with every input, and instead saving it for repeated use. Furthermore, we find that including additional few-shot information alongside task instructions significantly improves HINT model performance while using minimal additional compute during inference. Ultimately, our work pushes towards directly generating cheap, customised models from task data, without requiring any expensive task-specific finetuning.
In summary, our findings are:
• We introduce HINT models, which make use of a text-conditioned hypernetwork to generate parameter-efficient modules based on task descriptions and few-shot examples.
• HINT models, by reducing input lengths, are able to achieve similar performance to strong full-input baselines while reducing inference cost (measured in FLOPs) by up to 4×.
• As the compute used by HINT models is effectively independent of the length of the instruction and amount of few-shot data provided
with the instruction, HINT models provided with additional few-shot data simultaneously outperform and use up to 4× fewer FLOPs than baselines without few-shot data.
• HINT models outperform strong decoder-only baselines. While decoder-only models allow for input caching, we find that instructiontuned GPT-2 models significantly underperform HINT models (8-9 point difference), matching prior work suggesting that encoderdecoder models work better for instructiontuning (Wang et al., 2022a; Iyer et al., 2022).

heading: 2 Related Work
text: Instruction Following Further finetuning large pretrained language models on instructions has been found to greatly improve zero-shot generalisation, as the finetuned model learns to make use of the instructions to perform the given task (Weller et al., 2020; Wei et al., 2022; Mishra et al., 2022; Chung et al., 2022; Wang et al., 2022b). Additionally, Sanh et al. (2022) found that training models on multiple prompts per task also resulted in improved performance, suggesting that further increasing prompt diversity aids generalisation, even when using the same pool of underlying tasks. The majority of these popular instruction-tuning approaches involve concatenating the instruction with the input directly and training a text-to-text model on these combined inputs. As the instruction can be as long as, if not longer, than the input3, this can greatly increase the computation needed to process inputs compared to task-specific models.
In-Context Learning Similar to instructionbased models, in-context learning (Brown et al., 2020), where example instances are used in place of or in addition to instructions, also requires extremely long and expensive-to-process inputs for every test example, with Liu et al. (2022) showing that parameter-efficient finetuning (PEFT) can be cheaper and more effective when dealing with many test examples. In this work, we propose a halfway step between PEFT and instruction concatenation, where we train a model to predict parameter-efficient modules based on instructions, avoiding the few-shot training required by Liu et al. (2022) while also avoiding repeatedly processing lengthy inputs.
3As is the case for Super-Natural Instructions, see Appendix A.1.
Hypernetworks In NLP Hypernetworks (Ha et al., 2017; Schmidhuber, 1992) in NLP have recently gained popularity in multitask and multilingual setups due to their ability to softly share parameters while avoiding negative interference through the use of shared parameter generation module. Several approaches (Karimi Mahabadi et al., 2021; Tay et al., 2021; He et al., 2022b) learn per-task embeddings along with a shared hypernetwork to generate task-specific adapter or prefix modules. This means making the model perform new tasks requires at least few-shot learning to learn a task embedding. Recent work has explored using text-conditioned hypernetworks for parameter-efficient multitasking (Ivison and Peters, 2022) or improving out-of-domain generalisation (Volk et al., 2022), removing the need to train taskspecific embeddings. Hypernetwork-based methods have also been highly successful in multilingual settings, where generating language-specific models via shared hypernetworks often results in improved performance across various tasks (Platanios et al., 2018; Baziotis et al., 2022; Ustun et al., 2022, inter alia)
Our work primarily builds on Ye and Ren (2021), which explored generating adapters from task descriptions. We expand their approach to larger models and datasets and find that pretraining and a significantly different hypernetwork architecture are important for achieving strong performance.
Our work is also similar to the concurrently developed Phang et al. (2022) and Deb et al. (2022), both of which examine how well hypernetworkbased meta-learning can improve model performance in zero- and few-shot settings. Deb et al. (2022) examine hypernetworks and model-agnostic meta-learning for instruction-finetuning and find that they can yield improved performance on difficult unseen tasks in Super-Natural Instructions. However, they still struggle to achieve overall good zero-shot performance and do not investigate eliminating task descriptions from the model input itself. Phang et al. (2022) find that training a hypernetwork to produce model adaptations provides an initialisation better than pretraining for parameterefficient adaptations and that this initialisation improves with more few-shot examples provided to the hypernetwork. They also explore eliminating the instruction from the underlying model input but find this severely underperforms baseline approaches. We have similar findings, but find
that our novel hypernetwork design and use of instruction fusion closes the gap with baseline approaches. We also perform further analysis of the hypernetwork-based models and show that when controlling for inference compute budgets, our hypernetwork-based model still outperforms strong baselines.

heading: 3 HINT Model
text: Here, we introduce the main elements of our HINT model. The model has two core parts: the hypernetwork, which takes in text instructions and outputs parameter-efficient modules, and the underlying model, into which we insert the generated parameter weights. The underlying model is simply an encoder-decoder4 transformer model (Vaswani et al., 2017) with additional parameter-efficient adaptations inserted in, while the hypernetwork has a more complex architecture which we describe below. Figure 1 provides a visual overview.

heading: 3.1 Hypernetwork
text: The first step in our model is to make use of a hypernetwork to convert an instruction to parameter-efficient modules. Our hypernetwork consists of three core elements: an encoder (or ‘hyperencoder’) to transform instruction and fewshot text into continuous (contextual) representations, saving the encoded instructions for instruction fusion during decoding, and a parameter generator to then convert these embeddings into the parameter efficient modules.
HyperEncoder To encode our text, we use a pretrained language model encoder. We initially experimented with using different encoder configurations, and find that re-using the encoder from the underlying model we wish to augment works well, and tying the hypernetwork and underlying encoder model weights works best.
Instruction Fusion We save the instruction representations produced by the hyperencoder and allow the decoder of the underlying model access to them by concatenating them with input examples during inference and training. This is inspired by the fusion-in-decoder method used in open-domain QA (Izacard and Grave, 2021).
4We use encoder-decoder models as they generally outperform decoder-only models for zero-shot generalisation – see Section 5.1.

heading: 3.1.1 Efficient Parameter Generators
text: Parameter Generators Our generator design consists of two parts. First, we use a trainable set of embeddings and perform multi-head crossattention with the encoded instruction and these embeddings. Each embedding represents a unique column or token in each parameter-efficient module (e.g., prefix, adapter - see below) for each layer. This allows us to effectively collect the information required for different parameters in different embeddings via cross-attention with the instruction:
embed = [αe11 ,αe12 , ...,αe21 ,αe22 , ...,πe11 , ...]
embed′ = Cross-Attention(embed, instr.)
Where αe11 refers to an embedding we will use as the first column of the first layer adapter weight, αe12 is the second column, αe21 is the first column for the second layer adapter, πe11 is the first token of the first layer prefix, etc. We then take the subset of the embedding representing all columns/tokens for a particular model adaptation and pass it through a two-layer MLP to generate parameters. We use a unique network for each adaptation and share between layers (i.e. one network for prefixes for all layers, one for all adapter weights for all layers, etc.).
Adapter1 = reshape[MLPa(α ′ e11 );MLPa(α′e12); ...]
Prefix1 = reshape[MLPp(π′e11);MLPp(π ′ e12 ); ...]
Where Adapter1 and Prefix1 are the first layer adapter and prefix, respectively.
Generated Parameters We generate two types of parameter-efficient modules: adapters (Houlsby et al., 2019) and prefixes (Li and Liang, 2021). Adapters (Houlsby et al., 2019) are small bottleneck networks inserted into a transformer model. We follow He et al. (2022a) in placing our adapters parallel to the feed-forward layer:
x = FFN(x) + f1(GELU(f2(x))) (1)
Where f1, f2 are linear layers that project an input x to a small bottleneck size na and then back up to the hidden size of the model respectively. Prefixes (Li and Liang, 2021) are short continuous sequences concatenated with the key and values in the self- and cross-attention modules in every layer of the underlying model.
Scaling Down Parameters A naïve hypernetwork implementation may suffer from poor scaling with the size of the parameter-efficient modules. Consider a case where we wish to convert a single embedding of size ne to an adapter weight matrix of size nd × na (the model hidden dimension size by adapter bottleneck size). Our hypernetwork generator will have nd ∗ ne ∗ na parameters, and so increasing the adapter bottleneck na quickly becomes extremely expensive, especially if nd is large - as is the case for large language models. We address this by decomposing the adapter weight into columns and assigning an embedding per column. Thus, our hypernetwork now has to convert a sequence of embeddings with size na × ne to an adapter weight of size na × nd, meaning that the network only needs ne ∗ nd parameters. This means that the size of our parameter-efficient modules is independent of the size of the hypernetwork, and we can effectively scale the size of our adapters or number of prefixes without extreme parameter blowup. Note that we set ne = nd in our experiments for simplicity.

heading: 3.2 Underlying Model
text: Once our hypernetwork has produced a set of parameter-efficient modules, we then insert these into our underlying network, and can then perform training and inference as normal. The underlying model can be any pretrained encoder-decoder model that works with our parameter-efficient modules. We make use of T5 (Raffel et al., 2022) as our underlying model in our experiments.

heading: 3.3 Training and Inference
text: Hypernetwork Pretraining To help better generalization, we pretrain the hypernetwork on a large corpus (C4; Raffel et al., 2022) before finetuning on multitask prompted datasets. Given a single input string, we split our input string into random-length chunks a, b, c, and feed a to the hypernetwork, b to the main network, and predict c. This resembles the input in used in instruction finetuning (as the instruction precedes the input in the default prompt used for Super-Natural Instructions). We fully fine-
tune all parameters during pretraining.
Training HINT HINT training looks similar to pretraining, except we replace a with the task instruction (and any few-shot examples), b with the main input, and c with the gold generation. We used mixed-task batches such that a unique adaptor and prefix set is generated for every input in each batch.
This means, for every batch, we first generate a set of adapters, prefixes, and encoded instructions from a batch of tasks using the hypernetwork. The adapters and prefixes are placed within the underlying model to act as parameter-efficient modules (i.e., insert them into the model), and the encoded instruction is concatenated with encoded inputs during decoding. We then perform a forward pass of the underlying model with the inputs associated with each task in the batch and perform backpropagation using cross-entropy loss as standard for text-to-text models. As we fully finetune all parameters, the parameter generator will produce different weights for the same task inputs after a gradient step, meaning that we have to rerun the hypernetwork for every batch. This means that HINT requires more compute to train than a baseline transformer - although it provides significant compute reductions during inference, as we will see.
Inference The inference process is similar to training, but we do not use mixed-batch inputs: instead, we generate the parameters for one task, insert them into the underlying model, and then process all test-time inputs for that task. This prevents redundant processing of the instruction.
We also consider the cost of HINT models during inference. We consider a case where we have to process n samples from a single task. Assume each sample has length i and the task instruction has length t. We will ignore the cost of processing (typically short) output sequences. Following prior work (Kaplan et al., 2020; Liu et al., 2022), we use FLOPs as an estimate of the amount of compute required to run particular models and estimate that processing a token with an encoder-decoder model takes N FLOPs to process a single token, where N is the total number of model parameters.
In this scenario, a standard instruction-trained model which concatenates every input with the instruction (e.g., Tk-Instruct) uses Nn(t+ i) FLOPs to process all examples. Meanwhile, HINT models
process the task instruction only once and so use roughly N(t+ ni) FLOPs5. This makes clear that HINT models (a) scale better with more sametask inference examples than input concatenation approaches (increasing n), and (b) require relatively few extra FLOPs to process long instructions (large t), allowing them to benefit from adding more few-shot examples without incurring significant compute increases.

heading: 4 Experimental Details
text: We evaluate our approach on two popular instruction-based datasets: Super-Natural Instructions (SNI) (Wang et al., 2022b) and the T0 split of P3 (Sanh et al., 2022; Bach et al., 2022). We use t5x and seqio (Roberts et al., 2022) to handle data preprocessing and model training. We use T5 v1.1 + LM adaptation (Lester et al., 2021) as our base models, using the 3B size unless otherwise stated. Unless otherwise stated, the hypernetwork generates prefixes of length 30 and adapters with a bottleneck size of 512, matching the sizes recommended by He et al. (2022a). We use the Adafactor optimizer (Shazeer and Stern, 2018) with a constant learning rate of 0.001. Unless otherwise stated, we report results from single runs.
Pretraining We pretrain all models for 10,000 steps (7,000 for 11B size models) using C4 (Raffel et al., 2022) with a batch size of 1,024 samples and sequences of length 512.
Super-Natural Instructions (SNI) For SNI, we examine two settings: providing the hypernetwork with the task definition and the underlying network with the instance input only (‘Def’), and providing the hypernetwork with the task definition and two few-shot task examples (‘Def + 2 Pos.’). To train, we finetune our pretrained HINT models for 1,000 steps with a batch size of 1,024, with a maximum sequence length of 1,024 for both the underlying model and the hypernetwork input. We then evaluate the final checkpoint on the test split of SNI, which is a set of 119 unseen tasks. We use v2.6 of Super-Natural Instructions.
P3 For P3, we explore two settings: (a) ‘joint’, where we give the hypernetwork a templated form of the prompt with instance information removed and give the underlying model the full prompted
5When reporting FLOPs, we use a more detailed formula described in Appendix C.1 that takes into account extra (albeit small) hypernetwork costs.
input. (b) ‘split’, where we give the hypernetwork the templated prompt without instance information and give the underlying model only the instance information without the prompt. In both cases, we fully finetune our model for 10,000 steps with a batch size of 2,048. We use a maximum input length of 1,024 for the underlying model and 512 for the hypernetwork. We train and evaluate on the same tasks and splits as T0 (Sanh et al., 2022).
Baselines We primarily compare against T0 and Tk-Instruct, models fully-finetuned on P3 and SNI respectively with all task information concatenated with the input. We replicate these models, matching the finetuning settings used for HINT models, and find that our replications significantly outperform previously reported results, making these baselines extremely strong. We note where results are our replications or reported from prior work. We additionally compare against ‘X + PEFT’, the prior models with adapters and prefixes added in before finetuning, HyperTune (Phang et al., 2022), a concurrent work that primarily makes use of a pretrained hypernetwork but without instruction fusion, Hypter (Ye and Ren, 2021), a prior hypernetwork-based model that does not use pretraining, GPT-2 (Radford et al., 2019), a strong decoder-only model, which we fully finetune, and OPT (Zhang et al., 2022), another strong decoderonly model, which we also fully finetune.

heading: 5 Model Performance and Efficiency

heading: 5.1 Super-Natural Instructions
text: We report the performance and inference costs of HINT models and baselines in Table 1 and Figure 3 and find that:
HINT models outperform baselines when
FLOPs-matched. As seen in Figure 3 (left), when FLOPs-matched, HINT models outperform TkInstruct, a strong baseline that fully concatenates the instruction with every input. This holds for both ‘Def’ and ‘Def + 2 Pos.’ settings.
HINT models are up to 4× more efficient than similarly-sized baselines. we find that HINT models use 2–4x fewer FLOPs than similarlysized state-of-the-art Tk-Instruct baselines (Table 1). While other hypernetwork-based models are able to achieve similar compute savings, their performance is significantly worse than HINT (≥ 8 points). HINT has similar cost to a model trained without including instructions in the input (‘NoInstruct’), while performing over 30 points better.
HINT models improve performance with fewshot examples, but do not cost more FLOPs. When introducing additional few-shot data (‘Def + 2 Pos.’), HINT models improve dramatically (5- 8 points) but the compute used barely increases (Figure 3, centre), as HINT models only need to encode the task data (instruction and few-shot examples) once per task. In contrast, while Tk-Instruct similarly improves with few-shot examples, the compute needed during inference increases dramatically, usually costing around 1.5× more. Overall, we find that HINT models require much less compute to deal with longer instruction and few-shot data inputs than Tk-Instruct (Figure 3, right).
HINT models outperform a strong decoderonly baseline. HINT significantly outperforms GPT-2 and OPT-13B, in line with prior work that shows encoder-decoder models often significantly outperform even much larger decoder-only equivalents (Wang et al., 2022a; Iyer et al., 2022). In particular, 11B-size HINT outperforms OPT-13B by 5 points or more despite using a similar number
of FLOPs. This highlights the utility of improving efficiency for encoder-decoder-based models. We also note that caching key/value attention pairs, the simplest way to reduce inference costs with decoder-only models, scales worse than HINT. The size of cached key/value pairs for GPT-2 is ∝ lds, where l is the number of layers, d is the size of the model hidden dimension, and s is the cached sequence length. In contrast, the size of the saved PEFT parameters for HINT is ∝ sd + ld, which scales better with respect to sequence length (larger s) and model size (larger d, l)6.

heading: 5.2 P3
text: We report results on the T0 evaluation set in Table 2, with full results in Appendix B. We find that:
Our T0-3B replication significantly outperforms the results reported by Sanh et al. (2022). This matches prior suggestions that T0 is undertrained (Phang et al., 2022; Wu et al., 2022). We provide further details in Appendix E.
HINT outperforms hypernetwork baselines. The HINT model consistently outperforms Hypter, a prior hypernetwork-based approach, and learns to make use of the P3 prompts as evidenced by
6We provide details for these calculations in Appendix C.2.
its improved performance over a baseline model trained without prompts (‘No Prompt’).
HINT remains cheaper than T0 for inference. HINT uses significantly less flops than T0-3B, albeit with smaller savings compared to SNI, likely due to the different style of prompts: P3 prompts tend to be shorter, and interleave task inputs (e.g. ‘Does <sentence 1> entail <sentence 2>?’). Despite this, HINT still provides reasonable FLOPs savings. We suggest that the performance of HINT could be greatly improved by leveraging additional few-
shot information, further exploiting the efficiency of HINT models in encoding task data.
We investigate if HINT models can provide benefits even when the input and instruction are concatenated through training and evaluating in the ‘joint’ setting of P3, and find that HINT performs similarly to T0-3B with additional parameter-efficient modules, which suggests that the hypernetwork is unable to improve on the baseline model through additional customisation, and so is primarily useful as a mechanism for reducing inference costs and cheaply incorporating few-shot data.
HINT performs similarly to the baseline in few-shot settings. In Table 3, we show that HINT remains within 1 point performance of T0 in fewshot settings, despite the large reductions in FLOPs cost, using up to 6× fewer FLOPs. This makes HINT especially useful in few-shot scenarios.

heading: 6 Analysis

heading: 6.1 Pretraining
text: We compare using no pretraining, our pretraining scheme, and the pretraining scheme proposed by Phang et al. (2022) (‘CACLM’) in Table 4. As the pretraining scheme is primarily for improving the parameter generators, we evaluate its effect both
with and without using instruction fusion (‘HINT’ and ‘HINT - No Instr. Fus.’, respectively).
We find that: (a) using pretraining gives a large boost in performance for hypernetwork-only models, showing that pretraining is essential to good hypernetwork performance, and (b) using our pretraining scheme works best overall. We hypothesise this reflects the fact that our scheme is closer to the Super-Natural Instructions format than CACLM. Unlike Phang et al. (2022), we found that further pretraining did not aid performance. This is likely due to the fact that we tie the underlying model encoder and hypernetwork encoder weights together, meaning that the model weights must balance between acting as the hypernetwork and underlying model encoder.

heading: 6.2 Inference Speed
text: While HINT provides significant FLOPs reductions compared to baselines, these do not necessarily translate to real-world inference speedups. We examine this by measuring the average speed of HINT to process 100 samples of the same task, assuming the average input lengths given in Appendix A.1.
As seen in Figure 4, while baseline decoding remains faster for small input lengths on GPU7, it lags compared to HINT for longer sequences. In fact, HINT’s inference latency increases at a much slower rate compared to the baseline as the input size increases (with the number of shots), highlight-
7This is likely due to the small additional overhead of running the HyperEncoder, which must be run before the rest of the model.
ing that HINT is especially effective in few-shot scenarios and scenarios with lengthy inputs.

heading: 6.3 Architecture Ablations
text: We experiment with a series of ablations to determine the best architecture for HINT, and find that:
Adapters and prefixes work best together. We consider alternatives to using adapters and prefixes together: using adapters alone, using prefixes alone, and using LoRA (Hu et al., 2022) instead of either. In order to isolate the effect of these choices, we test without using instruction fusion. We find that adapters and prefixes provide the best overall performance, with prefixes-only and LoRA-only performance substantially worse, even when increasing the number of parameters generated. This suggests that our hypernetwork approach is more adept at generating certain types of PEFT modules.
PEFT and instruction fusion are complementary. We find that using just the generated parameter-efficient modules or the encoded instruction alone (‘-Instr. Fus.’ and ‘-PEFT Gen.’ in Table 6) perform significantly worse than using both methods together, suggesting that these methods provide complementary improvements.
Cross-Attention Layer wins over Full Decoder. We compare using a full T5 decoder (with selfattention removed) as the hypernetwork weight gen-
erator as in Phang et al. (2022) with our approach, and find that our single multi-head cross-attention layer performs better at a much cheaper cost than using the full decoder (‘+ Decoder’ in Table 6).

heading: 7 Conclusion
text: We introduce Hypernetworks for INstruction Tuning (HINT) models and show that they consistently outperform strong full-input baselines when controlling for inference compute. This is primarily due to the fact that HINT models process their task instructions once per task, while current state-ofthe-art models re-encode instructions with every task input. We show that the success of HINT models relies on a pretrained hypernetwork, which converts task instructions into parameter-efficient modules and an encoded instruction, both of which we insert into the underlying model.
Future work could investigate how HINT aids in few-shot settings, further building on HINT’s strong few-shot efficiency and taking advantage of the improved initialisation provided by hypernetworks (Phang et al., 2022). Overall, HINT models combine the benefits of parameter-efficient learning with the benefits of instruction-based learning, allowing one to easily turn pretrained language models into efficient, task-customised models.
Limitations
While promising, HINT comes with several drawbacks related to its ease of use. First, HINT takes advantage of the fact that (a) instructions are often long, and (b) often we want to perform inference over a larger (> 100) amount of examples with the same instruction. If either of these items are not true in a setup, then HINT is unlikely to provide a large benefit over simply including the instruction with the input text. This can be seen in the smaller compute savings provided by HINT for P3 in Table 2. Second, while HINT is computeefficient at inference time, it is far more costly to train, as it effectively requires running the underlying model together with the hypernetwork for every batch. This means that while HINT may be useful for practitioners with limited compute budgets, it may be difficult to train HINT models with the same limited budget. Finally, we train and test on English data only, and do not explore the generalisation of our approach to multilingual setups. Considering the success of hypernetworks in multilingual settings (Platanios et al., 2018; Baziotis
et al., 2022; Ustun et al., 2022), we believe this is a promising direction for future research. As such, while promising, HINT is limited by certain assumptions made about the length and format of instruction-augmented data, and we hope further improvements of the method work towards loosening these assumptions.
Ethics Statement
We believe that the impact of our work is largely beneficial, examining a novel method to make instruction-based models cheaper to use. This may aid in reducing the carbon footprint of large language models running in inference (Schwartz et al., 2019) and in making these models more accessible to people with limited compute budgets. However, we also note that our approach requires unsupervised pretraining on a large corpus, making it difficult to document exactly the data it has seen during training and making it likely to reflect problematic or even dangerous biases within the corpus (Bender et al., 2021). We believe that future research could investigate reducing the need for hypernetwork pretraining and further investigate the behaviour of hypernetwork-augmented language models.

heading: Acknowledgements
text: Research supported with Cloud TPUs from Google’s TPU Research Cloud (TRC). We thank AllenNLP members and Jonas Pfeiffer for encouraging and useful coversations in earlier stages of this project.

heading: A Dataset Details
text: A.1 Input Lengths When calculating FLOPs estimates, we use the median sequence length of the inputs and outputs to calculate inference costs. We compute the median over the train split of Super-Natural Instructions and over 10,000 random samples from the T0 train split of P3. We calculate the medians for each format separately, rather than adding the instance and instruction-only values together (hence the mismatch in values). We provide the calculated values in Table 7. We find that P3 inputs mostly consist of the instance, with prompt templates consisting of relatively few tokens, while SNI inputs consist mostly of instructions. This explains why HINT models are much cheaper than Tk-Instruct models, but not that much cheaper than T0 models, as HINT models reduce FLOPs by avoiding reprocessing the instruction with every input.
A.2 Split Sizes We report the sizes of splits here. For Super-Natural Instructions, we use the default setting from Wang et al. (2022b) where 100 examples are provided for each task in train and test splits. We also note that we follow the sampling procedure used by Sanh et al. (2022), where we “treat any dataset with over 500,000 examples as having 500,000 /
num templates examples” during training. Taking this sampling into account results in the much smaller dataset size seen in Table 8. We refer readers to Sanh et al. (2022) for more details on P3.

heading: B Full P3 Results
text: We report the full results of models on the P3 dataset from Table 2 in Table 9.

heading: C Model Compute Calculations
text: We provide a more thorough description of the compute and memory costs associated with various models we discuss here.
C.1 Compute Costs We will let i be the sample length, t be the task instruction length, o be the output sequence length, n be the number of same-task samples we wish to process, and N be the number of parameters in the model. We will assume we are only processing examples from the same task.
Tk-Instruct As Tk-Instruct concatenates instruction and sample together as input, it uses roughly Nn(i+ t+ o) FLOPs.
HINT The cost of the HINT model is more complicated. Let N ′ be the cost of the hypernetwork generator, and A be the cost of the parameterefficient modules inserted into the underlying model. The cost of running the hypernetwork is
t(N +N ′) (since the hypernetwork encoder is the same size as the underlying model). The cost of then running the underlying model with parameterefficient modules is n(N + A)(i + o). We sum these two terms to get the total cost of HINT: t(N +N ′)+n(N +A)(i+o). We do not consider the additional cost of inserting the instruction in the decoder as this only affects the few (usually 1-2) output tokens and the decoder cross-attention only, and so is negligible. We can simplify the HINT compute cost down by observing that in most cases N >> A and N >> N ′, resulting in the cost of HINT being roughly tN+nN(i+o). This simpler formulation highlights the main benefit of HINT: the instruction no longer is processed with every sample, and so compute cost is ∝ t+n as opposed to ∝ tn.
C.2 Memory Costs Here, we will let l be the number of layers, d the model hidden dimension, h the number of heads, k the size of the keys/values, and s be the length of the sequence we want to save. We ignore bias terms for simplicity.
Decoder-only Models If we want to cache the key/value pairs for a given sequence, we will store 2lhks values - a key and value for every head in every layer, for each item in the sequence. We note that typically kh = d in models, and so in the main text we simplify this to 2lds ∝ lds. HINT In the default HINT setup, we save three elements: the processed instruction sequence, which contains ds values (one vector per token); the adapter weights, 2 ∗ 512ld values (one adapter comprising of two weight matrices per layer, where each weight matrix has size 512 × d); the prefix values, 2 ∗ 30lhk values (a 30-length prefix and key per layer per head) ). This sums to give a total memory cost of ds+1024ld+60lhk. Note that in the default HINT settings, we use prefixes of length 30 and adapters with bottleneck size 512, but these settings could be adjusted to reduce memory costs. Applying the simplification kh = d, we get that the HINT memory cost is ∝ ds+ ld.

heading: D GPT-2 Instruction Finetuning
text: When finetuning GPT-2 for Table 1, we trained for [3, 5, 10] epochs with a batch size of 32. We use AdamW (Loshchilov and Hutter, 2019) and swept learning rates of [1× 10−5, 2× 10−5, 5× 10−5],
using a linear warmup and decay schedule with 1,000 steps of warmup. We report the highest overall results in Table 1. Following Iyer et al. (2022), we minimise the loss only over the target tokens (with EOS token added after the target answer), not the inputs, since these are always provided during test time. Note that we calculate the FLOPs used by GPT-2 during inference based on the estimates provided by Kaplan et al. (2020) that GPT-series models use 2N FLOPs per token, where N is the number of parameters in the model.

heading: E T0 Replication
text: During initial experiments, we replicated the T0 training in the t5x framework, using the same training set and mixing proportions as Sanh et al. (2022). We found that our replications performed significantly better than the reported T0 performance when trained for longer. We train 3B and 11B size models on the T0 training mixture for 20,000 steps using a batch size of 2048, a maximum input sequence length of 1024, a maximum output sequence length of 256, and the Adafactor optimizer with a constant learning rate of 0.001. We start from the T5 v1.1 + LM adaptation checkpoints and fully finetune the model. As seen in Table 10, our replications significantly outperform both T0 models, suggesting that T0 was undertrained. We also compare the variances in prompt performance in Figure 5.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations section after conclusion (non-numbered).
3 A2. Did you discuss any potential risks of your work? Ethics statement section after conclusion (non-numbered).
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and section 1 (introduction)
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Primarily section 4 and 5, where we discuss models trained and datasets used.
3 B1. Did you cite the creators of artifacts you used? Section 4
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? All artefacts used were created and shared for research purposes, which we use them for.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We use these artefacts only for the research in this work, for which they were intended.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We do not create any new datasets or significantly alter the datasets we use, and make use only of extremely popular existing research datasets. We refer to the papers introducing the data we use for these details.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Our work is focused on modelling, rather than data, and so we point readers to the papers proposing the datasets we use for this information.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4 and Appendix A (in particular, A.2).
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 4 explains the experimental setup and Section 5 reports the main experiments. Section 6 contains ablation experiments.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4 and Appendix C provide details.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4 and Appendix D (for GPT-2 experiment).
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4. Appendices B and E provides additional information on P3, and Appendix D on GPT-2 experiments.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Future work could investigate how HINT aids in few-shot settings, further building on HINT’s strong few-shot efficiency and taking advantage of the improved initialisation provided by hypernetworks (Phang et al. , 2022). Overall, HINT models combine the benefits of parameter-efficient learning with the benefits of instruction-based learning, allowing one to easily turn pretrained language models into efficient, task-customised models. Limitations
While promising, HINT comes with several drawbacks related to its ease of use. First, HINT takes advantage of the fact that (a) instructions are often long, and (b) often we want to perform inference over a larger (> 100) amount of examples with the same instruction. If either of these items are not true in a setup, then HINT is unlikely to provide a large benefit over simply including the instruction with the input text. This can be seen in the smaller compute savings provided by HINT for P3 in Table 2. Second, while HINT is computeefficient at inference time, it is far more costly to train, as it effectively requires running the underlying model together with the hypernetwork for every batch. This means that while HINT may be useful for practitioners with limited compute budgets, it may be difficult to train HINT models with the same limited budget. Finally, we train and test on English data only, and do not explore the generalisation of our approach to multilingual setups. Considering the success of hypernetworks in multilingual settings (Platanios et al. , 2018; Baziotis
et al. , 2022; Ustun et al. , 2022), we believe this is a promising direction for future research. As such, while promising, HINT is limited by certain assumptions made about the length and format of instruction-augmented data, and we hope further improvements of the method work towards loosening these assumptions