From Instance Training to Instruction Learning: Task Adapters Generation from Instructions
Authors: Huanxuan Liao
Shizhu He
Yao Xu
Yuanzhe Zhang
Yanchao Hao
Shengping Liu
Kang Liu
Jun Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: Hypernetwork
Generalization
Instruction Learning
Primary area: deep_learning_architectures
TL;DR: The paper introduces TAGI, a novel approach that enables large language models to learn from instructions rather than extensive data instances, significantly enhancing their adaptability and efficiency in real-world tasks.

Abstract: Large language models (LLMs) have acquired the ability to solve general tasks by utilizing instruction finetuning (IFT). However, IFT still relies heavily on instance training of extensive task data, which greatly limits the adaptability of LLMs to real-world scenarios where labeled task instances are scarce and broader task generalization becomes paramount. Contrary to LLMs, humans acquire skills and complete tasks not merely through repeated practice but also by understanding and following instructional guidelines. This paper is dedicated to simulating human learning to address the shortcomings of instance training, focusing on instruction learning to enhance cross-task generalization. Within this context, we introduce Task Adapters Generation from Instructions (TAGI), which automatically constructs the task-specific model in a parameter generation manner based on the given task instructions without retraining for unseen tasks.  Specifically, we utilize knowledge distillation to enhance the consistency between TAGI developed through Learning with Instruction and task-specific models developed through Training with Instance, by aligning the labels, output logits, and adapter parameters between them. TAGI is endowed with cross-task generalization capabilities through a two-stage training process that includes hypernetwork pretraining and finetuning. We evaluate TAGI on the Super-Natural Instructions and P3 datasets. The experimental results demonstrate that TAGI can match or even outperform traditional meta-trained models and other hypernetwork models, while significantly reducing computational requirements. Our code will be available at https://github.com/Xnhyacinth/TAGI.

From Instance Training to Instruction Learning: Task Adapters Generation from Instructions
Huanxuan Liao!.2, Shizhu Hel.2, Yao Xul.2, Yuanzhe Zhang Yanchao Hao Shengping Liu Kang Liu Jun Zhao 1.2
1 The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China 2 School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
3 Platform and Content Group, Tencent, Beijing, China Unisound, Beijing, China 200 @ @i..ac.en {shizhu.he, yao.xu, kliu, jzhao) @ nipr.ia.ac.en
Abstract
Large language models (LLMs) have acquired the ability solve general tasks by utilizing instruction finetuning (IFT). However, IFT still relies heavily on instance training of extensive task data, which greatly limits the adaptability of LLMs to real-world scenarios where labeled task instances are scarce and broader task generalization becomes paramount. Contrary to LLMs, humans acquire skills and complete tasks not merely through repeated practice but also by understand- ing and following instructional guidelines. This paper is dedicated to simulating human learning to address the shortcomings of instance training, focusing on in- struction learning to enhance cross-task generalization. Within this context, we introduce Task Adapters Generation from Instructions (TAGI), which automati- cally constructs the rask-specific model in parameter generation manner based the given task instructions without retraining for unseen tasks. Specifically, we utilize knowledge distillation to enhance the consistency between TAGI de- veloped through Learning with Instruction and task-specific models developed through Training with Instance, by aligning the labels, output logits, and adapter parameters between them. TAGI is endowed with cross-task generalization capabil- ities through a two-stage training process that includes hypernetwork pretraining and finetuning. We evaluate TAGI the Super-Nature Natural Instructions and P3 datasets. The experimental results demonstrate that TAGI can match or even out- perform traditional veta-trained models and other hypernetwork models, while significantly reducing computational requirements. Our code will be available at https: /ttl::///gttiie github...... com /tthyacittt////aceeeecccaiaataaa TAGI..
Introduction
Large language models (LLMs) have acquired the ability to solve general tasks by utilizing instruc- tion finetuning (IFT), which describes different tasks the same natural language format [3; 6: 23]. However, IFT still relies heavily instance training of extensive task data ((Description, [Demostra- tions), Source, Target)) [37; 39], which faces significant limitations in adapting LLMSs to real-world scenarios where labeled task instances scarce and broader task generalization becomes paramount.
Therefore, for better cross-task generalization, the "zero-shot" learning ability LLMs is crucial for real-world applications: models learned with instructions can achieve non-trivial performance unseen tasks with just a single instruction that provides a comprehensive description the task (e.g..
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

"You will be given sentences in which your task is to recognize the name of a person.") "). Traditionally. achieving this capability involves meta- veta-training the model by associating each imput with specific task instructions [21; 37J. For example, GPT-3 [25] has demonstrated strong "zero-shot" capabilities through meta-training. However, these methods heavily depend on the foundation model's abilities and are inefficient for various unseen tasks [22; 44], as they require reprocessing extensive task
instructions and some supplementary task data (e.g., examples from few-shot instances) for each
input (see the top of Figure 1).
In recent years, researchers have begun to ex- plore meta- leta-lea to enhance the cross-task generalization capabilities of LLMs, aiming to construct flexible, reusable and robust task- specific models [1; 34). For example, task- specific models such as Adapter [11], LoRA [12], and Prefix [17] have been constructed by hypernetwork [8]. This approach significantly enhances task generalization by processing in- structions efficiently, reducing redundant com- putations [27]. However, these methods heavily depend on a substantial corpus of training in- stances, which can hinder their capacity to effi- ciently learn and construct task-specific models based on provided instructions [13].
Pratning with festances
Exalestion
Tidk
Irgech
Paveur (p46ar
Ansi- PeppiPpey
Excriction
model (9,)
Be Aardy
Insc
Algenn
lippeement
14
(96)
Leanning Instraatioss
Figure 1: Comparison of the typical Training with Instance and the proposed Learning with Instruc- tion: The former involves training the model at the instance level with parameter updates, while the latter generates a task-specific adapter at the task level with parameter generation.
In fact, contrary to LLMs, humans acquire skills and complete tasks not only through repeated practice but also by understanding and following instructional guidelines [15]. For example, a tourist with basic knowledge of riding vehicles can easily learn to use new ones abroad for the first time with the help travel guides. This paper aims to mimic the way humans learn skills by understanding instructions. This shift represents modest evolution in task model construction, transitioning from traditional instance training models to a contemporary approach focused on instruction learning. By providing task instructions, the novel paradigm offers an automated solution for generating task-specific adapters and seamlessly integrating them into the base model. This approach aims to streamline the development of task-specific models while enhancing their ability to generalize across diverse tasks with instructions.
Guided by this goal, we introduce Task Adapters Generation from Instructions (TAGI), which converts instructions to task-specific adapters using hypernetork. Under the knowledge distillation framework [10; 36], we enable models to the "Learning with Instruction" paradigm in a manner analogous to the "Truining with Instance" paradigm. TAGI will enhance the alignment between the task-specific model 0% (acting the teacher) and the vanilla LLM du combined with the generated task adapters As (acting as the student) (see the bottom of Figure 1). This alignment is achieved not only through instance training but also by incorporating parameter learning for task- -seciffii models based on instructions. Specifically, we align the student under two distinct paradigms, encompassing not just the targets and logits, but also the adapters' parameters by L.2 regularization within instruction, which represents the enhancement of the understanding of instructions and the ability to generate more efficient task-specific adapters. Moreover, TAGI endows the model with task generalization capabilities through a two-stage training process: hypernetwor pretraining on standard text pretraining data (e.g.. C4 [29], followed by finetuning on meta-training tasks. This allows it to generalize effectively across unseen tasks without sacrificing performance.
We evaluate TAGI the Super-Naturaa Instructions (SNI) [37] and P3 [30] datasets. Experimental results demonstrate its ability to effectively generate adapters for unseen tasks, surpassing meta- trained models by 2% in SNI and 5% in P3, while significantly reducing computational demands by 60%, and outperforming other hypernetwork models by 7%. Notably, our method does not require additional parameter updating or gradient back pock-propagation, and it avoids the inefficiency of repeatedly encoding instructions during inference. We summarize our contributions as follows:
We propose a novel model construction paradigm by imitating human learning abilities, Learning with Instruction, for the cross-task generalization of the I.LMs. To the best of our knowledge, it is the first time that a task-specific model has been generated based on instruction learning, and its capabilities and parameters are distilled from teacher model trained instance learning.
2

---

We used knowledge distillation framework to develop task-specifer models within the instruction learning paradigm. By aligning model parameters comprehensively. the TAGI method improves the model's ability to understand instructions and solve unseen tasks more accurately and efficiently.
Comprehensive quantitative and qualitative assessments have highlighted the effectiveness of TAGI on two publicly available large-scale instruction datasets, with lower inference costs.
2 Related Work
TAGI draws inspiration from previous research on instruction following, hypernetworks and knowl- edge distillation. In this section, we will delve into the pioneering work in these areas.
Instruction Following is often used to evaluate the cross-task generalization of LLMs, and it is dedicated to handling any task described in natural language. Recent findings suggest that additional finetuning of LLMs with instructions substantially improves their zero-shot capabilities [6; 38; 39]. Moreover, large-scale multi-task meta-training has been shown to equip models with the ability to address new tasks in zero- or few-shot scenarios, facilitated by standard task formats and prompts [30; 44] alongside providing concise task instructions and select examples [24; 37]. However, the instructions and examples can significantly escalate the computational burden compared to task- specific models. Existing works attempt to mitigate this issue involved creating adapters to separately process instructions and examples [13; 42] with reduced performance. To overcome these limitations, we introduce new paradigm that draws on instruction-basee learning, simulating instance training to enhance the perception and processing capabilities of LLMs for handling unseen tasks.
Hypernetworks [8; 31] are neural networks that generate weights for other neural networks [4], which are designed to use fewer parameters dynamically build task-specific models [9; 33). Notable works such as Hyper Tuning [27]. HINT [13]. and Hypter [42] have all adopted hypernetworks to convert task instructions and demonstrations into adapters for LLMs. And MEND [5] utilizes hypernetworks to compress demonstrations for distilled vectors. Although they all avoided processing lengthy instructions repeatedly and utilized adapters make training and testing more cost-effective [19], they still have performance loss compared to meta- training [7]. The proposed method TAGI incorporates the utilization of hypernetworks, which are instrumental in generating task task-specific adapters that are seamlessly integrated into LLMs. Compared to existing models based on hypernetworks, TAGI not only trains at the instance level but also incorporates knowledge distillation to supervise the adapters generated by hypernetworks, thereby achieving both efficiency and effectiveness.
Knowledge Distillation is technique where smaller model (student) learns mimic the predictions of a larger model (teacher), aiming to retain performance while reducing computational resources [10]. Indeed, the application of knowledge distillation is the essential difference between the proposed method in this paper and other hypernetwork-
methods such as HINT [13] and Hypter [42].
Recently, some works [32] utilize knowledge distillation to finetune small language models such as
T5 [29], enabling them to act as LLMs with pre- prompting without any given prompts. Compared with the typical knowledge distillation methods of LLMs, the proposed method TAGI in this paper further utilizes model parameter alignment and aims to mimic another learning paradigm of human skill learning. We not only calculate the Kullback-Leibles (KL) divergence [14] between teacher and student models [10], but also compute the 1.2 regularization between the generated adapter by instruction learning and task-specific models by instance training.
3 Methods
3.1 Problem Setting
Cross-task Generalization: Given set of tasks T= (TI...,,TTil where each task T; contains set of (source, target) samples DD = {(11, tz), (sn, f.)). We categorize these tasks into three distinct non-overlapping groups for validating dfitt---iittiibution generalization: meta-trait (Ttrain), meta- valid (Tvatid). and meta-test (Trest), assuming all tasks adhere to a text-to-text format. For example, Teaain comprises tasks like translation and question answering, the Tealid and Trest encompass tasks such as paraphrasing and natural language inference respectively. Within the Terain, the goal is to utilize the data for training and transfer knowledge to facilitate learning to resolve the test tasks. For all methods discussed, aside from the original unsupervised pretraining of the language model
3

---

Question Antwering
Question
Ground Truth
Pregram
Oupuu
Exccution
Rellopeetec Loitds
Deonder X
LoRA Weights
Step 2: Hypernetwork Preiraining
Step
Tarae
lnstrctionn:
Student Distribution
Racoial Iesr
Sacode lenu
Linear
Instuction:
Iepus:
Ourpur Taggel
Outpat
Uncoder
Closs Attention
Step 3: Hy Hypernett Finctuning
Decader
(Distillation and Aligansent)
MelilHiud Amertii
Demoo Soures
+ Encoder
Indrnction:
Tencher Distrbbtton
Instruction
Cu (1)
Demostrations
Instruction
Inrpat
Imput
Trainable
Teucher: LLM with
Inslance Training
Task-specific Adapter
Studont: LLM with
HyperNetwork
Concar
Para of LaRA
Figure 2: Overview of TAGI. The hypernetwrk takes instruction as input and generates adapters subsequently integrated into the vanilla LLM, and constructed the task-specific model as student. After training the task models through instances on multiple basic tasks as teacher, TAGI constructs task-specific models by aligning the labels, output logits, and adapter parameters between teacher and student models. To improve compliance with task instructions and the efficacy of weight generation, TAGI undergoes two-stage hypernetwor training process: hypernetwork pretraining and finetuning. a-c are random divisions of the sampled sentences from pretraining datasets.
backbone on separate corpora, the model learning primarily takes place through multi-task training on the T Train.
3.2 Task Adapters Generation from Instructions (TAGI)
In this section, we will introduce the detailed method of TAGI. For each (unseen) task, TAGI consists of core components: a hypernetwork $ 3.2.1 which receives task instructions and generates purameter-efficiee adapters, and a specific model which combines the vanilla LLM and the generated adapters from hypernetoork.
Unlike traditional meta-training methods, we transition from training with instance to learning with instruction, which not only addresses efficiency issues the instance level but also incorporates parameter alignment for the task-specific model parameters at the instruction level. Specifically, the complete process is shown in Figure 2, we initially train the LoRA modules 3.2.2 on various upstream tasks (seen tasks) with task datasets of meta-train (Teroin). Specifically, for N distinct upstream tasks, we independently train N LoRA modules, with each module denoted as A, for task T, T. presumed to represent the optimal model for its respective task. Subsequently. TAGI is committed to building proprietary models for downstream tasks (unseen tasks). Its training process bifurcated into two primary phases: hypernetwork pretraining $ 3.2.3 and hypernetwork finetuning 3.2.4 which encompasses distillation and alignment.
3.2.1 Hypernetwork for Converting Instructions into LoRA
A pivotal element of our model is the hypernetwork that converts task instructions (descriptions and demonstrations) into a parameter-effiiien module. Our hypernetwork comprises two crucial components: the encoder, derived from the vanilla L.M2 designed to minimize encoding biases by converting task instructions into a continuous contextual representation. This representation is then fused with LLM input and concated with encoded input for the decoder. Additionally, the adapter generator, utilizing a basic MLP design [18], is both lightweight and efficient, effectively converting encoded instructions into parameter-efffcient modules.
We find that re-using the encoder from the vanilla LLM works well [13].

---

Encoder: Prior studies simply concatenated encoded instructions with inputs, overlooking the interactions between them. To address this, we integrated a hierarchical cross- coss-attention layer into the encoder of the LLM to refine the input representation with embedded instruction details. Specifically, for an input J and its corresponding task instruction ins we initially employ the encoder within the hypernetwork to encode the instruction into representations I, E Rexd Then, we feed the into the model and obtain the output representation S₁ from the self-attention sublayer in the (-th layer. Ultimately, S; is processed through the (-th cross-attention layer, resulting in text representation that is enriched with instruction information:
Fj CrossAAtentionnLayer(...
where CrossAttentionnLayee conducts multi-head attention on the query, key, and value matrices, followed by residual connection and layer normalization. The final input to the decoder is the concatenation of the encoded instruction and the encoded fusion input, i.e., (I2; Ft).
Considering the efficiency and effectiveness, we utilize a two-layer multi-layer perceptron (MLP) to generate parameter-efficient modules (e.g.. LoRA) for the encoded instruction. To differentiate between the query Q and value V matrices well as the layers, ids idx, (Q.V) E {O..... 2 #blocks} as positional information. We use a unique network for each layer and share it between Q and V (i.e., one network is used for a certain layer LoRA generation).
Adapter Generator:
introduce layer
LoRA, (2,V] LoRA(SM) MLP, idx ().V)
i = 21, idx) 21 + 1)
where LoRA, and LoRAY are the 1-th LoRA of Q and V, respectively.
3.2.2 LoRA Tuning for Task-specific Models
LoRA [12] efficiently reduces the number of trainable parameters by decomposing the update of the LLM's attention weight matrix (denoted as Wa E Rd* into low-rank matrices. Specifically, LoRA updates the weight matrix Wo oW Wo AB, with A Rddr and B E R being trainable low-rank matrices of rank r, significantly smaller in dimensions than d and k. We finetune robust baseline to derive the LoRA parameters A; for task-specific models for i-th task, facilitating LLM instruction learning and parameter alignment. SNI is categorized into 60 types based task types, while P3 encompasses 36 categories, corresponding to 60 and 36 parameter modules, respectively.
3.2.3 Hypernetwork Pretraining for Preliminary Generalization
Previous research [5; 27] has demonstrated that pretraining hypemeeworks can substantially improve the model's cross-task generalization capabilities. Adhering to the HINT [13), we pretrain the hypernetwork on C4 [29] before finetuning it on diverse multi-task prompt dataset. As illustrated in the right segment of Figure 2, given an input sequence, we partition it into randomly sized segments a, b, and c, where is fed into the hypernetwork, b into the LLM, and is the segment to predict. During this stage, training is conducted by minimizing the cross-entropy loss Epeed, aiming to ensure that the hypernetwork learns to recognize instructions enhance generalization ability.
pred
log (LLM ( (c) b)
(3)
3.2.4 Hypernetwork Finetuning for Instruction Learning
At this stage, TAGI is finetuned on multi-task prompt dataset, enabling it to learn the generation of optimal parameters from task instructions, thereby ensuring effective generalization to future unseen tasks. Similar to the pretraining phase, task instructions (alongside some few-shot samples) replace a, the main input replaces b, and the target replaces c. In each iteration, the hypernetwork generates LoRA parameters and encodes the instructions. LoRA is parameter-efffcieet module (i.e., inserting into the model), and the encoded instructions are integrated with the encoder's 's embeddings for information fusion and concatenated with the fused encoding input during decoding. Beyond the standard Cpred, we employ knowledge distillation for instruction learning: strong baseline combining complete task instructions and input, serves as the teacher, while the model incorporating generated LoRA parameters with the input, acts as the student. The KL divergence Lu measures the discrepancy in word probability distributions between the two models as an implicit learning outcome, and the MSE loss Lins calculates the difference between the generated parameters and
5

---

those of task-specific marameter-effiiiee modules as an explicit learning intermediate result. The formulation of finetuning is as follows:
Lis MSE( A, Hypermetoork(a)
(4)
Lu (L(PPuMMAnNE (a: 6)) || (LLMM-llrrreewo)((())))))))) (x (z]6)) b))
(5)
Limetune Lpeed Ai + Xzhins
(6)
where Tú A; is the optimal LoRA modules of the i-th task, a1 and X2 are the hyper-parametts parameter to control the importance of distillation in finetuning.
4 Experiments
We first present the datasets 4.1) and baselines ($ 4.2) used in our evaluation and then discuss three research questions (RQs):
RQI: Can the proposed instruction learning paradigm effectively learn the ability of instance training?
Can it support cross-task generalization LLMs? ($ 4.4)
RQ2: How many foundation tasks does TAGI need to learn to achieve better results? ($ 4.5)
RQ3: What is the impact of different modules and learning stages TAGI? ($ 4.7)
4.1 Datasets
To demonstrate the generality our method, we evaluate our approach on two popular multi-task instruction datasets': : Super-Natural Instructions (SNI) [37] and TO split of P3 (P3) [30].
SNI comprising over 1,600 task datasets, each dataset includes a task definition and set of fixed positive and negative demonstrations. We follow the previous research [13; 27] and examine two settings: only using the task definition as the input to the hypernetwork (*Def"), and using the definition along with two few-shot positive examples (*Def 2 Pos"). We only use the English tasks in the dataset and the model's generation is evaluated on a set of 119 unseen tasks using ROUGE-L.
P3 composed of 62 task datasets, the TO model is trained with these tasks divided into meta-training and meta-test sets. The format of the prompts takes into consideration 0-shot reasoning and typically includes instructions or possible answer options. We follow the precedent work [41] by using the TO training subset tasks to train our model. The evaluation is conducted based on the accuracy scores of multiple-choice questions for unseen 11 tasks in the meta-test set (MTestl
4.2 Baselines
We compare the characteristics of TAGI against eight primary groups of baselines (as shown in Table 1): I)
Table 1: Compare the characteristics of all comparison meth- ods and the proposed TAGI. More comparisons can be in C.I.
No FT: models without finetuning. 2) HyperTunnig [27]: models that use hypernetwork to convert demonstra- tions into adapters without instruction fusion. 3) Hypter [42]: models based on hypernetwork do not use pretrain- ing. 4) HINT [13]: models pretrain hypernetwork and concat instruction. 5) TO and Tk-Instruct: strong base-
Pre-
Instr. Fus.
Low Infer.
Instr. Leaming
Unscen
Method
Train
Cost
Task
Simple FT
X
TO [30] Ins-ntt (37]
x
Hypter [42]
X
x
x
HyperTuning [27]
x
X
HINT [13]
x
X
TAGI (Ours)
lines fully finetuned on P3 and SNI respectively with instruction concatenated, 6) Full FT: models fineuned on target tasks. 7) Decoder-only model: decoder-onke models fully finetuned like GPT-2 [28] and OPT [43]. 8) FiD-ICL [41]] ICL method use encoder intermediate fusion.
We provide the full list of datasets and more details in the A.2.

---

Table 2: RougeL results on Super-Natura Instructions. The best results are in bold, while the second-best are underlined. *, i means that those results are from HINT [13] and [27] respectively, "-" means not reported. $ indicates that there is no parameter alignment loss in the hypernetwork finetuning because the model too large, leading to significant amount of time required for LoRA tuning for each task. The Average Relative FLOPs cost is calculated relative to Tk- We use number of FLOPs required by each model to process one task (containing 100 examples).
Def (Zero-shot)
Def 2 Pos. (Few-shot)
Avg. Rel.
Method
Base (250M)
XL (3B)
XXL (11B)
Base (250M)
XL (3B)
XXL (11B)
FLOPs
No FT
8.8
14.3
26.2
9.4
13.6
30,5
x
Tk-Instruct!
35,3
48.0
53,6
42.1
54.0
62.0
x1.0
Decoder-only model"
GPT-2 XL (1.5B)'
38.2
45.3
x0.33
OPT (13B)'
44.8
51.5
x0.36
Hypereetwor-bban model
Hypter'
12.1
16.8
15.5
10.6
14.2
13.4
x0.35
HyperTumingg
38.9
48.6
x0.34
HINT
33.3
47.2
51.1
41.8
53.2
56.4
x0.37
TAGI (Ours)
35.3
48.4
52.3
42.5
56.3
58.4
x0.39
4.3
Implementations
We limit our scope to encoder-decoder models for our experiments*. We use T5-LM-Adapt" and TO [30] initializations in experiments. The model groups have the same architectural framework but differ in weight; TO uses 15-LM-Adapt for initialization and undergoes multi-task training the P3 meta-training set. For SNI, only T5-LM-Adapt is considered, and three different sizes are tested: Base (250M), XL (3B), and XXL (1 (I))), with the teacher model being TK-Instruct [37]. For P3, we experimented with two sets of models of three different sizes: Base (250M), Large (800M), and XL (3B) with the only template as input, while the teacher model being FiD-ICL [41] with 16-shot examples. The A.4 contains more implementation details and experimental settings.
4.4 Main Results
Super-Natural Instructions. We report the performance and inference costs of TAGI models and baselines in Table Our analysis and findings yield several key insights:
Firstly, methods lacking finetuning exhibit subpar performance. As shown in the first row of the table, the performance of No FT significantly lower than other baseline methods by approximately 30 points (except for Hypter), which underscores the critical role of inductive bias, introduced during meta-training, enhancing the model's ss instructional adherence and cross-task generalization.
Secondly, TAGI demonstrates notable improvements over other hypernetwork-Be -baed baselines, with only marginal increase in inference overhead (see Table 2 last column). We find that TAGI still outperforms the advanced method HINT (2 2 points) while achieving similar computational savings. This highlights the efficacy of instruction learning with knowledge distillation. The underperformance of HINT and Hypertuning may stem from their sole reliance on cross-entropy with the target during meta traininn, lacking explicit supervision of intermediate task specific module parameters and implicit supervision of the teacher outcome. This deficiency impedes their ability to fully leverage instruction tasks for generating superior adapter parameters during meta-test.
Thirdly, TAGI consistently matches or even surpasses robust baselines in both zero- and few-shot settings. Comparing TAGI with multi-task finetuning approaches such as Full FT and TK-Instruct, we observe that TAGI achieves comparable performance (0 2.3 points) except for 11B while utilizing approximately 2.5 fewer FLOPs. TAGI's performance somewhat lacking, potentially attributable to either insufficient training due to resource limitations or a decrement in performance stemming from the omission parameter alignment constraints due to time constraints In alignment with prior research, TAGI significantly surpasses GPT-2 and OPT-13B in comparative analyses with decoder-only models (2 10 points in GPT2 and 2 7 points
the 1BB model is
"We have discussed in detail the encoder-decoder and decoder-only models in B.1. 5htpps: /huggi a ace. co/ googl e e/t55x11 x1= adapt
aeg.cce....
We discuss the trend and possible reasons in B.2
7

---

Table 3: Average accuracy results over TO evaluation tasks after training on the TO P3 train set. means results are from [41]. trained by us followed the Ik-Instruct (meta-training) [37]. Our method uses only template inputs without demonstrations yet achieves competitive performance with ICL-based methods using 16 shots, with much-reduced inference overhead. The Average Relative Inference Time is calculated relative to the Metatrain. We use the inference time required by each model to process
11 test tasks with batch_size of
T5-LM
TO
Avg. Rel.
Method
Buse (250M)
Large (800M)
XL (3B)
Base (250M)
Large (80UM)
XL (3B)
Infer. Time
MTestll Avg.
Zero-shot
43.9
41.5
42.6
49.1
57.6
Full FT
44.6
45.5
47.2
51.9
61.4
Mctatrain 9
44.1
52.4
53.1
50.1
52.4
56.8
x1.0
ICCL-based method
Concat-ICL.
44.2
47.6
48.6
53.2
x4.1
RDD-ICL"
47.0
55.2
60.0
51.0
58.2
Ensemble-ICL
44.6
54.5
52.6
49.9
53.7
57.7
x3322
Hypernehworkk model
Hypter
56.2
IINT
60.3
TAGI (Ours)
45.6
54.7
58.9
50.8
53.8
58.8
x 0.88
HyperTS Avg. (Without SCloze dataset)
FID-ICL
46.9
55.8
60.6
51.7
$3.9
58.5
HyperTuning'
54.6
59.6
TAGI (Ours)
46.7
56.0
59.8
51.7
59.2
x 0.88
in OPT-13B), affirming the superiority of encoder-decoder models within similar leeanlea frameworks. Overall, TAGI fulfills its objective by enhancing cross-task generalization capabilities through instruction learning and striking an optimal balance between performance and efficiency.
P3. We report results on the TO evaluation set in Table 3, with full results in C.2.
Firstly, examining the ICL-based methods presented in the middle section, it is evident that all three ICL aggregation strategies achieve superior performance. This underscores the utility instructions and demonstrations in aiding LLMs. However, these methods require concatenating extensive demonstrations during both training and inference, which significantly increases computational demands and reduces efficiency (x2 x 13.2 inference time). In contrast, TAGI by leveraging solely task instructions one time, attains comparable or superior accuracy levels while significantly curtailing computational burdens (x 0.88). TAGI demonstrates slight disadvantage (merely 1.2 points) to FiD-ICL [41] on T5-LM, yet it outperforms other methods (2 point). For TO, it is only 1.5 points lower than Full FT and exceeds all ICL-based methods. Notably, TAGI does not require the 16 examples like the ICL-based method, nor does necessitate repeated processing of instructions like the baselines, significantly reducing inference overhead.
A comparison the first three lines of results indicates that for large or models, initializing with T5-LM outperforms TO. We hypothesize that the process of training T5-LM to transition into TO might result the dilution of world knowledge or the diminishment of certain specific capabilities, thereby attenuating the benefits derived from meta- training.. Conversely, for models of base size, TO serves as more effective initialization point.
Furthermore, TAGI outperforms competing hypernetwork models?. By comparing the last two columns, it is evident that the performance in 1 surpasses HINT and Hypertuning by 0.5 and 4.6 points respectively. Additionally, in the HyperTS evaluation, the performance exceeds Hypertuning by point. This aligns with prior findings, suggesting that instruction learning augments the hypernetoork's ss task comprehension and its capacity to generate task-specific adapters.
4.5 Varying Number of Meta-Trainin Training Tasks
fundamental component of methodology is incorporating parameter alignment in instruction learning, Consequently, it is imperative to examine the effect of varying the number of tasks on which
Beeaase HINT is designed for TPU and Hypertuning is not open-soureed, didn't 't calculate their inference time. However. based on SNI experiments, it can be inferred that the trend of time expenditure is consistent.

---

(6) Def Pos. Results
(b) Def Results
(c) P3 Results
343
Rougol.
Resgel
Number Metetrain Tisks
Number Melatraia
Number Motatraia Tosks
Figure 3: The performance of different numbers of meta-training tasks. The backbone model is T5-LM-Bass, all trained for 20,000 steps.
Dut- (75-/c)
Det (T5-ftaa)))
P1 (TT--BB
reccte
13.5
Delcte rabe Niaatia
ralc
3533
Noatica
40.4
40.2
02
-03
243
Perer Rec
Figure 4: The percentage of generated parameters (%) against performance (RougeL.). The backbone model is T5-LM-Base, all trained for 20,000 steps.
parameter alignment applied on outcomes and its influence on the generalization capabilities of LLMs. To this end, we conduct a comprehensive experimental analysis to compare the efficacy of instruction learning with parameter alignment across a spectrum of task quantities against instruction learning devoid of parameter alignment. Tasks are organized in descending order based on the number of datasets encompassed within each. Subsequently. predetermined number of tasks are sequentially selected for meta-training purposes. This approach allows us systematically evaluate the impact of parameter alignment on learning and generalization as the number of tasks varied.
From Figure 3, we find that, firstly, an increase in the number of tasks correlates with improved performance across all methods, suggesting that meta-training across broader array of tasks enhances the model's sufficient quantity of tasks for meta-training must be acknowledged. Secondly, it was observed that the TAGI model exhibits lower overall performance in the absence of parameter alignment for instruction learning, yet it demonstrates smaller relative standard deviation and less variability in performance in response to the number of tasks. This pattern aligns with the expected outcomes of instruction learning, highlighting the efficacy of our approach in bolstering the model's ability to adhere to task instructions and generate task-specific adapters.
cetrrcttionfollowinn capabilities. However, the practical limitations of sourcing
4.6 Parameter Size against Performance
We analyzed the proportion of generated parameter sizes relative to the total parameter size during the generation of various ranks, and compared this to the performance of the full meta-training fine-tuning method, as demonstrated in Figure 4 and Table 7. We can find that TAGI requires only about 10% 10 the parameters to outperform full meta-training fine-tuning which indicates that the limited parameters generated by the Hypernetwork serve as an optimal solution for task completion. The ability to adaptively construct models tailored to specific tasks removes the necessity for additional fine-tuning, underscoring TAGI's effectiveness and efficiency.

---

4.7 Ablation Study
To evaluate the significance of each component within the TAGI model, conducted series of experiments across two meta-task datascts utilizing the T5-LM-XL (3B) model. The results as depicted in the Table 4, high- light that the instructions fusion plays a pivotal role in enhancing model performance. This process fa- cilitates dynamic interaction between the input and the instructions, enriching the model's s input with additional contextual information, reminiscent of the substantial benefits observed with ICL. Moreover, pretraining emerges as critical phase, markedly improving the capabilities of models that have not undergone pretrain- ing, thereby significantly enhancing their proficiency in interpreting and executing task instructions. Further- more, the systematic removal of various components during the finetuning phase indicates a consistent decline in performance, underscoring the integral con- tribution of each component to the model's 's overall efficacy.
Table 4: Ablation study of TAGI model. All models utilized are T5-LM-XL (3B) and training for 20,000 steps. The P3 dataset was selected by the HyperTS evaluation.
Method
Def
Def 2Pos.
P3
TK-Instruct
48.0
$4.0
TK-Instruct-LoRA
47.5
54.6
TK- IK-Instruct--reet
42.6
54.2
Hypertuning
38.9
48.6
59.6
HINT
47.2
53.2
60.3
TAGI
48.4
56.3
60.6
Ablation Study
w/o pretraining
47.1
55.6
58.3
w/o Instr. Fus.
35.1
40.6
44.2
w/o Eav
47.6
55.4
59.8
w/o Cat
45.7
53.9
57.3
w/o Lins w/o Hypemetwork
47.5
55.2
59.4
43.8
50.7
Compared to meta-learning methods such as LoRA fine-tuning (rank=32) "TKsInstruct-LoRA" prefix fine-tuning (num_ caon__irtull_tttiii tokens=32) "TK-Instruct-prefii"" and full fine-tuning "Tk--Instruct', our TAGI method enhances task comprehension and utilization which achieved through hypernetwokk that dynamically generates adapter LoRA insertions into the LLM based on input, leads to better cross- task generalization capabilities. Notably, prefix fine-tuning excels in the Def + 2Pos scenario, likely due to its effective integration of information from positive examples. Conversely, the Def scenario performs less satisfactorily, indicating that instructions alone are insufficient for optimal results. Comparative analysis with other hypernetwork models reveals that TAGI's ablation performance remains robust, affirming the effectiveness of each step in bolstering TAGI's operational efficiency.
5 Conclusions
In this paper, we introduce an innovative method of instruction learning designed to emulate instance training. This approach enables the model to achieve specified tasks and learn from instructions on how to address a category problems. The proposed TAGI seamlessly integrates instruction into the input and processes the instruction simultaneously, thereby ensuring minimal inference overhead. Concurrently, we employ knowledge distillation framework facilitate instruction learning for distilling skills and aligning task-specific models. This allows the hypernetwork transform task instructions into an efficient module inserted into the LLMs, thereby boosting generalization perfor- mance. Remarkably, TAGI consistently equals or surpasses the efficacy of conventional meta-training approaches while requiring fewer FLOPs and obviating the need for additional model parameters updating gradient Pack-propagation. Future work will investigate pretraining techniques and develop superior instruction fusion methods to augment the hypernet- work's expressive capability, thereby enhancing the model's 's ability generalize unseen tasks. Moreover, future work will investigate various task type classifications and the generalization effects of cross-modal tasks in instruction learning.
potent hypernetwork
Acknowledgements
This work was supported by National Key R&D Program of China (No. 2022 1900) and the National Natural Science Foundation of China (No.62376270, No.62276264). This work was supported by the Youth Innovation Promotion Association CAS.
References
[1] Jonathan Baxter. Learning to Learn. Springer US, 1998.
10

---

[2] Christos Baziotis, Mikel Artetxe, James Cross, and Shruti Bhosale. Multilingual machine translation with hyper-adapiers, 2022.
[3] Tom Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[4] Vinod Kumar Chauhan, Jiandong Zhou, Ping Lu, Soheila Molaei, and David A. Clifton. brief review of hypernetworks in deep learning, 2023.
[5] Tong Chen, Qirun Dai, Zhijie Deng, and Dequan Wang. Demonstration distillation for efficient in-context learning, 2024.
[6]
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-finetuued language models, 2022.
[7] Budhaditya Deb, Guoqing Zheng, and Ahmed Hassan Awadallah. Boosting natural language generation from instructions with meta-learning, 2022.
[8] David Ha, Andrew Dai, and Quoe V. Le. Hypernetworks, 2016.
[9] Yun He, Huaixiu Steven Zheng, Yi Tay, Jai Gupta, Yu Du, Vamsi Aribandi, Zhe Zhao, YaGuang Li, Zhao Chen, Donald Metzler, Heng-Tze Cheng, and Ed H. Chi. Hyperprompt: Prompt-based lask-conditioning of transformers, 2022.
[10] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv: 1503. 053302531, 2015.
[11] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nlp, 2019.
[12] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[13] Hamish Ivison, Akshita Bhagia, Yizhong Wang, Hannaneh Hajishirzi, and Matthew Peters. Hint: Hypernetwork instruction tuning for efficient zero-shot generalisation. ACL, 2023,
[14] James M. Joyce. Bellback-Leibled Divergence, pages 720-722. Springer Berlin Heidelberg, Berlin, Heidelberg, 2011.
[15] Sharon Kim, Mahjabeen Raza, and Edward Seidman. Improving 21st-century teaching skills: The key to effective 21st-century learners. Springer US, 2019.
[16] Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Heike Adel and Shuming Shi, editors, Proceedings of the 202/ Conference on Empirical Methods in Natural Language Process- ing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
11

---

[17] Xiang Lisa Li and Percy Liang, Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli. editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the Ith International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 4582-4597, Online, August 2021. Association for Computational Linguistics.
[18] Huanxuan Liao, Shizhu He, Yao Xu, Yuanzhe Zhang, Kang Liu, Shengping Liu, and Jun Zhao. Imagination augmented generation: Learning to imagine richer context for question answering over large language models. arXiv preprint arXiv:2403.15220 2024.
[19] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficiet fine-tuning is better and cheaper than in-context learning, 2022.
[20] Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V. Le. Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning, 2023.
[21] Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. MetalCL: Learning de Marneffe, and Ivan Vladimir
to learn in context. In Marine Carpuat, Meza Ruiz, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2791-2809, Seattle, United States, July 2022. Association for Computational Linguistics.
[22] Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. Refram- ing instructional prompts to gptk's language, 2022.
[23] Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task general- ization via natural language crowdsourcing instructions. In ACL, 2022.
[24] Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task general- ization via natural language crowdsourcing instructions, 2022.
[25] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
[26] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach Vito, Martin Raison, Alykhan Tejani. Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: an imperative style, high-performance deep learning library. Curran Associates Inc., Red Hook, NY, USA, 2019.
[27] Jason Phang, Yi Mao, Pengcheng He, and Weizhu Chen. Hypertuning: Toward adapting large language models without back paok-proppgation, 2022.
[28] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. 2019.
[29] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67. 2020.
[30] Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization, 2022.
12

---

[31] Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4:131-139, 1992.
[32] Charlie Snell, Dan Klein, and Ruiqi Zhong. Learning by distilling context, 2022.
[33] Yi Tay, Zhe Zhao, Dara Bahri, Donald Metzler, and Da-Cheng Juan. Hypergrid transform- ers: Towards a single model for multiple tasks. In International Conference on Learning Representations, 2021.
[34] Sebastian Thrun and Lorien Y. Pratt. Learning to learn: Introduction and overview. In Learning to Learn, 1998.
[35] Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. What language model architecture and pretraining objective work best for sero generalization'. 2022.
[36] Wenhui Wang. Furu Wei, Li Dong, Hangbo Bao, Nan Yang. and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33:5776-5788, 2020.
[37] Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap. Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson,
Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. Super-Naturallnnstrettee declarative instructions on 1600+ NLP tasks. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085-5109, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
Generalization via
[38] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022.
[39] Orion Weller, Nicholas Lourie, Matt Gardner, and Matthew E. Peters. Learning from task descriptions. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1361-1375, Online, November 2020. Association for Computational Linguistics.
[40] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Qun Liu and David Schlangen, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, October 2020. Association for Computational Linguistics.
[41] Qinyuan Ye, Iz Beltagy, Matthew Peters, Xiang Ren, and Hannaneh Hajishirzi. FID-ICL: A fusion-in-decoder approach for efficient in-context learning. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki, editors, Proceedings the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 81 8158-8185, 85, Toronto, Canada, July 2023. Association for Computational Linguistics.
[42] Qinyuan Ye and Xiang Ren. Learning to generate tasc-specific adapters from task description, 2021.
[43] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
13

---

[44] Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. Adapting language models for zero-shot learning by meta-tuning on dataset and prompt collections. 2021.
[45] Ahmet Üstüm, Arianna Bisazza, Gosse Bouma, Gertjan van Noord, and Sebastian Ruder. Hyper- A unified hypernetwork for multi-task multilingual transfer, 2022.
A Experimantal Settings
A.1 Problem Setting
Meta-Training and Inference: Our methodology rigorously adheres to the protocol outlined in
MetalCL [21]. In the meta-train phase, we commence by selecting a task T from Tirains followed by the sampling of l support examples ((x.fe) .u ) and m query examples ((x!!) yy )) from the chosen task. The proposed hypernetwork is then adjusted to minimize the overall loss, focusing on generating a task model that can accurately predict the target sequences (e.g., answer) for source sequences (e.g. question). During the seet-ttes//nnference phase, for each novel task in Trest, we employ instructions to create the task-specific adapter, to optimize the model's performance across all query examples
{(r{ (9) (4) )-
Table 5: Number of samples in given splits for each dataset.
Dataset
Examples per Task
Train
Test
Super-Natural Instructions
100
75,417 00,897,454 290,000
11,810
P3
2,940,068
P3 (Sampling)
1000
2,940,068
A.2 Datasets
During the pretraining phase, utilized the C4 dataset [29], truncating each sequence to 1024 tokens. For the training phase, we employed Super-Natural Instructions (SNI) [37] and P3 datasets [30] for meta- veta-training and meta-test. For SNI, we adhered to the default settings [13; 37], which include 100 examples per task for both the training and test splits. For P3, we used the data and prompts provided by TO. All prompts related the meta-training tasks were included in the meta-training process, while the meta-test phase utilized evaluation prompts specified by TO [30]. We treated ANLI R1, R2, and R3 as three distinct tasks, resulting in 11 tasks for the original meta-test in P3 (Meta-Test-11), Due to resource constraints, we deviated from the sampling procedures of prior work, opting to sample 1000 examples per task for each prompt template. This approach yielded smaller dataset size, as detailed in Table 5. For further information on P3 refer to [30]. Additionally, to facilitate comparison with the Hypertuning method, we excluded the Story task from the evaluation since it was not included in the datasets for the HyperT5 evaluation.
A.3
Split Sizes for Varying Number of Meta-Meaaaaa Training Tasks
As shown in Table 11 and Table 12, we present a comprehensive list of the two datasets, including the number of tasks or templates contained in each task and the task divisions from $ 4.5 experiments. The divisions in the table are cumulative; thus, the second division includes both the first and the second divisions. For SNI, tasks were sorted in descending order based on the number of tasks they contained and then divided into specified sizes (6, 15, 30, 60). For P3, we selected a specified number of tasks (5, 10, 20, 36) based on the task classification in the original paper, which includes categories such as Muultipe-Chooce QA, Closed-Book QA, Summarization, Structure-Tor Tocccture-T-Teeee Text, Paraphrase Identification, Sentiment, Topic Classification, and Extractive QA.
We obtain all our data from huggingface datasets [16]. In the following, provide the dataset links:
Satur-Natural Instructions: natural-instruct.c
https:/ httts::/github..com com/ a /
* P3: https: http:////w.....tttttta/////eeeee
14

---

Additionally, the Super-Natura Instructions dataset (previously known as Natural Instructions-v2) has undergone some changes over time. In our experiments, we use the v2.6 version.
A.4 Implementations
Our inplementations are based on huggingface transformers v4.23.1 [40] using PyTorch v1.13.1 [26] and deepspeed v0.10.0. All experiments were conducted on A100 NVIDIA GPUs, each equipped with 80GB of memory, and eight A6000 NVIDIA GPUs with 48GB of memory. Unless otherwise specified, the rank of LoRA generated by the hypernetwork 32, and we use the Adamw optimizer with learning rate of 5e-5 and linear warmup rate of 0.02. We pre-train all models for 50,000 steps using C4 [29] with a batch size of samples and sequences of length 1024.
A.5
TO-Base/Large/3B
TO [30] provides model checkpoints only in sizes 3B and 11B. Additionally, HINT [13] and FiD-ICL [41] re-pretrained TO and found that the model was not sufficiently trained, achieving better results after reproduction. Therefore, we used the TO model reproduced by FiD-ICL to conduct series of experiments.
Table 6: Hyperparameters for Training TAGI Models and LoRA Tuning.
Finetuning
SNI
P3
LoRA Tuning
Pretraining
Base (250M)
XI.(3B)
XXL (IIB)
Base (250M)
Large (800M)
XI,3B)
Input
1024
1024
1024
1024
512
Max Output Len
128
128
128
128
64
64
64
Optimizer
alamw
adamw
Learning Rate
le-4
le-3
le-4
5e-5
Se-5
le-4
le-4
Se-5
precision
floa132
bil6
bf16
bf16
# Training Steps
10000
50000
20000
20000
20000
20000
20000
20000
Warmup Steps
training steps
Ratch Size
8
8
2
4
2
Gradient Accumulation
LoRA Rank
32
A.6 Hyperparameter
The complete stable hyperparameter set used for training runs can be found in Table 6.
B Additional Experiments and Findings
B.1
Why we choose Enc-Dec Models?
Previous work has suggested that models with an encoder-decoder (enc-dec) structure have advantages over decoder-only (dec-only) models in terms of task generalization and instruction instruction-oolloiing capabilities [20; 35; 41]. Therefore, in our experiments, we only considered models with an enc-dec structure (T5-LM and TO). Our experimental results demonstrated that enc-dec models indeed have an advantage when compared, although dec-only models might have higher computational efficiency due to their ability to cache KV and have fewer layers. However, our method, TAGI, significantly improves performance in various aspects with only slight increase in computational overhead. We encode the task instructions only once based on the original computation.
B.2 T5-LM-XXL Training Trend
In this section, we detail how the performance of the T5-LM-XXL (11B) model surpasses the hypernetwork models but falls short of the meta-trained strong baseline In- by 1-4 points, as mentioned earlier in $ 4.4. The primary reason insufficient training; when replicating the Tk-Instruct experiment, our results
significantly lower than reported when finetuning for only
hnttps://gi thub. com/mi wrwsoft crospf//eeepeeeee co/
15

---

T5-LM-XXL (1IB) Performance w/o Alignment
M--LM-XXL (II Performance in Def 2 Pos.
Def
14
51-
10000
15000
20000
25000
30
Training Steps
Number of Tacks
Figure 5: Analysis of T5-LM-XXL (1IB).
20,000 steps. Consequently, we analyzed the performance of our TAGI model at different finetuning steps. As shown on the left side of Figure 5, performance steadily improves with more steps with substantial growth. Thus, we reasonably predict that increasing the steps to 50,000 or more could surpass Tk-Instruct. Another possible reason the lack of parameter alignment for the 1 1IB model due to limited resources. Our previous analysis has shown that parameter alignment is crucial, with larger models benefiting more. Therefore, we analyzed performance with small number tasks for parameter aligmment. As shown the right side of Figure 5, performance with parameter alignment for 6 and 15 tasks is better than without alignment. Based on these trends, it can be inferred that performance with full task parameter alignment could surpass Tk- Tk-Instruct.
B.3 Analysis on Hyperparameters
To explore the optimal hyperparameter settings for experiments, conducted series of tests and error analyses using the T5-LM-Base (800M) model. The findings presented in Table 7 reveal that variations in hyperparameters can lead to performance fluctuations, particularly with higher learning rates or reduced finetuning steps. Given the varying pre-training conditions of models of different sizes, size-specific analysis is essential; however, details larger models are omitted here due to resource limitations.
We observed that different settings of LoRA minimally affect performance, leading to select balanced size of 32. Similarly, the impact of the warmup ratio negligible; thus, based on our experience, we chose warmup ratio of one percent of the maximum finetuning steps. While more finetuning steps generally correlate with improved performance, excessive finetuning can result in overfitting meta-training tasks, thereby diminishing generalizability. Moreover, increased finetuning steps require greater computational resources. Consequently, we determined that the optimal number of finetuning steps is 20,000 based experimental outcomes.
B.4 How A and A2 are tuned?
In the experiment, set A and ' to two different values: A 5 and Xe sigmoid(Ciss). The effects of these different values on the results are illustrated in Figure and Table 8. We maintained all other conditions constant and only varied to perform ablation experiment at Def+2Pos. scenario.
B.5 Inference Cost
To analyze the computational efficiency of the TAGI model compared to the standard instruction training model (full fine-tuning), let's consider scenario where have to process samples, each length i, along with a task instruction of length t. We assume the output sequence length is negligible and thus ignore it in our computations.
In a typical full fine-tuning setup, such as Tn-Instruct each input is concatenated with the task instruction, requiring the model to process the combined input sequence. If we denote the number
16

---

Table 7: Performance variation due to different hyperparameters. The base model is T5-LM- Basee and all experiments follow the previous hyperparameter settings, changing only the target hyperparameter, where underlines indicate experimental defaults.
Learning Rate
LoRA Rank
Training Steps
Warmup Ratio
Method
5c-5
le-4
3c-4
le-3
16
32
64
15000
20000
25000
0.01
0.02
0.03
SNI
Def + Pos.
Tk-Instruct
41.3
41.8
42.2
38.9
41.4
41.8
42.1
41.5
41.8
40.6
TAGI (Ours)
42.1
42.5
40.3
39.7
41.8
42.5
42.3
41.8
42.5
42.4
42.3
42.5
41.9
Def
Tk-Instruct [37]
35.0
34.2
32.6
31.7
34.4
34.2
34.5
35.0
34.2
34.3
TAGI (Ours)
34.3
35.3
33.5
31.8
34.8
35.3
35.4
34.2
35.3
35.4
34.8
35.3
34.9
P3
M Arg.
Metatrain
43.3
44.1
43.6
40.9
44.0
44.1
44.3
44.2
44.1
43.6
TAGI (Ours)
44.0
45.6
44.0
41.6
44.8
45.6
45.5
44.3
45.6
45.2
45.1
45.6
44.8
Rougel,
42.5
Apsionoddt__ (irs)
43.00
41:55
43.3
1.50
4. 0031
40.
1.
43.55
10.0
41.00
41.25
30.0
10
30.7)
43.21
30
02
0.3
34
0.5
3.6
0.7
2.0
0.5 1.0
Figure 6: Ablation study on À hyperparameters. The backbone model is T5-Base.
of FLOPs required to process single token with encoder-decoder model as N, where N is the total number of model parameters, then the total computation cost for all samples can be estimated FLOPSsuanded N ntt + i) Here, each of the samples includes both the instruction and the sample input, leading to ra(1 + i) tokens being processed.
Our TAGI model, on the other hand, processes the task instruction only once, regardless of the number of samples. This unique feature significantly reduces the computation required, especially as the number of samples or the length of the instruction increases. The total computation cost in this model is given by: FLOPSTAGI = N (t + ni) In this case, the instruction length t processed only once, and each sample is processed separately, resulting in a total of (t + ni) tokens being processed.
C Extended Results
C.1 Characteristics Comparison of the Proposed TAGI and Other Baselines
Here, we report a full comparison of methods and the proposed TAGI in Table also visualized in Table 1. In this report, we compare various methods across eight dimensions. Finetuning on target tasks yields good performance; however, it necessitates retuning when applied unseen tasks and fails to address these effectively. Strong baseline meta-training methods excel at handling unseen tasks by enabling models to solve problems based on task-specific instructions. Nevertheless, these methods are limited to instance-level operations and entail repetitive processing of concatenated instructions and comprehensive finetuning, resulting significant parameter updates and high inference costs.
17

---

Table 8: Ablation study on À hyperparameters. The backhone model is T5-Base.
X
RougeL
0.5
sigmoid Lins
40.1
2
sigmoid( Lims)
40.9
sigmoid( Lins)
42.5
10
sigmoid( (iins)
38.7
5
0.2
41.3
5
0.5
41.6
5
1.0
41.2
Hypter [42] initially introduced the approach of considering tasks at the task level, treating identical tasks as a unified entity, and employing a hypernetwork generate adapters that represent specific task models from instructions. Building on this, Hypertuning [27] uses demonstrations to generate adapters and pretrains the hypernetwork to boost its expressive capabilities. Both strategies avoid the direct input of instructions and rely hypernetwork, which reduces parameter updates and lowers computational demands during inference. However, they suffer from notable performance degradation due to the lack of instructional information in the input.
HINT [13] addresses this issue by appending instructions post-encoder, thus eliminating redundant computations. Although these methods facilitate learning at the task level, they do not engage in instruction-uel based learning. i.e., they do not explicitly supervise the hypernetwork's generation process to aid in understanding instructions and generating parameters.
The proposed TAGI rectifies these deficiencies by integrating cross-attention for enhanced infor- mation fusion and supervised learning of adapter weights within HINT. This innovation aids in generalizing unseen tasks without increasing the computational burden.
Table Compare the characteristics of comparison methods and the proposed TAGI.
Meta- Train
Pre- Train
Instr. Concat
Instr.
Low Up.
Low Infer. Cost
Instr. Learning
Unseen
Method
Fus.
Params
Task
Simple FT'
X
X
X
[30] TK-Insre [37]
X
X
X
Hypter [42]
X
X
XXX
V
HyperTuning [27]
V
X
X
HINT [13]
TAGI (Ours)
C.2 P3 Full Results
Table 10 reports the per-task performance and average accuracy on P3 reported in Table 3.
D Limitations
Large Language Models. Due to computational constraints, most of our experiments were conducted using models with < 3B parameters. Given the complexity our research, we restricted our focus to encoder-decoder models, which have demonstrated superior performance in cross-task generalization [35], which we explore further in B. Consequently, remains uncertain whether instruction learning can be effectively scaled to larger models (> 7B parameters) commonly used decoder-only models. However, since our method preserves the original model parameters without compromising performance, anticipate its applicability to broader research in the future.
Training Costs. Although TAGI is computationally efficient during inference, training cost significantly higher. This is due to the additional requirements beyond the foundation laid by previous work, including the introduction of knowledge distillation, running hypernetwork to generate adapters for each batch, and pre- pre-training some downstream task-specific models. Consequently, while
18

---

Table 10: Main Full P3 Results. "-" means not reported. and $ mean the results are from FiD-ICL [41] and Hypertuning [27] respectively. & Computed as the average R1/R2/R3 (except for HyperT's Tows where the numbers are quoted), More ICL-based results and details can be seen FiD-ICL [41].
Method
ANLI o
(R2)
(R3)
HSwwg
CB
COPA
RTE
WIC
WSC
WGD
MTeesll
Avg.
HyperTS
Avg.
Random
33.4
33.4
33.4
33.4
25.0
50.0
5000
52.7
50.0
63.5
50.0
56.0
44,7
46.8
# Base(250M)
T5-LM
49.7
57.9
43,9
TS-LM FTll
33,8
34.5
33.4
33.5
24.8
66.5
45.7
51.1
53.7
46.3
49.8
50.9
44.6
46.5
29.5
50.2
58.4
47.4
T5-LM-RDD
33.0
32.4
33.1
33.4
26.7
42.5
58.8
54.6
51.1
57.9
50.3
76.3
47.0
46.9
TS-LM-TAGI
32.1
31.5
31.7
33.1
25.0
44.5
54.7
53.7
52.3
60.5
50.8
64.0
45.6
46.7
32.4
33.1
26.5
09.3
51.6
56.7
49.1
TO FT
33.5
32.6
33.9
33.9
29.1
73.2
66.3
68.0
53.1
50.9
51.0
79.0
51.9
53.1
Metatrain
32.1
31.5
31.5
29.5
50.4
47.7
61.6
52.8
50.1
TO-FDD
32.7
31.7
32.9
33.6
26.2
54,9
68.2
68.1
51,9
60.3
51.3
82.3
51.0
51.7
TO-TAGI
32.7
31.1
31.9
35.0
29.8
49.3
67.1
70.0
49.0
61.2
54.4
79.6
50.8
51.7
(orge(S00MM)
TS-LM
32.7
32.1
33.4
32.7
25.3
33.8
50.5
49.0
51.0
50.4
50.5
47.8
41.5
42.9
TS-LM Full FT
34.1
35.1
33.6
33.6
26.1
65.4
47.1
51.7
53.5
47.5
49.9
56.5
45.5
46.9
TS-LM Metatrain
31.3
30.0
30.5
33.4
27.0
60.4
77.6
71.9
47.0
56.4
54,8
87.2
52.4
53.3
TS-LM-FID
34.4
33.9
33.4
35.8
28.3
60.2
81.1
72.6
50.7
63.7
55.6
91.6
55.2
55,8
TS-LM-TAGI
33.7
33.5
32.5
35.1
27.8
62.9
79.0
52.9
57.9
58.2
86.2
54.7
56.0
TO
34.1
32.2
34.2
36.0
26.1
56.8
76.6
65.3
50.8
56.4
53.9
88.4
52.4
52.5
54.1
TO Metatrain
32.9
31.5
31.8
35.5
24.5
59.4
77.0
65.1
48.8
56.7
57.6
88.0
$2.4
52.8
TO-FD
33.4
31.8
32.8
35.7
26.1
60.7
77.6
67.1
52.1
59.1
54.7
89.5
53.4
53.9
TO-TAGI
32.7
31.5
32.9
27.3
48.2
59.9
56.4
89.4
53.8
54.6
HyperIS-Prefix
33.4
32.3
60.1
73.9
71.5
51.1
63.0
51.1
54.6
HyperTS-LoRA
33.0
52.0
A X2238)
TS-LM
32.7
32.2
33.4
32.7
24.5
32.7
53.1
48.8
50.8
57.6
50.9
51.4
42.6
43.9
TS-LM FT
34.6
34.3
67.8
53.7
47.7
63.3
47.2
TS-LM Metatrain
32.7
31.5
32.3
34.3
33.3
39.5
74.8
69.5
52.6
53.8
54.2
88.4
53.1
53.8
TS-I
37.6
40.4
78.8
50.4
96.5
TS-LM-TAGI
37.7
37.8
36.1
39.3
32.0
68.2
89.4
76.6
53.6
61,2
59.6
94.2
58.9
59.8
38.0
38.4
35.7
40.0
26.5
67.7
82.2
80.1
53.5
57.3
57.8
94.0
57.6
57.9
Full FT"
61.4
TO Metatrain
37.0
37.3
33.2
40.4
24.8
66.9
81.9
78.9
52.7
60.2
55.6
92.8
56.8
57,3
36.5
52,1
58.2
TO-TAGI
38.7
39.5
35.6
41.0
26.5
68.7
87.8
78.2
52.2
61.8
39.8
95.6
58.8
59.2
HyperTS-Preee Prefffx
38.7
33.6
69.6
88.4
79.5
53.1
57.6
56.6
59.6
LoRAA
35.3
30.8
66.4
83.3
68.5
50.3
60.0
56.1
56.4
TAGI may be highly efficient for inference and suitable for users with limited resources, training unique TAGI model presents considerable challenges.
Datasets. In the SNI study, our investigation was limited to tasks in English, leaving the generalization capabilities in multilingual context unexplored. However, given the proven effectiveness of hypernetwork methods in achieving multilingual generalization [2; 45], we are optimistic about the potential directions for our future research in this domain. Furthermore, in P3, we adopted the methodologies of TO [30] and FiD-ICL [41], concentrating primarily natural language processing (NLP) tasks amenable to ranking classification. This focus included tasks related classification and multiple-choice questions but excluded other types of generative tasks. Looking ahead, we aim to develop new research resources and broaden our experimental scope and evaluations to encompass a more diverse array of categories.
19

---

Table 11: Meta- Meta-Train dataset of Super-Natura Table 12: P3 dataset tasks. means evaluation Instructions.
without story_cloze _cloze.
Task
Num of Task
Task
# Num of Prompts
First Split (6 Tasks)
Meta- Train
Question Answering
157
First Split (5 Tasks)
Program Exccution
90
Question Generation
51
cosmos qa
13
Sentiment Analysis
42
kilt_taskshottpotaa kilt_tasks_ hotpotta
5
Misc.
36
amazon pollriit
9
Toxic Language Detection
32
Lun_dailymail3.000
9
Sccond Split (15 Tasks)
common gen
9
Text Categorization
28
Second Split (10 Tasks)
Commonsense Classification
glue_mrpe __rp
7
Text Matching
adversarial qa dbert
5
Named Entity Recognition
ag_news
7
Information Extraction
17
dream
5
Wrong Candidate Generation
15
gigaword
Text Completion
14
Third Split (20 Tasks)
Question Understanding
13
Text to Code
12
paws
12
Third Split (30 Tasks)
wiki_qa
11
Summarization
12
ropes
12
Dialogue Generation
11
quoref
11
Word Semantics
10
dbpedia 14
4
Story Composition
multi_news
Speaker Identification
imdb
Pos Tagging
quail
Linguistic Probing
quartz
8
Fill in The Blank
8
wiki bio
5
Text Quality Evaluation
Fourth Split (36 Tasks)
Stereotype Detection
7
adversarial qa dbidaf
5
Sentence Composition
7
adversarial_gee patrrarrialgg_ddbbertt drobera duore_ Selfe_SSeCCC
5
Negotiation Strategy Detection
Gender Classification
9
Coherence Classification
duore ParaphraseRC
9
Word Relation Classification
cos_e_vl.11
11
Fourth Split (60 Tasks)
qasc
8
Explanation
sciq
5
Text Simplification
glue_qqp
6
Sentence Perturbation
4
social i
6
Paraphrasing
4
wiki hiki_hop_oo ooigonai wiqa
9
Mathematics
4
8
Intent Identification
app app_reviews rotten_ yelp_review full samsum
4
Dialogue State Tracking
10
Code to Text
7
Sentence Ordering
7
Fact Verification
Answer Verification
Xsum
10
Translation
Meta- Test
Style Transfer
2
super glue wic.lixed
Stance Detection
2
winogrande_
sriggrand_wwnograaddd...
Speaker Relation Classification
2
syper_glue_ch super_glue_rte anli(r1/1243)
Question Decomposition
2
Number Conversion
Trony Detection
super_gher super_glue_coyy
Grammar Error Detection
hellaswag
Spelling Error Detection
super_glue_wis _ii
Spam Classification
story_cloze cloze
Sentence Expansion
Sentence Compression
Punctuation Error Detection
Preposition Prediction
Poem Generation
Entity Relation Classification
Entity Generation
Discourse Relation Classification
Discourse Connective Identification
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We can find the limitations in D.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We can reproduce the main experimental results following our settings in and 4.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification: We'll open source the code to an anonymous site https:://// h 4open. science/eeeeeeececccccce E/TAGI and put on github after review.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https:/ /ttp///niis.... cc/
public /wwlc//gu/s///eeeee Code Suiomi ss missooloicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips, httppppllllcggggll
CooeeooSSuuii Submi ssi PisiooPolli oolycy for more details. The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We can find the experimental settings (hyperparameters and datasets) in 4.3 and A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We examined the effect of different hyperparameters on results in B.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
23

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We can find in 4.3 and A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //nuriiisss cc httpp/ccppppllcccccccceotoooo del ines?
Answer: [Yes]
Justification:
All of our studies follow the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We follow their open-source protocols in all our uses.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers aw! thcode com da set. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "lEZjUH2GyI", "reviewer": "Reviewer_z6rG", "summary": "The authors introduce Task Adapters Generation from Instructions (TAGI), which converts instructions into task-specific adapters using a hypernetwork. They employ the Knowledge Distillation framework and a two-stage training process: first, hypernetwork pretraining on standard text pretraining data, followed by fine-tuning on meta-training tasks. They evaluate the proposed method on the Super-Natural Instructions and P3 datasets, demonstrating that TAGI can effectively generate adapters for unseen tasks.", "questions": "- Please explain how $\\lambda_1$ and $\\lambda_2$ were tuned.\n- Could you provide more details on the process of updating LoRA parameters via $\\mathcal{L}_{ins}$?\n- How were the tasks categorized? Please specify the criteria used.", "limitations": "Please refer to the Weakness section.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The limitations of existing methods were clearly explained.\n- The writing was clear, making it easy to grasp the proposed methodology.\n- The concept figure was intricate and facilitated easy understanding.\n- An ablation study was conducted, demonstrating the validity of the proposed methodology's elements.", "weaknesses": "- Following a method that assigns indices to the LoRA layer, the flexibility of the LoRA architecture structure is reduced.\n- This methodology seems limited to encoder-decoder structured LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer z6rG:\n\nWe greatly appreciate your professional review of our article. Here is our feedback.\n\n### Response to W1:\n\n> W1. Following a method that assigns indices to the LoRA layer, the flexibility of the LoRA architecture structure is reduced.\n\nOur method dynamically generates LoRA weights, offering the same level of flexibility as the LoRA approach. **We can customize the rank of the LoRA as well as select which modules require the addition of LoRA weights**. Furthermore, our method can be extended to generate parameters for other efficient fine-tuning modules.\n\n### Response to W2:\n\n> W2. This methodology seems limited to encoder-decoder structured LLMs.\n\nThank you for your insightful comments, which align with our own concerns. In Appendix B.1, we explained our rationale for selecting Encoder-Decoder models.  Encoder-decoder (enc-dec) structure has advantages over decoder-only (dec-only) models regarding task generalization and instruction-following capabilities. \n\nWe also recognize the potential of current decoder-only large models (such as Llama3, Qwen2) that have undergone extensive fine-tuning on instruction tasks, potentially serving as a strong baseline similar to TK-Instruct. However, the effectiveness of our method on these models remains to be determined through further experimental evaluation.\n\n### Response to Q1:\n\n> Q1. Please explain how $\\lambda_1$ and $\\lambda_2$ were tuned.\n\nIn the experiment, we set $\\lambda_1$ to two different values: $\\lambda_1 = 5$ and $\\lambda_2 = \\text{sigmoid}(L_{\\text{ins}})$. The effects of these different $\\lambda$ values on the results are illustrated below. We will include this supplementary experiment in the appendix in the near future.\n\n|                      |   $\\lambda_1$   |  $\\lambda_2$    | RougeL |\n| -------------------- | ---- | ---- | -------------------- |\n| Def + 2POS (T5-Base) | 0.5         |     $sigmoid(L_{ins} )$                | 40.1 |\n|         | 2 |  $sigmoid(L_{ins} )$| 40.9 |\n|                      | 5 | $sigmoid(L_{ins} )$ | 42.5 |\n| | 10 |  $sigmoid(L_{ins} )$| 38.7 |\n| | 5 | 0.2                 | 41.3 |\n| | 5 | 0.5 | 41.6 |\n| | 5 | 1.0 | 41.2 |\n\n**The global author rebuttal Q2** has updated the images, where you can further examine the trends.\n\n### Response to Q2:\n\n> Q2. Could you provide more details on the process of updating LoRA parameters via $L_{ins}$?\n\nCertainly, as detailed in Equation 4 of the main text, we calculate **the Mean Squared Error (MSE) loss between the LoRA parameters generated by the hypernetwork and those of the specific task model that have been fine-tuned using LoRA**. As outlined in Section 3.2.2, similar to LoraHub, we fine-tune specific task models during meta-training with LoRA and save the LoRA parameters. During the training process, the hypernetwork generates LoRA parameters based on the provided instructions and examples during the forward pass. We then compute the MSE loss between the LoRA parameters $\\Delta$ of the specific task model corresponding to the instruction and the parameters generated by the hypernetwork. This loss is subsequently backpropagated to update the hypernetwork.\n\n### Response to Q3:\n\n> Q3. How were the tasks categorized? Please specify the criteria used.\n> \n\nFor the SNI dataset, we adhere to the original methodology and dataset setup for pure English tasks as established by the authors [1], with the number of meta-training tasks set to 756. The major task categories and the number of associated subtasks are detailed in Appendix Table 10. The meta-test set comprises 119 previously unseen tasks. For the P3 dataset, we follow the task division approach outlined in the original T0 work[2], separating it into meta-training and meta-testing sets. The specific divisions are presented in Appendix Table 11.\n\n[1] SUPER-NATURALINSTRUCTIONS: Generalization via Declarative Instructions on 1600+ NLP Tasks, EMNLP 2022\n\n[2] MULTITASK PROMPTED TRAINING ENABLES ZERO-SHOT TASK GENERALIZATION, ICLR 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IyFhWzS5Cu", "reviewer": "Reviewer_UQYV", "summary": "This work proposes a new learning paradigm to train large language models (LLMs) for its better task adaptation and generalization ability. Specifically, they propose the method called TAGI, which follows a two-staged teacher-student fashion by firstly learning a set of task-specific LORA weights and then utilizing the outputs from these to align and meta-learn the construction of unseen task adapter hypernetworks. The encoded task instructions will be transformed to the adapter hypernetwork weights for the downstream task adaptation. Such networks undergo generation distribution-based alignment training as well as task-specific training in wider ranges. TAGI is evaluated on the SNI multi-instruction tasks, demonstrating good performance against the baseline learning methods.", "questions": "- What if you use significantly more positive examples to define the task and generate the hypernetworks?", "limitations": "- The authors address the limitations of the work quite thoroughly in the appendix.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is neat and cleverly integrates the hypernetwork style of training with the teacher-student alignment methods.\n- The demonstrated efficiency with yet other hypernetwork-based methods is intriguing.\n- The additional studies on ICL as well as self-ablation studies are appreciated.\n- The paper is quite easy to follow and well illustrated.", "weaknesses": "- While I do appreciate the meta-train varying experiments, the distribution shifts of tasks in meta train or test datasets seem to be consistent (i.e., I did not find explicit evidence in the manuscript saying that some of the meta-test tasks are altered). Experiments on a few trials on the train-test splitting combinations are worth investigating to provide a more comprehensive analysis (with mean and variance).\n- Why is there no comparison between TAGI to prefix-based [1] methods?\n- Slightly minor: Particularly for the ICL, how would this work complement or benefit methods such as sample selection [2] or of example optimization style?\n- This is more like a suggestion: It would be good to also have a plotted analysis on the parameter size (can be presented in terms of percentage of the entire network size) of the generated hypernetworks against the corresponding performance.\n\n[1] Li, Xiang Lisa, and Percy Liang. \"Prefix-tuning: Optimizing continuous prompts for generation.\" ACL 2021.\n\n[2] Gupta, Shivanshu, Matt Gardner, and Sameer Singh. \"Coverage-based example selection for in-context learning.\" EMNLP-Findings 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer UQYV:\n\nWe greatly appreciate your professional review of our article. Here is our feedback.\n\n### Response to W1:\n\n> W1. While I do appreciate the meta-train varying experiments, the distribution shifts of tasks in meta-train or test datasets seem to be consistent (i.e., I did not find explicit evidence in the manuscript saying that some of the meta-test tasks are altered). Experiments on a few trials of the train-test splitting combinations are worth investigating to provide a more comprehensive analysis (with mean and variance).\n\nAs demonstrated in Tables 10 and 11, using the P3 dataset as an example, the meta-test tasks include Sentence Completion, Natural Language Inference, Coreference Resolution, and Word Sense Disambiguation. In contrast, the meta-train tasks consist of Multiple-Choice QA, Sentiment Analysis, Structure-to-Text, and Summarization, among others. The distributions of these tasks are not expected to be consistent.\n\nFigures 3 and 4.5 illustrate the effects of varying the number of meta-training tasks on performance. We will include mean and variance in the results in future updates.\n\n### Response to W2:\n\n> W2. Why is there no comparison between TAGI to prefix-based [1] methods?\n\nCertainly, during our experiments, we attempted to use generated prefixes for fine-tuning, but the results were not as promising as expected. We also explored generating prefixes alongside LoRA, but the benefits remained minimal. This may be due to potential issues in our implementation, which we have yet to identify. We appreciate your feedback and will revisit the experiment with prefixes to address this issue. Thank you for highlighting it.\n\nHere, we provide supplementary experimental results for **LoRA fine-tuning (rank=32)**(Tk-Instruct-LoRA) and **prefix fine-tuning (num_virtual_tokens=32)** (Tk-Instruct-prefix), as well as results of TAGI from replacing the hypernetwork with LoRA (w/o Hypernetwork):\n\n| Method                 | Def  | Def + 2Pos. |\n| ---------------------- | ---- | ----------- |\n| Tk-Instruct        | 48.0 | 54.0        |\n| **Tk-Instruct-LoRA**   | 47.5 | 54.6        |\n| **Tk-Instruct-prefix** | 42.6 | 54.2        |\n| Hypertuning        | 38.9 | 48.6        |\n| HINT               | 47.2 | 53.2        |\n| **TAGI**               | 48.4 | 56.3        |\n| **Ablation Study**     |      |             |\n| w/o pretraining        | 47.1 | 55.6        |\n| w/o Instr. Fus.        | 35.1 | 40.6        |\n| w/o $L_{ce}$           | 47.6 | 55.4        |\n| w/o $L_{kl}$           | 45.7 | 53.9        |\n| w/o $L_{ins}$          | 47.5 | 55.2        |\n| w/o Hypernetwork       | 43.8 | 50.7        |\n\nWe observe that the **prefix performs exceptionally well in the \"Def + 2Pos.\" scenario**, likely because it effectively incorporates information from the positive examples. In contrast, **the \"Def\" scenario shows less satisfactory performance**, suggesting that instructions alone are insufficient for achieving optimal results.\n\n### Response to W3:\n\n> W3. Slightly minor: Particularly for the ICL, how would this work complement or benefit methods such as sample selection [2] or for example optimization style?\n\nBetter examples of Instructed Codeword Learning (ICL) typically result in improved performance. **Our method leverages this principle by incorporating ICL instances into the hypernetwork's encoding process, alongside the instructions**, as illustrated in Figure 2. The information from ICL examples is integrated into the decoder through instruction fusion and concatenation. This approach allows the hypernetwork to generate more accurate inference parameters based on high-quality examples. We have included results comparing regular ICL examples with those without ICL examples, demonstrating that we benefit from ICL. Similarly, our method also shows advantages from enhancements to ICL techniques.\n\n| # Num of Meta-train | Def + 2POS (T5-Base) | Def (T5-Base)    | Def + 2POS (T5-Large) |\n| ------------------- | -------------------- | ---------------- | --------------------- |\n|                     | w/o ICL /  w ICL     | w/o ICL /  w ICL | w/o ICL /  w ICL      |\n| 6                   | 38.48/38.85          | 31.73/32.28      | 52.00/52.93           |\n| 31                  | 40.91/41.12          | 32.66/33.68      | 53.08/53.66           |\n| 60                  | 41.94/42.51          | 34.11/34.45      | -                     |\n\n### Response to W4:\n\n> W4. This is more like a suggestion: It would be good to also have a plotted analysis on the parameter size (which can be presented in terms of percentage of the entire network size) of the generated hypernetworks against the corresponding performance.\n\nIn Table 7 of the appendix, we present **the results of varying LoRA ranks and their impact on model performance**. The table illustrates how different parameter sizes affect the model's efficacy. The corresponding table is included below:\n\n|                      | Rank | percentage | RougeL |\n| -------------------- | ---- | ---------- | ------ |\n| Def + 2POS (T5-Base) | 16   | 11.3%      | 41.8   |\n|                      | 32   | 11.8%      | 42.5   |\n|                      | 64   | 12.2%      | 42.3   |\n| Metatrain            | -    | 100%       | 42.1   |\n| Def (T5-Base)        | 16   | 11.3%      | 34.8   |\n|                      | 32   | 11.8%      | 35.3   |\n|                      | 64   | 12.2%      | 35.4   |\n| Metatrain            | -    | 100%       | 35.3   |\n| P3 (T5-Base)         | 16   | 11.3%      | 44.8   |\n|                      | 32   | 11.8%      | 45.6   |\n|                      | 64   | 12.2%      | 45.5   |\n| Metatrain            | -    | 100%       | 44.1   |\n\n**The global author rebuttal Q3** has updated the images, where you can further examine the trends.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q7Vnv6iNr0", "reviewer": "Reviewer_Wqw9", "summary": "The paper addresses limitations of current instruction fine-tuning approaches for large language models, which rely heavily on instance training with extensive task data. This limits adaptability to real-world scenarios where labelled task instances are scarce and broader task generalisation is needed.\n\nThe contributions of this work are listed as follows:\n\n1. The paper introduces Task Adapters Generation from Instructions (TAGI), which automatically constructs task-specific models in a parameter generation manner based on given task instructions, without retraining for unseen tasks.\n\n2. TAGI uses knowledge distillation to enhance consistency between models developed through \"Learning with Instruction\" and \"Training with Instance\" by aligning labels, output logits, and adapter parameters.\n\n3. TAGI enables cross-task generalisation through a two-stage training process including hypernetwork pretraining and fine-tuning.\n\n4. Experimental results on Super-Natural Instructions and P3 datasets show TAGI can match or outperform traditional meta-trained models and other hypernetwork models, while significantly reducing computational requirements.\n\n5. TAGI demonstrates effectiveness in generating adapters for unseen tasks without additional parameter updating or gradient back-propagation, avoiding inefficiencies of repeatedly encoding instructions during inference.\n\nThe authors state their approach \"enables the model to achieve specified tasks and learn from instructions on how to address a category of problems\" while reducing inference overhead and boosting generalisation performance.​​​​​​​​​​​​​​​​", "questions": "Please refer to my weakness section.", "limitations": "Please refer to my weakness section.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "Overall, this work is of high quality, and its strengths from my perspective are listed below.\n\n1. *Novel learning paradigm*: Beyond the traditional instance-based training, the authors propose a new paradigm, \"learning with instructions\", to mimic human learning by understanding and transforming task guidelines to adapters (parameters), which represents a shift from instance-based training to instruction-based learning.\n\n2. *Novel problem reformulation*: This paper reframes the challenge of cross/multi-task generalisation by formulating it as a task-specific parameter generation problem. This addresses a key limitation in current approaches.\n\n3. *Thorough comparisons and ablation studies*: The authors evaluate TAGI on two large-scale instruction datasets (SNI and P3), which provides a robust test of their method. The work also compares TAGI against a wide range of baselines, including methods based on zero-shot in-context learning and hypernetwork. The authors conduct detailed ablation studies to understand the contribution of each component in their model, strengthening the validity of their results.\n\n4. *Significance on bridging human and machine learning*: This is the most interesting aspect of the work. By mimicking human instruction-based learning, this work contributions to the broader goal of developing AI systems that can learn more like humans.", "weaknesses": "While the paper presents a novel approach with promising results, there are several areas where it could be improved:\n\n1. *Limited model size exploration*: The authors primarily focus on models up to 3B parameters, with only limited experiments on an 11B model. This leaves open questions about how well TAGI scales to larger models, which are increasingly common in state-of-the-art NLP. The authors acknowledge this limitation, but could strengthen their work by solving the problem.\n\n2. *Narrow focus on encoder-decoder models*: The paper exclusively uses encoder-decoder models (T5-LM and T0), citing their superior cross-task generalisation. However, this limits the generalisability of their findings.\n\n3. *Limited analysis of instruction quality*: While the paper focuses on learning from instructions, there's little discussion on how the quality or format of instructions affects performance.\n\n4. *Limited exploration of hyperparameter sensitivity*: While the authors provide some ablation studies, a more comprehensive analysis of hyperparameter sensitivity would strengthen the paper. This could include:\n   a) Exploring the impact of different LoRA ranks beyond the chosen value of 32;\n   b) Analysing how the size of the hypernetwork affects performance and efficiency;\n   c) Investigating the trade-offs between pretraining steps and finetuning steps;\n\nAddressing these points would strengthen the paper's contributions and provide a more comprehensive understanding of TAGI's capabilities and limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer Wqw9:\n\nWe greatly appreciate your professional review of our article. Here is our feedback.\n\n### Response to W1:\n\n> W1. Limited model size exploration: The authors primarily focus on models up to 3B parameters, with only limited experiments on an 11B model. This leaves questions about how well TAGI scales to larger models, which are increasingly common in state-of-the-art NLP. The authors acknowledge this limitation but could strengthen their work by solving the problem.\n\nWe fully acknowledge this concern, as highlighted in the limitations section. Due to resource constraints and the challenges associated with training larger models, our results may not be as comprehensive as desired. We plan to refine our methodology and extend the scope of our experiments in future work.\n\n### Response to W2:\n\n> W2. The narrow focus on encoder-decoder models: The paper exclusively uses encoder-decoder models (T5-LM and T0), citing their superior cross-task generalization. However, this limits the generalisability of their findings.\n> \n\nThank you for your insightful comments, which align with our own concerns. In Appendix B.1, we explained our rationale for selecting Encoder-Decoder models.  Encoder-decoder (enc-dec) structure has advantages over decoder-only (dec-only) models regarding task generalization and instruction-following capabilities. \n\nWe also recognize the potential of current decoder-only large models (such as Llama3, Qwen2) that have undergone extensive fine-tuning on instruction tasks, potentially serving as a strong baseline similar to TK-Instruct. However, the effectiveness of our method on these models remains to be determined through further experimental evaluation.\n\nIn the near future, we plan to conduct experiments with the latest decoder-only large models to assess their applicability.\n\n### Response to W3:\n\n> W3. Limited analysis of instruction quality: While the paper focuses on learning from instructions, there's little discussion on how the quality or format of instructions affects performance.\n> \n\nWe primarily explored the impact of the number of instructions on performance, as shown in Figure 3. Under our experimental settings, the quality and format of the instructions, which are optimal (\"gold\"), play a crucial role. The main experiment results in Table 2 show that methods based on hypernetworks do not perform as well as Tk-Instruct which also hints at the importance of instructions to complete tasks. Furthermore, Table 4 demonstrates **a significant decline in performance in our ablation experiments when instruction fusion is omitted (w/o Instr. Fus.), underscoring the importance of instructions**.\n\n### Response to W4:\n\n> W4. Limited exploration of hyperparameter sensitivity: While the authors provide some ablation studies, a more comprehensive analysis of hyperparameter sensitivity would strengthen the paper. This could include: a) Exploring the impact of different LoRA ranks beyond the chosen value of 32; b) Analysing how the size of the hypernetwork affects performance and efficiency; c) Investigating the trade-offs between pretraining steps and finetuning steps;\n> \n\nThank you for the reviewers' valuable feedback.\n\nFirstly, in response to point a), we present the impact of varying LoRA ranks on performance in Table 7 in the appendix. We selected a **rank of 32 based on a balance between performance and computational overhead**.\n\nRegarding point b), we employed a **two-layer MLP** as the supernetwork for efficiency. While exploring different supernetwork sizes, such as larger models like T5 or GPT-2, might yield additional insights, it would significantly increase overhead, as observed in prior work.\n\nFor point c), Table 7 in the appendix details the effects of different fine-tuning steps on performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X8ftxNkdbU", "reviewer": "Reviewer_2ySK", "summary": "This paper tackled zero-shot learning of LLMs to acquire cross-task generalization. The authors focused on the LoRA adapter, one of the methods used for LLMs for parameter-efficient fine-tuning. They generate task adapters by feeding the task instruction to the hypernetwork. Experimental results based on T5-LM-Adapt and T0 models clarified that the proposed method outperformed previous hypernetwork-based methods.", "questions": "See Weaknesses.", "limitations": "1. This paper mentions the increased training cost instead of the reduced inference cost in the appendix. This is fiar discussion.\n\n2. See the third weakness.", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "1. The alignment of LLMs to task instructions is one of the most important capabilities of LLMs. The proposed method achieved comparable performance with the simple but effective standard fine-tuning baseline even with 39% of the inference cost.", "weaknesses": "1. Because the proposed method underperformed the standard fine-tuning on the RougeL metric, the advantage of the proposed method is inference cost. However, the explanation with respect to computational efficiency is limited. I understood the low inference cost is caused by separating the self-attention for the task input x and the instruction i. Let L be the length of x and M be the length of i. Because the proposed method adds the cross-attention between x and i, the computational cost of the proposed method is L^2 + M^2 + LM, in which each term corresponds to the self-attention of x, that of i, and the cross–attention. However, the standard self-attention requires only (L+M)^2. I consider that the difference is small and cannot understand why the proposed method saves 61% inference cost compared to the self-attention.\n\n2. This paper lacks the most important ablation study. Specifically, Table 5 lacks the ablation of LoRA weights. The low-efficiency cost discussed in the above item is caused by the separation of the self-attention. Therefore, the advantage of the main contribution of the proposed method, the LoRA weights generation from the hypernetwork, was not clarified from the experiments. I consider that replacing the standard self-attention with two separate self-attentions and a cross-attention does not worsen the performance largely if the pre-training is sufficient.\n\n3. This paper lacks implementation details. More importantly, the proposed method seems to increase  the number of parameters compared to the T0/Tk-Instruct baselines due to the addition of cross-attention and HyperNetwork. Even if the proposed method is superior in terms of time complexity (the number of FLOPS), the number of the parameters and the space complexity should be discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 2ySK:\n\nWe greatly appreciate your professional review of our article. Here is our feedback.\n\n### Response to W1:\n\n> W1. Because the proposed method underperformed the standard fine-tuning...\n\n1. **Training Phase**: **Gradient computations for each sample are unnecessary**. Our method aims to minimize training costs for new tasks. Unlike traditional methods that require extensive labeled data and gradient updates, our approach bypasses gradient updates by directly generating task model parameters from the task description.\n\n2. **Inference Phase**: **Instruction processing is only required once**. We agree with the reviewer's point about our method's reduced inference costs: **by processing the instruction a single time, we cut down on both time and computational expenses**. However, it appears there may be a misunderstanding—our method's time and self-attention costs for a single example are similar to those of standard approaches since the instruction isn't repeatedly processed.\n\nThe primary cost savings of our method are evident when inferring multiple instances for the same task. For \\( n \\) inference instances, our method incurs a computational cost of $ L^2 + n \\cdot (M^2 + LM) $, whereas the standard self-attention method costs $ n \\cdot (L + M)^2 $. Our approach is most effective when handling more instances of the same task with sufficiently lengthy instructions.\n\nTo compare inference costs, consider processing \\( n \\) samples of length \\( i \\) with a task instruction of length \\( t \\), ignoring the output sequence length. Let \\( N \\) represent the number of FLOPs per token for an encoder-decoder model. The total computational cost for full fine-tuning is $ FLOPs_{\\text{standard}} = N \\cdot n(t + i) $. For TAGI, the cost is $ FLOPs_{\\text{TAGI}} = N \\cdot (t + ni) $. It can be seen that **the effectiveness of TAGI is significantly increased with the increase of n and t**.\n\nDue to length limitations, we reported the analysis of length in the **global author rebuttal Q1**.\n\n### Response to W2:\n\n> W2. This paper lacks the most important ablation study. Specifically, Table 5 ...\n\nGenerating LoRA weights from the hypernetwork is central to our parameter-efficient fine-tuning approach. **The TAGI model converts instructions and task-specific data (like few-shot examples) into efficient modules within the pre-trained language model**. This not only reduces inference costs but also boosts few-shot learning in instruction-based meta-learning. Furthermore, our method can be adapted to create other efficient modules, such as adapters and prefixes, for fine-tuning.\n\nFor the ablation involving cross-attention, we combined it with the instruction for the experiment, as indicated by \"Instr. Fus.\" in Table 4. However, during our experiments, we observed that the cross-attention module's contribution was limited, and as such, we did not report it separately.\n\nWe have included the following ablation experiments: the LoRA results for Tk-Instruct, and the outcomes of **our method without the hypernetwork, directly using LoRA**. The results are as follows:\n\n| Method          | Def  | Def + 2Pos. |\n|-----------------|------|-------------|\n| **Tk-Instruct**        | 48.0 | 54.0       |\n| **Tk-Instruct-LoRA**        | 47.5 | 54.6       |\n| **Hypertuning** | 38.9 | 48.6        |\n| **HINT**        | 47.2 | 53.2        |\n| **TAGI**        | 48.4 | 56.3        |\n| **Ablation Study** |||\n| w/o pretraining | 47.1 | 55.6        |\n| w/o Instr. Fus. | 35.1 | 40.6        |\n| w/o $L_{ce}$         | 47.6 | 55.4        |\n| w/o $L_{kl}$         | 45.7 | 53.9        |\n| w/o $L_{ins}$        | 47.5 | 55.2        |\n| w/o Hypernetwork        | 43.8 | 50.7    |\n\nThe results in the last row show **a significant performance drop without the hypernetwork, highlighting its crucial role in cross-task generalization**. This is consistent with Table 6 of the HINT study [1]. The hypernetwork enhances task understanding by dynamically generating efficient module parameters from instructions, rather than just embedding knowledge, as with LoRA. This dynamic adaptability improves cross-task generalization. We will further explore the hypernetwork's advantages in the appendix.\n\n[1] HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisation, ACL 2023\n\n### Response to W3:\n\n> W3. This paper lacks implementation details. More importantly, the proposed...\n\nWe have detailed the experimental procedures in Section 4.3 and Appendix A.4, and the relevant code has been submitted. The hypernetwork described in Section 3.2.1 (L-237) uses a 2-layer MLP. We will address it by updating the appendix accordingly. \n\n**parameters Costs**\n\nThe T0/Tk-Instruct baselines represent full fine-tuning, as illustrated in Figure 2. Our approach involves updating only the hypernetwork and generating parameters for efficient fine-tuning modules. Adding cross-attention does not significantly increase the number of fine-tuning parameters since **it is incorporated only into the even layers of the encoder**. The additional fine-tuning parameters are as follows:\n\n| Method （T5-Base） | # Params |\n| ------------------ | -------- |\n| T0/Tk-Instruct     | 250M     |\n| LoRA               | 28.3M    |\n| TAGI               | 26.1M    |\n\n**Memory Costs**\n\nLet \\( l \\) be the number of layers, \\( d \\) the model hidden dimension, \\( h \\) the number of heads, \\( k \\) the size of the keys/values, and \\( s \\) the sequence length to be saved. \n\nIn the default TAGI settings, we store two main components: the processed instruction sequence with \\( ds \\) values (one vector per token), and the LoRA weights, totaling $ 2 \\times 32 \\times l \\times h \\times k \\times 2 $ values (32-rank LoRA for both key and value per layer and head). The total memory cost is $ ds + 128 lhk $. With \\( kh = d \\), **the memory cost of TAGI scales proportionally with \\( ds + ld \\)**. Note that the default LoRA rank is 32 but can be adjusted to manage memory costs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### Q1:
> Q1. Instruction Length Analysis
We supplement the instruction length analysis of the experimental dataset here, theoretically supplementing the effectiveness of the method and the savings in reasoning costs.

Median sequence length, given in the number of T5 tokens, for Super-Natural Instructions and P3: We calculate the medians for each format separately, rather than adding the instance and instruction-only values together (hence the mismatch in values). We provide the calculated values in Table. We find that **P3 inputs mostly consist of the instance, with prompt templates consisting of relatively few tokens, while SNI inputs consist mostly of instructions**.

| Text Sequence                   | SNI             | P3              |
| ------------------------------- | --------------- | --------------- |
|                                 | Median # Tokens | Median # Tokens |
| Instance only                   | 44              | 81              |
| Instruction only                | 69              | 24              |
| Instruction + Instance          | 133             | 103             |
| Instruction + 2 positives       | 197             | -               |
| Instruction + 2 pos. + instance | 199             | -               |
| Output                          | 1               | 6               |

###  Q2:
> Q2. How $\lambda_1$ and $\lambda_2$ were tuned.

In the experiment, we set $\lambda_1 = 5$ and $\lambda_2 = \text{sigmoid}(L_{\text{ins}})$. The effects of these different $\lambda$ values on the results are illustrated below. We will include this supplementary experiment in the appendix soon. 

|                      |   $\lambda_1$   |  $\lambda_2$    | RougeL |
| -------------------- | ---- | ---- | -------------------- |
| Def + 2POS (T5-Base) | 0.5         |     $sigmoid(L_{ins} )$                | 40.1 |
|         | 2 |  $sigmoid(L_{ins} )$| 40.9 |
|                      | 5 | $sigmoid(L_{ins} )$ | 42.5 |
| | 10 |  $sigmoid(L_{ins} )$| 38.7 |
| | 5 | 0.2                 | 41.3 |
| | 5 | 0.5 | 41.6 |
| | 5 | 1.0 | 41.2 |

The images can be seen in pdf.

### Q3:
> Q3. The parameter size of the generated hypernetworks against the performance

In Table 7 of the appendix, we present the results of varying LoRA ranks and their impact on model performance. The table illustrates how different parameter sizes affect the model's efficacy. The corresponding table is included below:

|                      | Rank | percentage | RougeL    |
| ------------------- | ---------------- | ---- | ------------------- |
| Def + 2POS (T5-Base) | 16   | 11.3%      | 41.8      |
|                      | 32   | 11.8%      | 42.5      |
|                      | 64   |    12.2%        | 42.3      |
| Metatrain | - |    100%       | 42.1   |
| Def (T5-Base)        | 16   | 11.3%      | 34.8      |
|                      | 32   | 11.8%      | 35.3      |
|                      | 64   |    12.2%        | 35.4 |
| Metatrain | -   |    100%       | 35.3   |
| P3 (T5-Base)         | 16   | 11.3%      | 44.8      |
|                      | 32   | 11.8%      | 45.6      |
|                      | 64   | 12.2%      | 45.5      |
| Metatrain | -   |    100%       | 44.1  |

The images can be seen in pdf.

Meta-review: This paper proposes a novel approach to zero-shot learning: infer the weights of a task-specific module for an LLM based on natural language descriptions of tasks. It can be viewed as a combination of hypernetwork training and teacher-student teaching. The empirical performance is strong. The authors were able to address most of the reviewers' concerns during the discussion period and the authors are encouraged to incorporate these clarifications into the final version of the paper.
Decision: Accept (poster)