Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Tasks
Authors: Po-Nien Kung
Fan Yin
Di Wu
Kai-Wei Chang
Nanyun Peng
Venue: EMNLP 2023 Main (2023)
Keywords: Instruction Tuning
Large Language Models
Active Learning
TL;DR: Method to actively select most informative tasks for the instruction tuning models

Abstract: Instruction tuning (IT) achieves impressive zero-shot generalization results by training large language models (LLMs) on a massive amount of diverse tasks with instructions. However, how to select new tasks to improve the performance and generalizability of IT models remains an open question. Training on all existing tasks is impractical due to prohibiting computation requirements, and randomly selecting tasks can lead to suboptimal performance. In this work, we propose active instruction tuning based on prompt uncertainty, a novel framework to identify informative tasks, and then actively tune the models on the selected tasks. We represent the informativeness of new tasks with the disagreement of the current model outputs over perturbed prompts. Our experiments on NIV2 and Self-Instruct datasets demonstrate that our method consistently outperforms other baseline strategies for task selection, achieving better out-of-distribution generalization with fewer training tasks. Additionally, we introduce a task map that categorizes and diagnoses tasks based on prompt uncertainty and prediction probability. We discover that training on ambiguous (prompt-uncertain) tasks improves generalization while training on difficult (prompt-certain and low-probability) tasks offers no benefit, underscoring the importance of task selection for instruction tuning.

Active Instruction Tuning:
Improving Cross-Task Generalization by Training on Prompt Sensitive Tasks
Po-Nien Kung, Fan Yin, Di Wu, Kai-Wei Chang, Nanyun Peng
University of California, Los Angeles (ponienkung, fanyin20, diwu, kwchang, violetpeng) ucla. cola.edu edu
Abstract
Update Training Task Pool with Useful Task
Instruction tuning (IT) achieves impressive zero-shot generalization results by training large language models (LLMs) massive amount of diverse tasks with instructions. How- ever, how select new tasks to improve performance and generalizability of IT models remains an open question. Training on all exist- ing tasks impractical due to prohibiting com- putation requirements, and randomly selecting tasks can lead to suboptimal performance. In this work, we propose active instruction tuning based on prompt uncertainty. a novel frame- work to identify informative tasks, and then actively tune the models the selected tasks. We represent the informativeness of new tasks with the disagreement the current model out- puts over perturbed prompts. Our experiments on NIV2 and Self- leff-Instruct datasets demonstrate that our method consistently outperforms other baseline strategies for task selection, achieving better -iittiibbttiiiti generalization with fewer training tasks. Additionally, we intro- duce task map that categorizes and diagnoses tasks based on prompt uncertainty and predic- tion probability. We discover that training ambiguous (prompt-uncertain) tasks improves generalization while training on difficult (prompt-certain and (ow-probability) tasks offers no benefit, underscoring the importance of task selection for instruction tuning.!
Task
Tose 159k2
Tock
Selected Tasks
Identify Useful Tasks
"Challenging"
Model
Training Tasks
Existing Tasks
Update IT Model by Training New Tasks
Figure I:
Our proposed Active Instruction Tuning frame- work. Given an instruction tuning (IT) model with a sizeable existing task pool, we actively identify and select useful tasks from it and add them to the training task pool. After that, we train new IT model with the updated training task pool. By continuing this loop, expect to improve the IT model's cross-task generalization ability efficiently. The main chal- lenge lies in identifying useful tasks, for which we propose select prompt- sensiiive tasks.
2023) can perform well on unseen task. The per- formance can be further boosted by increasing the number of diverse training tasks (Xu et al., 2022; Wang al., 2022b; Longpre et al., 2023; Chung et al., 2022). Based on this observation, many recent studies scale up cntrucctiontuningg datasets by manually or automatically curating more tasks with instructions. For example, TO and FLAN have 60 tasks. The NIV2 (Wang et al., 2022b) benchmark extends its dataset to over 800 English training tasks. Self-Instruct (Wang al., 2022a) and Unnatural Instructions (Honovich et al., 2022) prompt LLMs generate over 50K instruction tuning data, and recently, Dynosaur (Yin et al., 2023a) dynamically curates over 80K instruction tuning data from Huggingface datasets (Lhoest et al., 2021), which is still continuously expanding. However, as the scale of datasets grows rapidly, becomes impractical to train on all existing tasks due to overwhelming computing costs. One naive solution to randomly sample tasks for training,
1 Introduction
Recently, instruction tuning has shown great success in improving large language models' cross- task generalizability. When training large language models (LLM) with a wide range of tasks with instructions, models like TO (Sanh et al., 2021), FLAN (Wei et al., 2021), TK-Instruct (Wang et al., 2022b), Instruct-GPT (Ouyang et al., 2022), Al- paca (Taori et al., 2023) and Vicuna (Chiang et al..
'Our code and data can he found at tttps://github. com/Plune CommPluus.ablLP/Active

---

but it can potentially select less informative tasks, leading to suboptimal results (Wang et al., 2023). Therefore, it is crucial to employ an efficient task selection strategy that identifies the most novel and informative tasks for instruction tuning.
this uncertainty method does not require labeled instances for a task, we can also apply prompt un- certainty to determine novel tasks to manually an- notate needed.
We further explore using prompt uncertainty to understand task characteristics and diagnose poten- tial issues. Motivated by Data Map (S (Swayamdipta et al., 2020), which utilizes instance-level training dynamics categorize and diagnose data quality, we propose Task Map, the first task diagnosing method that categorizes tasks based on Prompt Un- certainty and Prediction Probability. Based on the Task Map, we categorize tasks into Ambiguous, Easy and Difficult, inspired by prior in-context learning research (Xie et al., 2021; Pan et al., 2023) to facilitate analysis.
Data selection has been explored under active learning and multi-task learning frameworks. De- spite its prevalence, we argue that they are not applicable to task selection for instruction tun- ing. Specifically, active learning methods have focused on selecting the most useful instances for a single task, using either uncertainty-base based intu- itions such as entropy (Settles, 2009), Monte Carlo dropout (Gal and Ghahramani, 2016), 6), or ensem- ble disagreement (Houlsby et al., 2011 I; Siddhant and Lipton, 2018). However, these uncertainty measurements can only measure uncertainty at instance-level, and will become less effective when applied to task-level selections as the scales of un- certainty values are not comparable across tasks. In multi-task learning, previous research (Ivison et al.. 2022; Poth et al., 2021; Kung et al., 2021) has ex- plored measuring task usefulness by assessing its similarity to the target task. While these methods can enhance performance when aware of the target tasks, they are not suitable for instruction tuning, which aims to improve overall generalization to arbitrary unseen tasks.
We conduct experiments on two instruction tun- ing setting: TK-Instruct models with NIV2 dataset, which generalize to unscen tasks, and Alpaca mod- els with Self-Instruct dataset, which generalize to unseen instructions, following our categorization in Kung and Peng (2023). Results show that our ac- tive instruction tuning method consistently outper- forms baseline methods (random sampling, genera- tion perplexity) for both instruction tuning setting, demonstrating the effectiveness of our approach. Moreover, we discover that while instruction tun- ing with Ambiguous tasks can improve general- ization effectively, Difficult tasks offers no benefit, underscoring the importance of task selection in instruction tuning. Our contributions can be sum- marized as follows:
In this work, we introduce Active Instruction Tuning, framework that aims to actively identify informative new tasks for IT model to contin- uously improve its cross-task generalization abil- ity (refer to Figure 1). While being related ac- tive learning, our task is more challenging. Unlike active learning, which focuses on improving per- formance on a single target task by identifying useful instances, our goal is to identify tasks that enhance overall generalization, a novel concept not explored in previous AL research.
We introduce Active Instruction Tuning, a framework to efficiently improve the IT model's generalization ability in large-scale instruction tuning.
We propose Prompt Uncertainty, a task-level uncertainty measurement for IT, which can identify novel/informative tasks to improve IT models' zero-shot generalization.
To identify informative new tasks for Active In- struction Tuning, we propose prompt uncertainty (refer to Figure 2), a novel task-level uncertainty metric that measures the sensitivity of an IT model against instruction perturbations for a task. Specifi- cally, with a task instruction and a few unlabeled instances, assess the disagreement of model pre- dictions against original and perturbed prompts on multiple instances to obtain an average disagree- ment score. We then select and train the model with the most uncepatnncerrain tasks to enhance the overall cross-task generalization ability. Since
We further propose Task Map, task diagnosing tool that categorizes tasks based on their prompt uncertainty and prediction probability, provid- ing insights into task characteristics and quality.
2 Method
2.1
Active Instruction Tuning
The Active Instruction Tuning framework illus- trated in Figure 1. In reality, when the number of

---

Instance nce Input X
Sentence: Renee played basketball after school, after which she was very tired. Question: How long did Renee play basketball?
Original Instruction
Prompt Uncertainty
In this task, ask you question
Prediction
POy Jo.
write answer that involves "event duration", based on given sentence.
Generate
[/2] 'Hour, 's1
Po.o 0.6
Given y
Given Popi
Perturbed Instruction hn
Perturbed Instruction 2/2 In this task, we ask you to write ancwar nuestion
PGr Xgp for W)
Poo Po.1
Po. 0.45
0.15
W
P6 1xp W)
Pay - Pacz
Po.2 = 0.7
0.1
Perturbed Instruction Is
Average
In this task, ask you to write an answer to question
Ply 1xx
Pox 0.1 Pa.0 Pox
D.5
that involves "event duration", based on given sentence.
Prediction Probability
Disagreement
We demonstrate how mcasure the prompt uncertainty of model W to a task. Given the original instruction fo and instance input zo of task to the model, we first get prediction y and its sentence probability po,o. Next, randomly drop words (highlighted in red) from the original instructions create perturbed instructions. We measure the model's prediction probability of y given each of the perturbed instructions and input æq. Finally, calculate the average absolute difference (disagreement) between the prediction probability of using original and perturbed instructions, providing an estimate of the model's prompt uncertainty for za. We further aggregate this prompt uncertainty scores across instances for task. Further details can be found in section 2.
Figure 2:
tasks is large and continuously expanding, training on all existing tasks becomes impractical due to the overwhelming computing cost. To efficiently improve an instruction tuning model, we can apply a task selection method to actively select the tasks that benefit the current model the most. By repeat- ing this model training and task selection loop, we can continually improve instruction-tunee models' generalization to unseen tasks.
of the model on the original prediction given com- plete and perturbed task instructions. By selecting those most prompt-uncertain tasks, we can select the tasks to which the current model is susceptible.
Prompt Uncertainty Measurement Our Prompt Uncertainty method is motivated from Bayesian Ac- live Learning by Disagreement (BALD) (Houlsby et al., 2011) in single task Active Learning. Instead of measuring the disagreement among ensemble models in a single task, we measure the disagreement of generation likelihoods on the original prediction over perturbed prompts and original prompts of a task. Figure 2 illustrates the process of measuring the prompt uncertainty of model to task's instance To- To measure the prompt uncertainty U, for task given model weights W, corresponding unlabeled dataset X, and instruction (prompt) Iý. we calculate the average disagreement of likelihood between perturbed and original instruction on n randomly sampled examples from X₁-
For the experiment, we use a large training task pool of fixed size. The training procedure consists of multiple iterations. In the first iteration, small number of tasks are randomly sampled to train weak instruction-tune tuned model. In subsequent iterations, we actively select the most useful tasks based on the previous model and train a new model with the selected tasks. We evaluate different task selection strategies by testing the model on unseen tasks at each iteration.
2.2 Prompt Uncertainty
Inspired by uncertainty-based active learning (Sid- dhant and Lipton, 2018), we aim to select those highly uncertain tasks as the most informative ones at each stage for training. While prior active learning work has proposed numerous uncertainty measurements the instance level for single task. these uncertainty values are usually not comparable across tasks. We propose Prompt Uncertainty, a lesk-level uncertainty measurement that estimates uncertainty values by assessing the disagreement
U
Pi. pigl,
pl, = P( 1 13. W),
where E [1, m]. E [0, kj).
P the likelihood of prediction y given model weights W, a task instruction / and corresponding task instance x. & is the number of perturbations.

---

For each example o E X will first get the original oulput yf and its corresponding likelihood Pl.- Then, we will perturb the instruction times and calculate the average absolute difference be- tween the likelihood of vf given original instruction Pl.o and perturbed instructions {pi,lJ E {), In order to perturb task instruction, it is possible to employ paraphrasing techniques, adding extraneous tokens or randomly omitting words, such that the altered instructions can mostly preserve their meaning (A more detailed discussion can be found in subsection 6.2). In our experiment, we assign a 0.2 drop rate for each word in the instruction to create perturbed instructions. After getting the prompt uncertainty for each remaining task, we will select the highly uncertain ones and add them to the training task pool.
Statistics / Models
NIV2
Self-Instruct
Dataset
# training tasks (instructions) # testing tasks (instructions) # data per task Testing on unseen tasks?
756
52K
119
252
2 200*
Active Instruction Tuning # of tasks in initial training set # of task to select at ith iteration Evaluation
68
500
68
500 2'
Evaluation Metries
Rouge-L
Human Eval
GPT Eval
Table I: Comparison hetween NIV2 (Wang et al., 2022b) and Self-Instruct (Wang et al., 2022a) datasets. Most tasks in NIV2 have more than 200 instances, while Self-Instruct only has one instance for each task. These two settings differ in terms of the definition of the task and generalization objective (zero-shot tasks v.s. cross task), described in Kung and Peng (2023).
setting and comparison, please see Table
3.1 Active Instruction Tuning Setting
Underlying Hypothesis We describe the under- lying hypothesis to propose Prompt Uncertainty. From an uncertainty perspective, when measuring the model's sensitivity toward sampled prompts from a task, we estimate the model's epistemic un- certainty, reflecting the model's 's lack of knowledge of particular task. Different from epistemic un- certainty using an ensemble of models (Gal and Ghahramani, 2016), we consider an ensemble of slightly different conditions, i.c., perturbations of prompts for the model, and use the original likeli- hood to represent the ensembled prediction. From the robustness of the in-context learning perspec- tive, if model cannot robustly map task instruc- tions to specific latent concepts, which is reflected by the sensitivity regarding perturbations in instruc- tions, its generalization ability to the corresponding task is limited (Xie et al., 2021; ; Pan al., 2023). To address this, we hypothesize that training the model on prompt-uncertain tasks will improve its ability to associate prompts with specific latent concepts (tasks), leading to better zero-shot performance unseen instructions.
Natural Instruction V2 dataset Wc utilize the NIV2 English tasks split, comprising 756 training tasks and 119 testing tasks, including classification and generative tasks, and run our experiment with five random seeds. 2 For each randomized setting, we first randomly select 68 tasks for the initial train- ing set and select another 68 tasks as the validation set, leaving the remaining 620 tasks the task pool. Afterward, we iteratively apply different task selection strategies to expand the training set and train new IT models, reporting the performance at each iteration [136, 204, 340).
dataset We first randomly sample 500 tasks the initial training set from the 52K tasks in the Self-Instruct dataset, leav- ing the remaining tasks as the remaining task pool. We conduct active instruction tuning and compare model performance at each iteration [1000, 2000, 4000, 8000, 16000].
3.2 Task Selection Strategies
Since we are the first to propose active instruction tuning, we construct several baseline task selection strategies: Random Sampling, High Perplexity and Low Perplexity, to compare with our proposed Prompt Uncertainty method. Random Sampling will randomly sample tasks from the remaining task pool. This usually a strong baseline in sesk-selcction experiments since we utilize a well-constructed dataset as the task pool, which has less noisy and duplicate data. High and Low Perplexity are the baselines inspired by prior active
3 Experiment Setting
In this work, we experiment with two well-known IT datasets: NIV2 and Self-Instruct (Wang et al., 2022b.a). NIV2 is the largest IT dataset with 1600+ cross-lingual tasks. It focuses improving model generalization to unseen tasks, while Self-Instruct is used to train the Alpaca model (Taori et al., 2023) and aims to enhance model instruction following ability, following the catcgorization in prior work (Kung and Peng, 2023). For detailed
We provide details of our experiments in subsection A.2.

---

Natural Instruction V2 (NIV2) Results
Test Overall
Validation Overall
49
Rouge-L Score
45
68
136
204
272
340
136
204
272
340
Test Classification
Test Generative
Validation Classification
Validation Generative
54
38
42
36
41
48
34
Number of Training Tasks
Random
High Perplexity
Perplexity
Fully Trained (680 Tasks]
Figure 3: Experiment results for NIV2 dataset. We compare Ours prompt uncertainty method with other baselines and report the Rouge-L scores testing and validation set at each active instruction tuning iteration [136, 204, 272, 340]. We report the average and standard deviation scores of five runs, each with different initial 68 tasks, 68 validation tasks, and 620 remaining task pool. Note that we validation set, leading high variance. Additionally, we report the Fully Trained results which train the entire task pool. Exact numbers this experiment he Table in the Appendix.
show the standard deviation on validation since each random seed will have different
learning work, which aims select difficult/easy tasks by measuring predicted sentence perplexity for generation tasks or entropy for classification tasks. As these uncertainty measurements are established at the instance level, we aggregate the uncertainty score of multiple (ten for NIV2 and one for Self- Instructt instances in a task to estimate task-level uncertainty. For our method, we measure the Prompt Uncertainty using π 10 random amples and k = 20 prompt perturbations in NTV2 (refer section 2). For Self-Instruct, we measure the prompt uncertainty using n = random examples and k 20 prompt perturbations.
with Random Sampling the 252 user-oriented test set (Wang et al., 2022a). We follow the evalua- tion Vicuna (Chiang al., 2023) report GPT-4, Chat-GPT (GPT 3.5) and Human evaluation scores, and provide more details subsection A.I.
4 Results
4.1
NIV2 Results
Figure 3 displays our experimental results on the NIV2 dataset. For each task selection method, we iteratively select batch (68) of tasks from the task pool (620 tasks) train new model, and compare model performance each iteration. A better task selection method should achieve consistent supe- rior performance at early iterations, when there are still plenty of tasks to select from. Figure 3 demon- strates that when selecting less than 340 tasks (half of the task pool), our proposed Prompt Uncertainty method consistently outperforms other baselines in terms of Overall scores for both the validation and testing sets. This shows that training on prompt- uncertain tasks is indeed the most effective way for better zero-shot cross-task generalization abil- ity. On closer examination, our method is highly effective for Classification tasks, surpassing all
3.3 Training and Evaluation
For NIV2, we follow the current SOTA TK-instruct model's s setting, to train the T5-770M model (Raf- fel et al., 2020) and report the Rouge-L score of Classification, Generative and Overall tasks, on both validation and testing set. During train- ing and testing, we will provide task definition and two examples as instruction demonstration. For Self-Instruct dataset, we train the LLaMA-7B model (Touvron et al., 2023) follows Alpaca model setting. For ovaluation, we report the blind pair- wise comparison of each task selection methods

---

Pairwise Comparison to Random Sampling (Win Lose)
GPT4 Evaluation
Chat-GPT Evaluation
Human Evaluation
Fully Trained (52K)
Ours
Win Tasks Lose Tasks
High Perplexity
Low Perplexity
40
20
26
30
-55
36
1000 2000 4000
8000 18000
1000 Number Training Tasks
2000
4000
8000
16000
1000 2000
4000
8000
16000
Figure 4: Pairwise comparison results between different task selection strategies and random sampling 252 user-oriented instruction (Wang al., 2022a),, evaluated hy GPT4, Chat-GPT, Human Annotators. For four types of methods each active instruction tuning iteration: [1000, 2000, 4000, 8000, 16000). separately conduct pairwise comparison with Random Sampling report the winning tasks (Number Win Tasks Number of Lose Tasks). Note that only conduct the human evaluation compare our proposed Prompt Uncertainty Random Sampling method due to high evaluation cost ($600 US Dollars). provide further details in subsection A.1 Table 4.
Average Uncertainty Scores of Selected Tasks
gies at each iteration. It is shown that when select- ing for more than half of the tasks in the training pool, all task selection strategies start deviating and choose tasks with unfavorable uncertainty scores. For example, High Perplexity method start select- ing tasks with low perplexity scores due to the lack of high perplexity tasks. Specifically, when extend- ing the training tasks from 340 to 408 using prompt uncertainty, the average uncertainty score of se- lected tasks already slightly lower than that of all tasks at the first iteration, indicating there are no high-uncertainty tasks to select from. Note that the lack of uncertain tasks would occur exclusively in an experimental setting. practical scenarios where the number of tasks grows rapidly, the ex- haustion of uncertain tasks is less likely to happen.
Uncertaitty
Perpleaity Tusis
cm
Perpiesty
con
Number Training Tasks
Figure The average uncertainty scores of the selected tasks at each active instruction tuning iteration. For All Tasks, we report the average uncertainty score the 620 training tasks predicted by the initial model.
other basclines. For Generative tasks, the Low Perplexity method performs well on testing tasks at early iterations but poorly the validation set. This inconsistency suggests that the model's 's overall enhanced, but rather the low perplexity tasks in the training set coincidentally benefit the generative tasks in the testing set. Con- versely, our proposed method achieves consistently good performance on both testing and validation tasks, outperforming Random Sampling on testing tasks and exhibiting similar performance valida- tion tasks.
4.2 Self-Instruct Results
We show the pairwise preference comparison of all task selection methods against Random Sampling in Figure First for Fully Truined, we use the offi- cial Alpaca release (Taori et al., 2023), which was trained on all 52K tasks. We compare it to Random Sampling at each active instruction tuning itera- tion. It is shown that for both GPT-4 and Chat-GPT evaluation, the Fully Trained model outperforms Random Sampling with great margin. However, as more training tasks are randomly sampled, the difference preferred tasks is diminishing, indi- cating that performance of the Alpaca setting scales with an increasing number of training tasks.
generalizability
We further investigate the trend of uncertainty scores during active instruction tuning. In Figure we illustrate the average uncertainty scores of the selected tasks using different task selection strate-

---

Task Map
Task Map Results
Test Overall
Validation Overall
13
Rouge-l Score
Easy
Ambiguous
Prediction Probabiltty
324
Number Training Tasks
Ambiguous
Random
Difficult
Easy
64
Figure The performance comparison hetween training on Ambiguous. Easy, and Difficult tasks NIV2 dataset. The setting is similar to Figure 3, we run all methods with three runs for the early active instruction tuning iterations.
Difficult
62
5 Task Map
Prompt Uncertainty
Figure 6: Task Map visualization. We measure the prediction probability and prompt uncertainty of an IT model against 620 tasks in NIV2 and plot the Ambiguous, Easy, and Difficult tasks.
Prior work tries understand dataset's charac- teristics in the field of Dataset Diagnosing. Mo- tivated by Data Map (Swayamdipta et al., 2020), wc propose Task Map, a model-based diagnosing tool that understands the contributions different groups of tasks towards instruction tuning. Differ- ent from previous work using data correctness and variability to construct data map, propose map tasks with the dimensions of Prediction Proba- bility and Prompt Uncertainty, in Figure 6. This follows a hypothesis from recent work in explain- ing in-context learning (ICL) (Xie et al., 2021); when the model performs a task in-context during test time (ICL), might implicitly map the prompt to corresponding latent concept and perform the task under the concept. Prediction Probability rep- resents the model's confidence perform task, indicating task difficulties. In comparison, Prompt Uncertainty represents the consistency of a model to map a prompt to a certain concept, indicating the task's 's ambiguity to the model. We further fol- low the above intuition to categorize the tasks into three types: Ambiguous tasks, where models fail to recognize them and have high prompt uncer- tainty; Easy and Difficult tasks, where models can map the prompts certain latent task knowledge (low prompt uncertainty) and perform the task with high/low confidence (sentence probability). respec- tively. We then use the tasks from these three cate- gories for instruction tuning on NTV2 (Wang et al., 2022a) to understand the contributions of different groups of tasks.
Secondly, for high/low perplexity and our pro- posed task selection method, we first fine-tune an LLaMA (Touvron et al., 2023) model with 500 tasks and then iteratively extend the number of training tasks to [1000, 2000, 4000, 8000, 16000). We then report the pairwise comparison results against Random Sampling at each iteration. Fig- ure 4 shows that Low Perplexity and High Perplex- ity are generally subpar with Random Sampling, indicating that applying inadequate task selection strategies can hurt the model's performance. In contrast, our prompt uncertainty method is almost consistently more preferable by all GPT4, Chat- GPT, and human assessors when selecting less or equal than 8000 tasks, showing that training with prompt-uncertain tasks can lead to better general- ization to the user-oriented test tasks. When the number of training tasks increases 16000, the performance improvement diminishes along with a smaller remaining task pool, which aligns with our results on the NIV2 dataset. Additionally, we discuss our observations regarding applying GPT4, Chat-GPT, and human assessors for pairwise com- parisons. It is seen that while the number of net winning tasks (Win task Lose Tasks) varies lot across each evaluation method, the overall trend is similar, showing certain aligmment of preference across these automatic or human assessors.
In conclusion, experiments NIV2 and Self-Instruct demonstrate that our prompt uncer- tainty method consistently improves cross-task generalization two different instruction tuning scenarios, surpassing random sampling and other uncertainty baselines.
We show the results in Figure 7. It is seen that while training on Ambiguous tasks can effectively improve IT generalization ability and outperform random baseline, training on Easy tasks and Difli- cult tasks is generally worse than randomly select-

---

ing tasks, Furthermore, when selecting more Easy tasks can still slightly boost the IT model's perfor- mance, Difficult tasks can useless, showing no benefit to the model's performance with more training tasks. We hypothesize that Difficult tasks can be too specific and hard to Iearn, therefore useless for improving the IT model's cross-task generalization. While our proposed Task Map can already help diagnose task quality for IT, we look forward future work conducting a more compre- hensive analysis discuss the role of these task categories to bring a comprehensive understanding instruction tuning and in-context learning.
Prompt Uncertainty Shift
Prediction Probability
24
Indiovat Tisk Mumure by ND:
Leott 17 plan MCs
tandi Moasune
tevd153 Pravene Mal
22
0.05
0.10
0.15 6.33 625
0:300.35
Prompt Uncertainty
Figure Tasks' prompt uncertainty shifts before and after training with four analogy tasks. We visualize all the tasks the task map with two models, (Blue) and (Orange), MO is the same instruction-tunes model as in Figure 6, which does not train on any analogy tasks. M1 MO. further trained with four analogy tasks: taskl 159, task 154, task/152 taskI 155. 155 Additionalll. we measure the prediction prohability and prompt uncertainty and four unseen analogy tasks, task 157, task/ 156, task 158, task/ 153, using both MO and M1. plotted in orange and blue. It can be seen that after training the model with analogy tasks (from MO to M1). the prompt uncertainty of the four unseen analogy tasks consistently decreases, while the distribution of other irrelevant tasks remains relatively unchanged.
6 Discussion
6.1 Prompt Uncertainty Reflects Task Novelty
620 irrelevant tasks
To demonstrate how prompt uncertainty reflects the novelty of tasks to model, we designed controlled experiment to visualize how the prompt uncertainty scores of tasks change after the model is trained with relevant tasks. To collect a of rel- evant tasks, we first gathered eight Word Analogy tasks from the NIV2 (Wang et al., 2022b) testing set, which is held unseen from the NIV2 training set. In Figure 8, we measured the prediction probability and prompt uncertainty for 620 unseen tasks (unrelated to analogy tasks) from the NIV2 training and four of the unseen analogy tasks using an instruction-tuned model, labeled as MO. and plotted the task map blue. We further trained the MO model with the other four analogy tasks, resulting in a new model called M1, and used to plot the task map for the 620 irrelevant tasks and four unseen analogy tasks again in orange. It is evident that after training the MO model with the four analogy tasks, the overall prompt uncertainty distribution of the 620 irrelevant tasks remains relatively unchanged, while the prompt uncertainty of the four unseen analogy tasks consistently and significantly decreases. TThis demonstrates that prompt uncertainty can effectively indicate the nov- elty of tasks within the model. When the model is trained with specific tasks, the prompt uncertainty of those relevant tasks notably decreases. Addi- tionally, please note that the prediction probability does not increase after training with similar tasks for these four analogy tasks. This observation highlights that using prediction probability alone cannot effectively reflect the novelty of tasks.
6.2 Prompt Perturbation Methods
While prompt perturbation methods are meant to slightly perturb the prompt without changing its meanings, it is difficult to 100% guarantee the preservation of instruction meaning after automatic paraphrasing methods. To ensure the prompt un- certainty is measured using an extreme per- turbation case, we perturbed all instructions 20 times in our experiments. We also tried several instruction perturbation methods at our early exper- iment stage, such as randomly repeating tokens or adding extrancous tokens, which achieved similar prompt uncertainty scores as randomly dropping words. Additionally, for the NIV2 and datasets we used, which have detailed instructions with many redundant tokens (average 56 words per instruction), randomly dropping 20% of tokens will mostly preserve the meaning of the instruc- tions. For other datasets with concise instructions, a higher dropping rate is needed to perturb the in- structions, leading to higher probability of chang- ing instructions meaning entirely.
'From MO to MI. the average decrease in prompt uncer- tainty scores is 0.0018 for the 620 irrelevant tasks and 0.039 for the four analogy tasks. The prompt uncertainty of analogy tasks decrcases 21 times more than that of the irrelevant tasks.

---

7 Related Work
task, and can be leveraged to detect adversarial out-o--distribution data (Feinman et al., 2017: Yin et al.. 2022). Recent works have also extended un- certainty quantification to autoregressive language models (Xiao and Wang, 2019; Malinin and Gales, 2020). In this work, we propose novel epistemic uncertainty measurement for instruction-tuig LLMs by measuring the disagreement of models conditioned perturbed instructions.
7.1 Instruction Tuning Paradigm
By training large language models (LLMs) with diverse tasks and corresponding instructions, it al- lows the model achieve decent cross-task gener- alization ability (Wei al., 2021; Sanh al., 2021; Wang et al., 2022b; Taori et al., 2023; Chiang et al., 2023; Ouyang et al., 2022). Following the obser- vation from prior research (Xu et al., 2022: Wang et al., 2022a) that scaling up the number of tasks can significantly improve zero-shot generalization, there is research on continuously adding knowledge large language models (Scialom et al., 2022: Jang et al., 2023), along with many large-scale IT datasets emerged. Wang et al. (2022a); Wei et al. (2021); Bach et al. (2022); Xu et al. (2022); Jiao et al. (2023) manually augment existing datasets to form large-scale IT datasets and Gupta et al. (2022); Finlayson et al. (2022) manually construct new IT datasets in specific domains. There also automatic approaches collecting large-scale IT datasets. Wang et al. (2022a); Honovich al. (2022) propose generating moderate-qualitty data from powerful IT models, like GPT-4 and Chat- GPT (OpenAI, 2023). Recently, Dynosaur (Yin et al., 2023a) proposes to curate instructions for the continuously growing huggingface dataset (Lhoest et al., 2021) using GPT-4 to create high-quality IT data with low costs. Additionally, we would like to highlight that IT models rapidly scale in performance with larger models and datasets, the concern of whether they adhere to instructions still remains (Yin et al., 2023b; Kung and Peng, 2023;
7.3 Active Learning and Task Selection
Our work also related to active learning, which iteratively annotates informative instances from an unlabeled pool for efficient training (Olsson, 2009;
Siddhant and Lipton, 2018; Zhang et al., 2022).
Strategies for querying informative instances fall into different categories. See Zhang et al. (2022) for a detailed survey. Our method is more related to disagreement-baasee et al., 201 I; Siddhant and Lipton, 2018; Shen et al., 2017), which queries for instances where multiple models disagree the most, and is usually combined with model uncertainty mcasurements (Gal and Ghahramani. 2016). However, different from ac- tive learning which selects informative instances, consider selections task-level. We show that simply adopting prior active learning strategies task-level do not work well and propose our own methods. There are also works doing task selection for specific target tasks (Parvez and Chang, 2021; Zhou et al., 2023). However, we do not assume knowledge of the target task but select tasks solely based on the uncertainty information of the model.
active learning (Houlsby
Min et al., 2022; Yang et al., 2023; Li et al., 2023; Xue et al.. 2023), and requires further investigation. For recent IT development, see (Zhang et al., 2023)
8 Conclusion
We propose Active Instruction Tuning with prompt uncertainty, framework to enhance the generalization ability of the IT model in large-scale instruction tuning. Our experiments on NIV2 and Instructt datasets demonstrate that training on prompt uncertain tasks consistently outperforms random sampling and other uncertainty baselines, highlighting the effectiveness of our approach. We also introduce Task Map, tool that categorizes tasks based on prompt uncertainty and prediction probability, revealing that while training am- biguous tasks improves generalization, some diffi cult tasks offer benefit. These findings motivate future investigations into prompt uncertainty and task selection strategies for better understanding cross-task task generalization and instruction tuning.
for detailed survey.
7.2 Uncertainty Estimation for LLMs
Uncertainty estimation is essential for ensuring safe deployments neural networks (Abdar et al., 2021). Prior works have decomposed the total uncertainty into aleatoric (data) uncertainty and epistemic (model) uncertainty, and proposed methods to quantify each of them, represented by Monte-Carlo Dropout (Gal and Ghahramani, 2016) and Deep Ensemble (Lakshminarayanan al., 2017). In particular, data uncertainty measures the intrinsic uncertainty from the data distribution. Model uncertainty measures the uncertainty due to lack of understanding of the

---

Limitations
#HRR01129900///R0000266666 CISCO and
ONR grant #N00004-23-1-2780.
While our experiments demonstrate the superiority of our proposed prompt uncertainty method over other baseline task selection methods on the NIV2 and Self-Instruct datasets, there are several limitations to consider. Firstly, our experiments are conducted on open-source instruction tuning mod- els and do not consider the impact of reinforcement learning with human feedback in Instruct-GPT (Ouyang al., 2022). Secondly, although conducted our experiments well-construeted instruction tuning datasets, it is important to note that this setting may not fully capture the challenges posed by noisy or poorly constructed tasks in extreme scenarios, which may require techniques such as noisy task filtering or batch active learning. Lastly, our current experiment on active instruction tuning focuses comparing task selection methods and does not incorporate the effect of continual learning, which could be valuable for improving IT models in realistic settings. In summary, our work primarily focuses on introducing active instruction tuning and com- paring task selection methods within a controlled environment. We look forward to future research to conduct further analysis to comprehensively examine the effects of all these factors.
References
Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U Ra- jendra Acharya, et al. 2021. A review uncertainty quantification in deep learning: Techniques, applica- tions and challenges. Information Fusion, 76:243- 297.
Stephen H Bach, Victor Sanh, Zheng-Xin Yong, Al- bert Webson, Colin Raffel, Nihal V Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, et al. 2022. Promptsource: development environment and repository for natural language prompts. arXiv preprint arXiv:2202.01277
integrated
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery. Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le. and Jason Wei. 2022. Scaling instruction -inettune language models.
Ethics Statement
We describe the computation resources and models we used to conduct our experiments. We conduct all experiments 4 to 8 48GB NVIDIA A6000 GPUs
Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner. 2017, Detecting adversarial sam- ples from artifacts. arXiv preprint 1703.0000
4 NVIDIA A100 GPUs, along with 48 TB disk storage and AMD EPYC 7413 24-Core Processor. The experiment takes around 5500 GPU hours for one 48GB NVIDIA A6000 GPU. Our experiments do not need to leverage private data. For the model, we use open-sourced Huggingface T5-large-Im-adapt models and LLaMA-7B, Stanford Alpaca-7B for our experiments, and we will release code once the paper is accepted.
Matthew Finlayson. Kyle Richardson, Ashish Sabhar- wal, and Peter Clark. 2022. What makes instruc- tion learning hard? an investigation and a new chal- lenge in a synthetic environment. arXiv preprint M2Xix:224.091444
Yarin Gal and Zoubin Ghahramani. 2016. Dropout bayesian approximation: Representing model uncer- tainty in deep learning. In international conference machine learning, pages 1050-1059, PMLR.
Aeknowledgements
Prakhar Gupta, Cathy Jiao, Yi-Ting Yeh, Shikib Mehri, Maxine Eskenazi, and Jeffrey P Bigham. 2022. Im- proving zero and few-shot generalization in dia- logue through instruction tuning. arXiv preprint 2rXiv:2205.1200
We would like to thank Hritik Bansal and Da Yin for their valuable insights during discussion, paper reviews, and constructive comments. We thank the anonymous reviewers for their feedback. This work was partially supported by AFOSR MURI via Grant #FA9550-22- 1--880, Defense Ad- vanced Research Project Agency (DARPA) grant
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2022. Unnatural instructions: Tuning lan- guage models with (almost) no human labor. arXiv preprini arXiv:2212.09689

---

Neil Houlsby, Ferene Huszár, Zoubin Ghahramani, and Máté Lengyel. 2011. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745.
Andrey Malinin and Mark Gales. 2020. Uncertainty esti- mation in autoregressive structured prediction. arXiv preprint
Sewon Min, Xinxi Lyu. Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstra- tions: What makes in context learning work? arXiv preprint 2202 12837.
Hamish Ivison, Noah A Smith, Hannaneh Hajishirzi, and Pradeep Dasigi. 2022. Data-efficient finetuning using cross- task nearest neighbors. arXiv preprint arXiv:2212.00196
Joel Jang, Scungone Kim, Sconghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyung- jae Lee, and Minjoon Seo. 2023. Exploring the bene- fits of training expert language models over instruc- tion tuning. arXiv preprint arXiv:2302.03202.
Fredrik Olsson. 2009. A literature survey of active machine learning in the context natural language processing.
OpenAL. 2023. Gpt-4 technical report.
Wenxiang Jiao, Jen Huang, Wenxuan Wang, Xing Wang. Shuming Shi, and Zhaopeng Tu. 2023. Parrot: Translating during chat using large language models. ArXiv.
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll I. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. 2022. Training language models to follow in- structions with human feedback. arXiv preprint arXiv:2203.02155.
Po-Nien Kung and Nanyun Peng. 2023. Do mod- els really learn to follow instructions? an empir- ical study of instruction tuning. arXiv preprint arXiv:2305.11383.
Jane Pan. Tianyu Gao, Howard Chen, and Danqi Chen. 2023. What co-tontext learning" learns" in-tont:xt: Disentangling task recognition and task learning. arXiv preprint arXiv:2305.097:
Po-Nien Kung. Sheng-Sian Yin, Yi-Cheng Chen, Tse- Hsuan Yang. and Yun-Nung Chen. 2021. Efficient multi-task auxiliary learning: selecting auxiliary data by feature similarity. In Proceedings the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 416 428.
Md Rizwan Parvez and Kai-Wei Chang. 2021. Evalu- ating the values of sources in transfer learning. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies, pages 5084-5116, Online. Association for Computa- tional Linguistics.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable pre- dictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30.
Clifton Poth, Jonas Pfeiffer. Andreas Rücklé, and Iryna Gurevych. 2021. What to pre-train on? ef- ficient intermediate task selection. arXiv preprint MiXiv:2104.084::
Quentin Lhoest, Albert Villanova del Moral, Yacine Jemite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gun- jan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matus- sière, Lysandre Debut, Stas Bekman, Pierric Cis- tac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: community library for natural language processing. In Proceedings of the 202/ Conference Empirical Methods in Natural Language Process- ing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic. Association for Computational I Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21(140):1-67.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafcai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, al. 2021. Multitask prompted training en- ables zero-shot task generalization. arXiv preprint 10..08207..
Zekun Li, Baolin Peng, Pengcheng He, and Xifeng Yan. 2023. Evaluating the catructtionfolllowiii robust- ness of large language models to prompt injection.
Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. 2022. Fine-tuned language models are continual learners. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 6107-6122, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301. 13688.
Burr Settles. 2009. Active learning literature survey.

---

Yanyao Shen. Hyokun Yun, Zachary Lipton, Yakov Kro- nrod, and Animashree Anandkumar. 2017. Deep active learning for named entity recognition. In Proceedings of the 2nd Workshop on Representa- tion Learning for NLP, pages 252-256, Vancouver, Canada. Association for Computational Linguisties.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoe V Le. 2021. Finetuned lan- guage models are zero-shot learners. arXiv preprint arXiv:2109.01622
Yijun Xiao and William Yang Wang. 2019. Quantitying uncertainties in natural language processing tasks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 7322-7329.
Aditya Siddhant and Zachary C. Lipton. 2018. Deep Bayesian active learning for natural language pro- cessing: Results of a large-scale empirical study, In Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, pages 2904-2909, Brussels, Belgium. Association for Com- putational Linguistics.
Sang Michael Xie, Aditi Raghunathan, Percy Liang. and Tengyu Ma. 2021. An explanation of in-context learn ing as implicit bayesian inference. arXiv preprint arXiv:2111.02083
Swahha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang. Hanmanch Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 9275-9293, Online. Association for Computa- tional Linguistics.
Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Yang- gang Wang. Haiyu Li, and Zhilin Yang. 2022. Zero- prompt: Scaling prompt-based pretraining to 1,000 tasks improves zero-shot generalization. arXiv preprint ar arXi::2201.06110
Tianci Xue, Ziqi Wang. Yixia Li, Yun Chen, and Guan- hua Chen. 2023, Tadis: Steering models for deep- thinking about demonstration examples.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-follovvii Ilama model. https:// github. wtt.cooo//ttt www.llaabbtaaddd....
Cheng-Fu Yang. Yen-Chun Chen, Jianwei Yang. Xiyang Dai, Lu Yuan, Yu-Chiang Frank Wang. and Kai-Wei Chang. 2023. Lacma: sagguuge--ligiinng contrastive learning with meta-actions for embodied instruction following.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint (tii::23021197711
Da Yin, Xiao Liu, Fan Yin, Ming Zhong, Hritik Bansal, Jiawei Han, and Kai-Wei Chang. 2023a. Dynosaur: A dynamic growth paradigm for inatrcction-tnnnn data curation. arXiv preprint arXiv: /4327.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannanch Hajishirzi. 2023. How far can camels go? exploring the state of instruction tuning open resources.
Fan Yin, Yao Li, Cho-Jui Hsieh, and Kai-Wei Chang. 2022. Addmu: Detection of boundauy adversarial examples with data and model uncertainty estimation. In Conference on Empirical Methods in Natural Lan- guage Processing.
Fan Yin. Jesse Vig, Philippe Laban, Shafiq Joty, Caim- ing Xiong, and Chien-Sheng Jason Wu. 2023b. Did you read the instructions? rethinking the effective- ness of task definitions in instruction learning. arXiv preprint arXiv:2306.00 150.
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Al- isa Liu, Noah A Smith, Daniel Khashabi, and Han- naneh Hajishirzi. 2022a. Self-instruct: Aligning lan- guage model with self generated instructions. arXiv preprint arXiv:2212 12.122121 222.11000.00
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tian- wei Zhang, Fei Wu, and Guoyin Wang. 2023. Instruc- tion tuning for large language models: A survey.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormo- labashi, Yeganeh Kordi, Amirreza Mirzaci, A. Arunk- umar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Gian- nis Karamanolakis, Haizhi Gary Lai, Ishan Puro- hit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patel, Kuntal Kumar Pal, M. Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravse- haj Singh Puri, Rushang Karia, Shailaja Keyur Sam- pat, Savan Doshi, Siddharth Deepak Mishra, Sujan Reddy, Sumanta Patro, Tanay Dixit, Xudong Shen, Chitta Baral, Yejin Choi, Noah A. Smith, Hanna Hajishirzi, and Daniel Khashabi. 2022b. Super- naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks.
Zhisong Zhang, Emma Strubell, and Eduard Hovy. 2022. A survey of active learning for natural language pro- cessing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6166-6166600 3666619,,446 Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Jing Zhou, Zongyu Lin, Yanan Zheng, Jian Li, and Zhilin Yang. 2023. Not all tasks are horn equal: Understanding zero-shot generalization. In The Eleventh International Conference on Learning Rep- resentations.

---

A Appendix
dataset (Wang et al., 2022a).
A.1 Evaluation details
NIV2 - Active Instruction Tuning Details We utilize the NIV2 English tasks split, comprising 756 training tasks and 119 testing tasks, including classification and generative tasks, We employ five random seeds without selection in our active in- struction tuning experiment. Each seed involves randomly sampling 68 tasks as initial training tasks and 68 tasks as validation tasks. The remaining 620 training tasks form the remaining task pool. In each active learning iteration, we maintain fixed classification and generative task ratio and select 24 classification tasks and 44 generative tasks us- ing different task selection strategies. This fixed ratio allows a more meaningful comparison of our results as we evaluate overall, classification, and generative task scores separately. After the new tasks are sampled, we add them to the previously selected training tasks and form new training task set. We further train a new instruction tuning model with the updated training task set.
Human Evaluation For human evaluation in sec- tion 4 and Figure 4, we recruit crowd-source work- ers on Amazon Mechanical Turk who are native English speakers, score at least 70% on a qualifica- tion test, and pass the attention test. For the anno- tation task, three annotators are presented with the task instruction, the input, and the expected output, followed by two models' outputs in random order. The annotators are asked to indicate whether the first model wins, loses, or has a tie. An example of the annotation interface is presented Figure 9. The final comparison decisions are aggregated from the annotations using majority voting. We assign tie label when all the annotators disagree. To calculate the inter-annotator agreement, we define no-contradiction as agreement with the tie entries removed since an annotator slightly supporting a model may also vote for a tie. Under this definition, the no-contradiction rate is measured as 60.4%. Among the cases with contradiction, we find two annotators agree for 82% cases, and all annota- tors disagree in only 18% cases. We set the per- item reward to $0.1 1 to reach an hourly rate of $15, We collect 3780 comparison annotations to com- pare Prompt Uncertainty with Random Sampling at each active instruction tuning itcration. The total annotation cost is approximately 600 US Dollars.
Active Instruction Tuning De- tails We utilize the 52K self-instruct dataset as the task pool. For the active instruction tuning ex- periment, we will randomly sample 500 tasks as the initial training set and further compare model per- formance at [1000, 2000, 4000, 8000, 16000] train- ing tasks. For task selection, we will first divide all tasks into 13 chunks by output sequence length [[1, 10), [11, 20), ..... [121, 130]], and then apply the task selection methods on each chunk of tasks, fol- lowing the ratio of the number of tasks in all chunks. We conduct this extra step normalize the output sequence length of the selected task for each task se- lection method. This ensures there no imbalance in output sequence length during task selection.
GPT Evaluation We conduct a blind pairwise comparison on GPT-4 and Chat-GPT (GPT-3.5) models using Open-AT API, following similar template as we used for human evaluation, which is shown in Table 2. To compare two models on one instance, we will randomly as- sign "(I)" and "(2)" to the model's predictions and prompt the model to reply with the better choice or "Equal" if two predictions are equally good. Note that when applying GPT evaluation, there are very rare cases (about 0.7% of the cases) that the GPT models will reply with unrelated output, which we will assign "Equal" to these instances. The total annotation cost is approximately 50 US Dollars for GPT-4 and 2 US Dollars for Chat-GPT. For full evaluation results, please refer to Table
Training Details For experiments NIV2 dataset (Wang et al., 2022b), we follow the TK- instruct setting, the SOTA model on the NIV2 dataset to train the T5-770M model (Raffel et al., 2020) with learning rate 2e-5, batch size 128 and 200 instances per task for eight epochs. We evalu- ate the model's 's zero-shot performance on the vali- dation set at each epoch and select the model check- point with the best validation score. For evalua- tion, we follow (Kung and Peng, 2023) setting to report the Rouge-L score of Overall, Classifi- cation, and Generative tasks on both validation and testing sets. For experiments on Self-Instruct dataset (Wang et al., 2022a), We follow Alpaca's ss
A.2 Experiment details
We provide the details of our experiments on two well-known instruction tuming datasets: NIV2 (Wang et al., 2022b) and Self-Tnstruct

---

settings train the LLaMA-7B model with learn- ing rate 2c-5, batch size 128 for four epochs.
Computing Resources For the experiment on NIV2 dataset (Wang et al., 2022b), we conduct our experiments using 4 to 8 Nvidia 48GB A6000 GPUs. For each uncertainty method, it takes around 1200 GPU hours, total of 5000 GPU hours(for single GPU), to run all experiments for Figure 3. For the experiment on Self-Instruct dataset (Wang et al., 2022a), we run with Nvidia 80GB A 100 GPUs. Each uncertainty method takes around 40 GPU hours, which sums to 160 GPU hours for all experiments in Figure 4.

---

Given the model predictions from Random Sampling and Prompt Uncertainty which trained with 1000 tasks, we first randomly assign "(1)" and "(2)" tags:
(1):
(2): Landoo-Sampling 1000
Now we fill the GPT prompt template with these two model predictions.
GPT Prompt:
You're re a helpful Al system that meant to imitate human judgment.
Given an instruction, imput, and two predictions "(1)" and "(2)", please tell me which prediction is most satisfying and correct.
If (1) is better, output "(1)"
If (2) is better, output "(2)".
If both predictions are equally good equally bad, you output "Equal".
Instruction:
If you could help me write an email to my friends inviting them dinner on Friday, would be greatly appreciated.
Input:
Now given two predictions:
(I): Dear Friends, would like to invite you all to join me for dinner on Friday. I am planning to make some of my favorite dishes, and I would love for you all to join me.
The dinner will be held at my place at 7pm on Friday, and am looking forward to seeing you all there.
Please let me know if you are able to make it that I can make sure have enough food.
See you then!
Sincerely,
Your Name]
(2): Dear [Name].
With sincerity,
[Your Name]
Output:
GPT Output: (1)
Table 2: We show GPT evaluation template. When comparing two models' predictions, will randomly assign "(1)" or "(2)" tags and then fill them into the GPT templates with provided instruction and inputs. In this specific instance, the input is an empty string. All the underlined text are the component we injected into the template.

---

Instruction:
What sound does this make?
Optional input:
X
Reference:
háo
Model 1:
The sound this character makes is "hăo", which means "good" in Chinese.
Model 2: Hao
Which one is better?
Model 1 is better
O Tie
Model 2 is better
Figure 9: An example of the annotation interface for the human evaluation in $4.2.
Rasles
10
33
54
18
30
62
AV
54
20
93
10/3
04491
01:11
1130
1179
6175
0.19
13:9
2180
17:4
18.10
1.9
13:30
606
1461
01
3.55
8631
10m
49.15
10:0
33.05
5120
69
1.3
43.0
4100
02.99
48.2
8353
4223
48.3
0.58
12.33
54.5
$640
126
4673
44.37
43.35
Sumplig
4552
6.55
4515
0.3
46.05
56.0
1001
48.6
6130
62.0
15.0
1991
1530
135
14.36
14.1
03.8
1331
13.6
01.
110
19100
914
6.11
100
1111
00
136
4357
435
4300
4450
4435
034
4541
48.4
433
45.0
47.6
48.7
022
4210
19.35
430
4100
4899
2131
1100
111
31.0
5601
6571
130
015
0311
0111
100
Lov Neaketty
68
64.07
43.1
4833
4379
4336
4415
038
43.9
47.0
4.33
4234
4516
2335
038
02.0
48.00
4183
4006
3500
48.9
46:21
4631
6633
1326
46:1
46.00
677
13.53
18.3
6.53
11.0
513
18.37
127
1385
13.%
4333
6.30
(141
4336
111
01
05.1
10/4
11:1
15:11
(1:11
0.21
13
01.65
1000
1914
16
619
(1:0
15
44.55
(1:)
099
Solazica Mecbals
TaikNNa
Gaucrate
Avy
Art
0L38
23.55
56.%%
$1.33
1..1
Fooggt Lioesaayy
18
1100
1110
8511
4018
4417
210
4033
4361
43.31
0.56
76.70
61.5
63.5
2.71
39.4
-1.5
35.43
33.55
4531
4630
17:6
13355
130
15-0
54
326
34.97
3000
1606
15:00
1996
114
14t0
6.66
195
11:11
43.27
41.0
4115
L23
44.0
1533
16.5
900
1130
04:11
410
43.0
4435
534
45.55
5441
1530
3748
1535
433
4430
135
35.99
630
613
207
805
35
Nepeety
4385
112
4.41
236
4357
5UM
1355
1350
3748
515
05.52
12.8
30
7001
2150
414
31.0
304
5345
13.5
16%
07.85
17:8
15.
1191
18
19
1535
1.5
179
17.45
Table 3; Full experiment results in Figure

---

GPT-4 Evaluation (Compare to Random Sampling)
Task Selection Methods
Task Num
Win
Lose
Tie
Error
Win Lose
Fully Trained
1000
143
55
54
0
88
2000
129
55
67
74
4000
104
70
78
0
34
8000
95
67
90
0
28
16000
67
101
17
Prompt Uncertainty (Ours)
1000
72
11
2000
100
73
79
27
4000
75
94
83
19
8000
89
75
88
14
16000
78
83
91
-
High Perplexity
1000
64
88
100
24
2000
83
81
88
2
4000
84
91
-7
8000
88
-8
16000
86
16
Low Perplexity
1000
73
81
98
-8
2000
78
96
78
18
4000
74
03
75
29
8000
73
04
75
31
16000
71
97
84
0
26
Chat-GPT Evaluation (Compare to Random Sampling)
Task Selection Methods
Task Num
Win
Lose
Tie
Error
Win Lose
Fully Trained
1000
117
62
71
2
55
2000
125
71
55
54
4000
99
74
76
3
25
8000
93
71
86
2
22
16000
72
10
Prompt Uncertainty (Ours)
1000
70
24
2000
94
73
82
3
21
4000
86
76
90
0
10
8000
86
76
88
2
10
16000
83
83
86
0
0
High Perplexity
1000
58
95
98
-37
2000
88
74
90
0
14
4000
86
-2
8000
81
5
1
16000
68
93
91
0
-25
Low Perplexity
1000
68
98
86
0
-30
2000
82
90
79
-8
4000
71
103
74
4
-32
8000
80
106
66
0
-26
16000
76
96
79
-20
Human Evaluation (Compare to Random Sampling)
Task Selection Methods
Task Num
Win
Lose
Tie
Error
Win Lose
Prompt Uncertainty (Ours)
1000
101
88
13
13
2000
74
66
8
8
4000
94
83
11
11
8000
93
84
9
16000
93
-2
Table 4: Full experiment results Figure


Reviews:
[{"review_id": "R5E2edLA90", "reviewer": "Reviewer_daei", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback.\n\n**Scalability**:\nThe computation cost of measuring prompt uncertainty will scale linearly with the number of new tasks. We will include additional details about the scalability of active instruction tuning and prompt uncertainty in the updated version.\n\n**Generalizability**:\nWe believe NIV2 and Self-Instruct datasets are two representative datasets for two different types of instruction tuning, described in lines 256 - 265. While the NIV2 dataset is human-curated and has clear task boundaries like FLAN and PromptSource datasets, the Self-Instruct dataset is GPT-generated and does not have clear task boundaries, similar to the ShareGPT dataset used in Vicuna training. We conduct experiments on these two different types of datasets to showcase that our method is generalizable in different scenarios.\nWe acknowledge that the current selection of datasets does not contain data from non-English and specific domains (medical, science) in the limitation Section of the final version.\n\n**Implicit Assumptions**:\nOur empirical results show that the proposed prompt uncertainty is associated with task usefulness during instruction tuning.\n\n**Experimental Weakness**:\nWe agree that testing on a larger scale and closed-source models might bring more insights, and also acknowledge this limitation at line 576 - 580. However, we do not have access to those closed-sourced models. While Instruct-GPT can be finetuned with OpenAI API, we cannot access the prediction logits and calculate the prompt uncertainty for these closed-source models.\n \n**The research did not incorporate continual learning**:\nWhile continual learning (CL) centers on preserving performance on historical tasks, our proposed active instruction tuning concentrates on enhancing generalization performance, which better aligns with the objective of instruction tuning. The retraining of new models during iterations of active instruction tuning facilitates a direct evaluation of the selected task utility and the efficacy of task selection methods.\nRegarding efficiency in an application setting, incorporating continual learning could potentially yield further improvements, which we have stated in the limitations section (lines 585 - 588). We leave the combination of continual learning and active instruction tuning as an exciting future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The presented work introduces a novel framework, Active Instruction Tuning with prompt uncertainty, aimed at enhancing the generalization capability of Instruction Tuned (IT) models in large-scale instruction tuning scenarios. The main reasons to accept this work are:\n\nNovelty: The proposed method of employing prompt uncertainty as a criterion for task selection is novel and addresses the need for efficient task selection in large-scale instruction tuning settings.\n\nThorough Experiments: Experiments conducted on two established datasets (NIV2 and Self-Instruct) validate the superiority of the proposed method over other baseline task selection techniques. This provides empirical evidence of the method's effectiveness.\n\nTask Map Tool: The introduction of the Task Map tool, which categorizes tasks based on prompt uncertainty and prediction probability, is a significant contribution. It offers insights into the nature of tasks and their potential impact on model performance, suggesting directions for future research.\n\nBroad Implications: The study's findings have implications beyond just the tested datasets. By revealing that certain \"difficult\" tasks offer no benefit in instruction tuning, the research prompts a deeper investigation into the nature and quality of tasks used for instruction tuning.\n\nComprehensive Discussion: The work not only presents experimental results but also discusses the limitations, offering a balanced perspective. This self-awareness suggests areas of improvement and highlights potential pitfalls.\n\nRelevance to Current Research: The research ties into current trends in the machine learning community, like the importance of active learning, uncertainty estimation, and the growth of instruction tuning paradigms, making it timely and relevant.", "reasons_to_reject": "Scalability: The method is designed to work when the number of tasks is large and continuously expanding. However, how it scales with an ever-increasing number of tasks in terms of efficiency and computational costs is not immediately clear.\n\nGeneralizability: The experiments seem to be focused on specific datasets (NIV2 and Self-Instruct). How the method performs on different types of datasets or real-world scenarios is a concern and what scenarios do the current selection of the datasets not cover needs to be discussed further.\n\nImplicit Assumptions: The method is based on certain hypotheses, such as the association between prompt uncertainty and the model's lack of knowledge about a particular task. If these hypotheses are incorrect, then the foundational premise of the method is weakened.\n\nExperimental Weakness: The experiments were conducted only on open-source instruction tuning models. As a result, the generalizability of the findings to more advanced models, especially those employing reinforcement learning with human feedback (as in InstructGPT by Ouyang et al., 2022), remains untested.\n\nThe research did not incorporate continual learning in its experiments. Given that continual learning could be a significant factor in realistic settings, especially when dealing with a constantly evolving task pool, the absence of this component might limit the applicability and adaptability of the proposed approach.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The study presents a novel framework called Active Instruction Tuning. This approach is designed to effectively enhance the generalizing capacity of Instruction Tuning (IT) models, especially when dealing with expansive instruction tuning scenarios.\n\nThe researchers have introduced a unique metric named \"Prompt Uncertainty\" at the task level for Instruction Tuning. This metric aims to pinpoint new or insightful tasks that can significantly elevate the performance of IT models in unfamiliar or zero-shot scenarios.\n\nAdditionally, the team has developed a diagnostic tool called \"Task Map.\" This tool is tailored to categorize tasks based on two criteria: the uncertainty in their prompts and the likelihood of their predictions. By doing so, it offers valuable insights into the nature and quality of different tasks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qer81ojGmE", "reviewer": "Reviewer_26oa", "summary": "", "questions": "Q1) During perturbation, how could you ensure that the instruction preserve their meaning? For instance, when the word \"not\" is chosen, it could totally change the meaning.\n\nQ2) It is questionable whether in this scenario, the performance of the already seen tasks are preserved. Does any forgetting happen? If so, would adding a replay buffer be the most viable solution while choosing new tasks to learn with Prompt Uncertainty? This is my biggest concern & question of this paper.\n\nQ3) Would the results hold with even larger dataset scale (e.g., Flan Collection, ShareGPT)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback.\n\n**[Q1] How to make sure the prompt perturbation preserve the meaning**:\nIt is difficult to 100% guarantee the preservation of instruction meaning after automatic paraphrasing methods. To ensure the prompt uncertainty is not measured using an extreme perturbation case, we perturbed all instructions 20 times in our experiments. We also tried several instruction perturbation methods at our early experiment stage, such as randomly repeating tokens or adding extraneous tokens, which achieved similar prompt uncertainty scores as randomly dropping words.\nAdditionally, for the NIV2 and Self-Instruct datasets we used, which have detailed instructions with many redundant tokens (average 56 words per instruction), randomly dropping 20% of tokens will mostly preserve the meaning of the instructions. \nFor other datasets with concise instructions, a higher dropping rate is needed to perturb the instructions, leading to a higher probability of changing instructions meaning entirely. We will add more discussions in the limitation section for the final version.\n\n**[Q2] Will the method forget seen tasks**:\nWe clarify that our setting re-trains a model at every active instruction tuning iteration without applying continual learning, and thus, there is no concern about forgetting seen tasks. \nWe do not incorporate continual learning to investigate historical task performance but focus only on generalization performance, which better aligns with the goal of instruction tuning.\nIn terms of efficiency, we acknowledge the limitation of this strategy in our limitation section (lines 585 - 588). \nWe leave the combination of continual learning and active instruction tuning as an exciting future work.\n\n**[Q3] Will the results hold for larger datasets**:\nOur work focuses on proposing active instruction tuning and prompt uncertainty with a comparable and reproducible setting for future work to further research. We test our method on NIV2 and self-instruct datasets, which have sizes comparable to FLAN and ShareGPT datasets. \n\n**Missing References**:\nThanks for suggesting these references. We will add these work to related work for the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The experiments are solid, tested on diverse settings with multiple runs.\n2. The authors suggest a practical solution of how to efficiently & continually learn new tasks.\n3. The TaskMap is very practical and could be used in future work.\n4. The proposed Prompt Uncetainty method is easy to implement and could be used in multiple scenarios.", "reasons_to_reject": "1. I do have some concerns regarding whether this method could also preserve the seen task performance (See Question 2)", "questions_to_authors": "Q1) During perturbation, how could you ensure that the instruction preserve their meaning? For instance, when the word \"not\" is chosen, it could totally change the meaning.\n\nQ2) It is questionable whether in this scenario, the performance of the already seen tasks are preserved. Does any forgetting happen? If so, would adding a replay buffer be the most viable solution while choosing new tasks to learn with Prompt Uncertainty? This is my biggest concern & question of this paper.\n\nQ3) Would the results hold with even larger dataset scale (e.g., Flan Collection, ShareGPT)?", "paper_topic_and_main_contributions": "The authors introduce a method that could be used to determine which task would be most beneficial to train on given that there are a lot of available instruction tuning datasets. The authors test a continual learning/active learning scenario of adding new tasks continuously. Moreover, the authors also propose a TaskMap that could be used to understand the difficulty of a task based on the uncertainty. While the previous DataMap was only used for tasks based on a single-task fine-tuned LM, TaskMap could be used on a variety of tasks with a single instruction tuned LM.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- Scialom, T., Chakrabarty, T., & Muresan, S. (2022, December). Fine-tuned language models are continual learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 6107-6122).\n\n- Jang, J., Kim, S., Ye, S., Kim, D., Logeswaran, L., Lee, M., ... & Seo, M. (2023). Exploring the benefits of training expert language models over instruction tuning. arXiv preprint arXiv:2302.03202.\n\n"}}, {"review_id": "xCC5VLHSbI", "reviewer": "Reviewer_wiNR", "summary": "", "questions": "see reasons for reject", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback.\n\n**Whether the model is retrained by shuffling the updated task pool or in a continual learning manner**:\nWe follow a similar experiment setting as prior active learning work in a single-task scenario to re-train a new model for each active instruction tuning iteration. \nWe do not incorporate continual learning to investigate historical task performance but focus only on generalization performance, which better aligns with the goal of instruction tuning.\nIn terms of efficiency, we acknowledge the limitation of this strategy in our limitation section (lines 585 - 588). \nWe leave the combination of continual learning and active instruction tuning as an exciting future work.\n\n**No well-known active learning methods as baselines**:\nPrevious Active Learning (AL) methods primarily work on multi-class classification tasks, while IT is generative, which obscures most methods to be directly applied. \nTo establish baselines for active instruction tuning, we apply perplexity, which is one of the previous AL methods for instance-level selection. Aggregating perplexity to the task level is a straightforward extension of instance-level AL to task-level AL. We show that our prompt uncertainty is better than this baseline.\nWe will add more detailed descriptions in our paper for better clarity.\n\n**Omitted reference to the NIV2 dataset in introduction**:\nThanks for noticing! We will add a reference in the updated version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The paper proposes a novel framework for active instruction tuning, which addresses the challenge of task selection to improve cross-task generalization in LLMs.\n2) The prompt uncertainty measurement is a unique contribution that allows for the selection of informative tasks based on the model's sensitivity to prompt perturbations.\n3) The experiments are well-designed and conducted on two different datasets, demonstrating the effectiveness of the proposed method.", "reasons_to_reject": "1) The training process appears to involve the use of an updated task pool. It would be of interest to know whether the model is retrained by shuffling this updated task pool, or whether it is trained in a continual learning manner, focusing only on the newly added task. If it is the former, I suspect this could be a time-consuming process.\n2) The authors exclusively compared their method with their own established baseline, without including well-known active learning methods from previous studies. Despite the authors' assertion that their method operates on a task level, it remains unclear why other baseline methods could not be applied in the same context.\n3) The authors omitted a reference to the NIV2 dataset in the Introduction section, which is a noticeable absence.", "questions_to_authors": "see reasons for reject", "paper_topic_and_main_contributions": "The paper proposes a framework called Active Instruction Tuning for improving the cross-task generalization of large language models (LLMs) trained on diverse tasks with instructions. The key idea is to actively select informative tasks based on prompt uncertainty, which measures the sensitivity of the model against perturbed prompts. The authors conduct experiments on two instruction tuning datasets and demonstrate that their method consistently outperforms baseline strategies for task selection, achieving better generalization with fewer training tasks. They also introduce a task map that categorizes tasks based on prompt uncertainty and prediction probability, providing insights into task characteristics and quality.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "see reasons for reject"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Overall, the paper offers a comprehensive overview of the active instruction tuning framework and highlights intriguing insights regarding the efficacy of prompt uncertainty for task selection. However, the distinction between task-level and instance-level selection remains somewhat unclear in the paper. I am left wondering whether it would be more straightforward to consider the selection process on an instance-level, given that each instance is encompassed by its own instruction.\n\n\n\n\n"}}, {"review_id": "SSOI5UXl2C", "reviewer": "Reviewer_Yx6p", "summary": "", "questions": "The data size of the selected task can also impact the training process, as the iteration steps for updating parameters may vary significantly. Is this a factor that needs to be considered?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback.\n\n**Performance gap between proposed schema and fully trained instruction tuning**:\nWhile surpassing a fully trained instruction tuning model is the ultimate goal, being able to select the most effective tasks to train on a given limited computation budget has many practical applications. In a real-world context, with vast tasks and data, while large corporations may be able to train on all existing tasks, small corporations may find training on all tasks impractical.\nSimilar to prior active learning work aimed to improve annotation/computing efficiency, our proposed active instruction tuning framework aims to make instruction tuning more applicable by finding efficient task selection methods and bringing further understanding to task characteristics.\n\n**Question: Is each task's data size a factor that needs to be considered?**:\nYes. We believe each task's data size can be a factor for instruction tuning performance on the NIV2 dataset. Since all tasks can have different difficulties and require different numbers of training instances, selecting the best amount of instances for each task can potentially improve instruction tuning efficiency and achieve better overall generalizability. However, since there are no clear task boundaries for the Self-Instruct dataset, each task/instruction only has one instance, so this is not considered a factor.\nIn our work, we fix the number of instances (200) used in each task for NIV2 to focus on task selection. Additionally, there are some potential methods for future research on determining the number of instances used in each task. For example, if we select tasks from the task pool without replacement (put them back into the task pool after selection) with only a few instances for each selection iteration, we can eventually select different amounts of instances from each task. We also look forward to future work research on the appropriate amount of instances for each task during instruction tuning, which can potentially improve active instruction tuning further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It's novel to select tasks with prompt uncertainty and experimental results show its superiority compared to the perplexity-based method.\n2. Task map that combines prediction probability and prompt uncertainty can be useful to select Ambiguous tasks to perform instruction tuning and yield a better generalization.", "reasons_to_reject": "There is still a certain gap between the overall performance of the proposed schema and fully trained instruction tuning. While the purpose of instruction tuning is to obtain better generalization, although training on only about half of the tasks can achieve comparable results with the method, I think most people will still choose to train on all data.", "questions_to_authors": "The data size of the selected task can also impact the training process, as the iteration steps for updating parameters may vary significantly. Is this a factor that needs to be considered?", "paper_topic_and_main_contributions": "This paper proposes a prompt-uncertainty-based task selection strategy that actively selects tasks that necessitate instruction tuning to mitigate the issue of extensive training resource cost caused by the increasing amount of data. Experimental results show that the proposed strategy can lead to better generalization over NIV2 and Self-instruct benchmark. The task map further demonstrates the effectiveness of selecting tasks with prompt uncertainty.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a new metric for identifying tasks for improving active instructing tuning. All reviews agree on the novelty of the proposed metric and usefulness of the study (for example task maps). Authors evaluate their method on large open-source LMs and two different instruction tuning dataset. Reviews also highlight some interesting follow-up questions and future research.
Decision: Accept-Main