Evaluating the Zero-shot Robustness of Instruction-tuned Language Models
Authors: Jiuding Sun
Chantal Shaib
Byron C Wallace
Venue: ICLR 2024 spotlight (2024)
Keywords: Instruction Tuning
Robustness
Large Language Models
Primary area: visualization or interpretation of learned representations
TL;DR: An evaluation of O.O.D. robustness of instruction-tuned LLMs with different instructions w.r.t. the tuning process

Abstract: Instruction fine-tuning has recently emerged as a promising approach for improving the zero-shot capabilities of Large Language Models (LLMs) on new tasks. This technique has shown particular strength in improving the performance of modestly sized LLMs, sometimes inducing performance competitive with much larger model variants. In this paper, we ask two questions: (1) How sensitive are instruction-tuned models to the particular phrasings of instructions, and, (2) How can we make them more robust to such natural language variation? To answer the former, we collect a set of 319 instructions manually written by NLP practitioners for over 80 unique tasks included in widely used benchmarks, and we evaluate the variance and average performance of these instructions as compared to instruction phrasings observed during instruction fine-tuning. We find that using novel (unobserved) but appropriate instruction phrasings consistently degrades model performance, sometimes substantially so. Further, such natural instructions yield a wide variance in downstream performance, despite their semantic equivalence. Put another way, instruction-tuned models are not especially robust to instruction re-phrasings. 
We propose a simple method to mitigate this issue by introducing ``soft prompt'' embedding parameters and optimizing these to maximize the similarity between representations of semantically equivalent instructions. We show that this method consistently improves the robustness of instruction-tuned models.

Under review as a conference paper at ICLR 2024
EVALUATING THE ZERO-SHOT ROBUSTNESS OF INSTRUCTION-TUNEE LANGUAGE MODELS
Anonymous authors Paper under double-bling review
ABSTRACT
Instruction fine-tuning has recently emerged as a promising approach for improving the zero-shot capabilities of Large Language Models (LLMs) on new tasks. This technique has shown particular strength in improving the performance of modestly sized LLMs, sometimes inducing performance competitive with much larger model variants. In this paper we ask two questions: (I) How sensitive are instruction models to the particular phrasings of instructions, and, (2) How can we make them more robust to such natural language variation? To answer the former, we collect a set of 319 English instructions manually written by NLP practitioners for over 80 unique tasks included in widely used benchmarks, and evaluate the variance and average performance of these instructions as compared to instruction phrasings observed during instruction fine-tuning. We find that using novel (unobserved) but appropriate instruction phrasings consistently degrades model performance, sometimes substantially Further, such natural instructions yield wide variance in downstream performance, despite their semantic equivalence. Put another way, instruction-tunee models are not especially robust instruction re-phrasings. We propose a simple method to mitigate this issue by introducing "soft prompt" embedding parameters and optimizing these to maximize the similarity between representations of semantically equivalent instructions. We show that this method consistently improves the robustness instruction-tuned models.
INTRODUCTION
Multi-lask instruction-luning
Clien tark de trettin and ippot eeply with ougut thes took you awe givan you 'C",
Instruction-tuned LLM
mudipli choice
connct option.
The numon of Augustus reterred to which of the fallowing di chanactoristis??
(A) Divino power (8) Sesual wirlity (D) Milltrr acumen (D) Philosophhc ca imtllest
Training
Oreerved Insirusion
detintion inout rption. queatine and bose
mutjoll-choise
Testing
Answor with option indeees f.c. 14; 18"; "C", and
Desee's are exberralyy environments, sn lequide will resurely be meager
Unobesenee Incroution
AX
tolineing muliple-hhizze question that espart level comatt knoaledge.
Please snlect the correc atmme to the queeston below twwm the ophone A: 10,, 'C: "D"
ator carolully examining coch anowa.
returely
(A) nenbow plentind (6) (P) tnlertte
Figure 1: How well do models trained on instruction-tuninn tuningg datasets generalize to novel instructions (unobserved in training)? Our analysis suggests that they do not do so very well. Shown is a case where pairing an example with an observed instruction yields the correct output, while providing a distinct but equivalent instruction produces incorrect response. We analyze this fragility in in-depth, and based on this analysis we propose and evaluate a simple method that mitigates it.
Large Language Models (LLMs) now dominate NLP, in part because they enable zero- and few-shot adaptation to new tasks via prompting (Brown et al. 2020 Chowdhery et al., 2022] Hotfmann et al.) 2022 Zeng al. 2022). Recent work has shown the promise of instruction- luning, or fine-tuning

---

Under review as a conference paper at ICLR 2024
LLMs with natural language instructions. This improves LLM performance in zero- and few- shot settings, sometimes dramatically, especially "mid-sized" models (Chung et al.] |202|]0 Ouyang
et al. 2022). For example, the instruction-tune suttionntteedddd Flan-T5-XL (3B parameters; Chung et al. 2022)
outperforms GPT-3 (175B) on several benchmarks, despite being dramatically smaller. Similarly,
Alpaca (Taori al. 2023) an instruction-tuned LLaMM-7B Touvron et al. (2023b) GPT-3 on
multiple NLP tasks. This has motivated curation of instruction-augment
task collections (Wang
al. 20226 Wei et al. 202T. and research into improving instruction-tuning (Longpre et al. 2023
Xu et al. 2022, Sanh et al. (02I). In this work, we ask: How sensitive are instruction-
LMs to
natural variations in instruction phrasings? This is important given that the primary motivation of
instruction-tuning is to permit zero-shot adaptation to new tasks via natural language: If models are overly sensitive to particular phrasings of instructions, this may greatly limit their utility in practice.
Prior work has established that L.MSs do not seem to intuitively "understand" prompts (Webson & Pavlick, 2022 Jang et al., 2023 Zhang et al. 2023a), but these efforts did not focus on instruction- tuned models and their sensitivity to plausible (appropriate) instruction phrasings. Recent, contempo- rancous work (Gu et al. 2023) investigated the robustness of instruction- iustruction-tuued models, and found that instruction-uun tuned T5 (Raffel et al. 2020 is robust instruction perturbations in few-shot settings, but less so in zero-shot application. We contribute a more in-depth analysis of this phenomenon across much wider set of instruction-tuned models and benchmarks. Based this analysis, we also introduce and evaluate a method for improving the robustness of such models.
We collect large of task instructions manually composed by NLP researchers; these are valid instructions but distinct from those used in instruction tuning corpora. We then assess the performance of LLMs fine-tuned on different instruction corpora when provided novel (but appropriate) instructions
on two large benchmarks: MMLU U (Hendrycks et al. 2020) and BBL (Srivastava el al., 2022). We find that using novel instructions in zero-shot application degrades accuracy considerably (Figure [ For example, for Flan-T's T5 XXL, using instructions not observed in training but which are appropriate for tasks leads to a 6.9 point drop in performance on average across benchmarks.
Our main contributions are summarized as follows. (1) We perform comprehensive and in-depth analysis of the robustness of instruction-tune tuned LLMS across three "families" of such models (Flan-T5, Alpaca, and TO) using large benchmarks. For this we collect a large set of new task instructions composed by researchers in NLP; we release this dataset to facilitate additional work on instruction robustness. We observe substantial performance degradation when using "novel" (unseen in training) instructions. (2) We propose a simple method to improve robustness by encouraging LLMs to induce similar representations for semantically equivalent instructions; this yields promising results.
2 RELATED WORK
Multitask learning and instruction-tuning. Training single text-to-text model capable of handling arbitrary queries has been an aspiration in NLP for at least half decade. Prior to prompting and instructing LLMs, there were efforts to unify disparate tasks by reframing them as instances of general question answering (McCann et al. 2018 Khashabi et al. 2020 Keskar et al. 2019). Recent efforts have focussed on compiling and tine-tuning LLMs on corpora comprising diverse tasks with associated natural language instructions (Wei et al. 1 Mishra et al. 2021 [annh et al. (02II)
ie., instruction- instruction-tuning. For example, Waan et (002bb) compiles over 1600 tasks and enriches these with both instructions and negative examples. Similarly, the recently released OPT-IMI Bench Iyer et al. (2022) comprises 2000 NL.P tasks. The Flan 2022 task collection Longpre et al. (2023)
additionally features Chain-of- Thouuhh (CoT) style "reasoning" chains in instruction templates.
These cott--rreesoore---olleeeeie of instructions, tasks, and samples- have facilitated the training of instruction-tuned model families such as Flan-T5, Flan-PaLM, and OPT-IML (Iyer al. 2022
Evaluating prompting and instruction capabilities. Instructions can be seen as special sort of model prompting, which few recent efforts have critically evaluated. Websoo & Pavlick (2022) asked whether models meaningfully "understand" prompts, finding that they largely do not: Performance is often unaffected when irrelevant and misleading prompts are provided. In follow up work, Jork, et al. (2023) evaluated performance on negated prompts and observed an severse-scalingg phenomenon where performance seems degrade with model size. PROMPTBENCE (Zhu et al. (203)) provides set of automatically generated datasets to assess the adversarial robustness of instruction-tunel LLMs. Similarly, Li et al. (2023) evaluated L.LMs under adversarial instruction "attacks".
2

---

Under review as a conference paper at ICLR 2024
Other work has attempted to characterize when and how in-context learning (ICL) ie.. including a few examples in prompts wooks (Min et al., [2022] Wang et al.. 2023 Dai et al. 2022; Akyürek
2022; Yu et al., 2022h. ICL is a form of prompting orthogonal to the present effort, are et al. primarily interested in the zero-shot adaptability of instruction-tuned L.MSs.
models are to instruction perturbations (e.g., dropping words). This is qualitatively in line with our findings. Our work differs in important ways from this coincident research: (1) We provide much more comprehensive analysis of robustness: Gu et al. (2023) considered only T5 instruction-tuncl on a single dataset, whereas we evaluate three LLM families (and varying sizes of each) using five instruction tuning datasets, and we evaluate over 80 test tasks in all (Gu et al. 21.3 considered only 12); from this we identify representational similarity as a source of fragility. (2) On the basis of this analysis. we propose and evaluate a new approach to improving the robustness of instruction-tune models: Gu et al. 22233) offered no mechanism to improve robustness.
In work contemporaneous to ours, Gu et al. (2223) investigated how robust instruction
Improving instruction-tuning. Past work has also sought to improve instruction-tume tunnnn in various ways. One means to do is to instruction tune based on human feedback (Ouyang et al. 2022] Glaese et al, 2022 Bai et al., 2022, Nakano et al.) 2021 Zhang et al.] (023b). This tends to improve open-ended model responses but degrade performance on downstream tasks. Another strategy is to leverage existing resources to automatically generate instruction-tuninn
datasets at scale. For example, Gaanp et al. (2022a) use LLMs generate instructions, inputs, and outputs and use these to improve their own cnstructionfollowinng capabilities. In similarly meta vein, Zhou and colleagues 2022] propose using LLMs to engineer prompts. Finally, Ye al.) (2022) propose "flipping" the standard task by tasking LLMs with generating inxtructions, given an input and label.
INSTRUCTION DATASETS
Evaluation benchmarks. We evaluate a set of instruction-tuned models on two large benchmarks- MMLU (Hendrycks et al. (020) and BIG-BENCH (Srivastava et al.] 2022)-and summarization datasets. MMLU multiple-hoice question-answering benchmark comprising 57 tasks requiring expert knowledge. domains; we use the BIG-BENCH LITE subset, and include only QA, multi-class, binary classification tasks, and translation datasets, yielding 15 tasks in all.
is a collaboratively built benchmark of 204 diverse tasks from various
Collecting new instructions from NLP researchers. We aim to evaluate instruction- models when they are provided instructions that are appropriate for given task but superficially different from instructions observed in training. To this end we enlisted NL.P researchers (graduate students) to compose novel instructions for the tasks considered; these particular instruction phrasings were therefore unobserved during instruction fine-tuning.
Briefly, we recruited 36 NLP graduate students working in NLP. All had at least some experience with instruction-tur
models and the downstream tasks included in the evaluation benchmarks. For all tasks in BBL and MMLU, 12 graduate students wrote instruction they would use for zero-shot inference with an instruction-une collection process in Appendix | We will release all 319 instructions collected for reproducibility and to facilitate further research on instruction-tune tuned model robustness.
model. We provide additional details on this instruction
EVALUATING THE ROBUSTNESS OF INSTRUCTION LLMS
4.1 MODELS AND DATA
We conduct experiments with publicly accessible model variants trained over three instruction collections (which provide observed task instructions): P3 (Sanh et al. 2021). Flan-2022 (Chung ct al. 2022). and Alpaca TTooi et al. 2023). We do not include Llama2-Chat (Touvron et al. 2023b because its instruction- tuninng data is not available. But we do offer a comparison between LLama--Alpace and ALama-Alppac in Appendix D (which suggests that better pre-training does not markedly improve robustness). We manually identified all instructions that correspond to the following task types: (a) Multiple-choice question answering (QA); (b) Binary classification (BC); (c) Multi-class classification (MC); (d) Summarization and; (e) Translation.
3

---

Under review as a conference paper at ICLR 2024
this task, you are given a multiple-choice question and you have to pick the
QA
correct option. Answer with option indexes (i.c., "A", "B", "C", and "D").
Q: {question] } A. [choiceA] B. (choiceB) C. [choiceC] D. [choiceD)
MC
Pick one category for the following text. The options are [options] [text]
(paragraph) Choose your answer: According to the above paragraph, the question "[question] is "(response)"?
BC
Table 1: Examples of observed instructions we collected for three general types of tasks.
OBSERVED INSTRUCTIONS
UNOBSERVED INSTRUCTIONS
Instruction Type
QA
MC
BC
Sum
Trans
Number of tasks
1
15
Flan
50
35
18
37
14
Instructions per task
20
10
Alpaca
20
20
11
20
Total instructions
20
140
P3
13
8
7
24
Table 2: Counts of instruction phrasings (unobserved and observed) we use for evaluations.
To evaluate model robustness to instruction phrasings, BIG- BIG-BENCH LITE (BBL.) along with the acquired set of novel instructions described above.
two large benchmarks: MMLU and
We include all 57 tasks from MMLU and 14 of 24 tasks from BBL use the same instructions for all tasks in the same category, taken from the instruction tuning datasets associated with each model. These instructions are general, e-g., the case of classification they request that the model consider an example with respect to categorization criteria and label space provided by the instance, and select an appropriate category (examples in Table TT. One can "mix-and-man and-match" such instructions so long as they are appropriate for the task type.
4.2 MAIN RESULTS
We present the main aggregated analysis results in Figure 2 and Table 3 The take-away here is that using instructions unobserved in training-but manually composed for the task at hand and so semantically appropriate-leads to considerable degradation in performance: On average, unobserved instructions reduce accuracy by over five points across models considered. disaggregated by task type; we observe that classification tasks are most harmed by use of novel instructions. We provide additional, more granular (dataset-level) results in Appendix K
reportss results
80
Observed
Unobserved
Ocserved Unobserved
MB U
RAI GA
-5.6
Paflerunne
35
30
107
10P
10P
10"
10F
10P
Flan-T5-3B
Flan-TS-11B
Alpacs-7B
Alpasa- 138
TO 110
Number
(a) Average zero-shot performance over tasks when using observed and unobserved instructions.
(b) Performances Flan-T5 using observed and un- observed instructions as a function of model size.
Figure Using novel but valid instructions time (phrasings unobserved in training) consistently degrades the performance of instruction-tuned LLMS (a). Scale does not necessarily fix this (b).
We present results for NLG tasks (translation and summarization) in Table 14
For these we use ROUGE (Ln 2004), BLEU (Papineni et al. 2002), and BERTScore (Zhang et al. 2019) as automatic
We exclude BBI exact-matching tasks, and 5 including tokens recognized by T5 (e.g., emojis).
4

---

Under review as a conference paper at ICLR 2024
MMLU Avg. Std.
BBL-QA Avg. Std.
BBL-BC Avg.
BBL-MC Avg. Std.
Overall Avg. Std.
Model
Std.
Flan-T5-3B
OBSERVED
48.1 (10.3)
59.0 (12.1)
66.5 (13.8)
(10.7)
57.3 (+1.7) 54.2 (15.1)
UNOBSERVED
47.5 (10.9)
56.0 (+7.3)
61.1 (=
52.1 (15.4)
Performance A
1 0.6
3.0
1 5.5
3.5
3.3.1
Alpaca-7B
OBSERVED
41.9 (10.6)
48.6 (12.8)
53.8 (13.4)
32.1 (12.2)
44.1 (12.3)
UNOBSERVED
39.7 (12.2)
45.3 (16.5)
52.4 (=..)
(13.5)
38.5 (14.7)
Performance A
2.2
1 3.3
1.4
15.7
5.61
T0+ 11B
OBSERVED
48.3 (10.9)
54.1 (14.1)
66.1 (1221)
42.0 (12.1)
52.6 (12.3)
UNOBSERVED
48.5 (10.9)
54.7 (13.7)
54.7 (=4.3)
41.4 (12.4)
49.8 (12.8)
Performance
0.2
0.7
1114
J 0.6
2.8
Flan-TS-11B
OBSERVED
53.2 (4.02)
67.9 (41.8)
(4-6.0)
58.7 (+.0.)
61.4 (4:2.1)
UNOBSERVED
52.7 (40.8)
64.6 (48.5)
63.6 (=6.1)
55.9 (45.5)
59.2 (+5.2)
Performance A
0.5
334
2.0
2.8
2.2
Alpaca- 13B
OBSERVED
47.8 (+0.5)
53.9
(+2.2)
57.9
(+4.8)
36.7
(+1.8)
49.1
(42.3)
UNOBSERVED
47.0 (+0)8)
51.7
(+5.7)
54.1
(=5.)
22.7
(47.5)
43.9
(+14.0)
Performance A
0.9
2.2
3.8
11.0
5.21
Table 3: Results using observed and unobserved instructions across benchmark tasks (grouped by by 10+ points-when one uses (UNOBSERVED) instruc- tions, suggesting that instruction-tuned models not particularly robust. BC, MC, and QA stand for binary classification, multi-class classification, and question answering, respectively. denotes a statistically significant difference (p 0.05) over datasets for given model under paired t-test.
type). Performance degrades-
metrics. To ensure a fair comparison, we evaluate only examples comprising <112 tokens (including instructions). We use 61.5% test set data for XSUM (Narayan et al. 2018 17.6% for CNN/DM Nallapati et al. 2016), and 4.0% for MEDIASUM (Zhu et al. 2021).
Held-in Datasets
Held-out Datasets
CNN/DM
XSUM R2 Avg. Std.
MEDIASUM
WMT19 FR-DE
Model
R₂
BN Avg. Std. Avg. Std.
R₂
Avg. Std.
Flan-T5-3B
OBSERVED
41.5 (#0.1)
36.1 (#1.0)
23,4
(#6.2)
19.2 (+1.8)
UNOBSERVED
32.3 (#2.4)
35.7 (#1.1)
18.0 (+1.4
18.4 (#3.0)
Performance
9.1
0.4
5.4
1 0.7
Alpaca-7B
OBSERVED
19.2 (1:34)
12.4 (+1.3
(#1.6)
16.5 (16.8)
UNOBSERVED
16.8 (+0)
8.3 (+0.4)
9.1
(+007)
16.2 (+6.9)
Performance
1 2.4
4.1
1 0.7
4 0.3
Table 4: Results using observed and unobserved instructions on NLG tasks. We measure the performance gap on both held-in datasets and held-out datasets for instruction-tuning. Performance is again worse when unobserved instructions are used. Rt refers ROUGE-2 Fl and B, refers BLEU with n-gram. These are representative results; detailed results with more metrics are in Appendix B
4.3 DOES INSTRUCTION ROBUSTNESS EMERGE AS A FUNCTION OF SCALE?
We repeated all experiments from Table with Flan-T5 model sizes ranging from small (80M parameters) to XXL (1IB). Figure 2b 2b/seoss shows that the disparity between results achieved with observed versus unobserved instructions does not decrease with model scale, at least up to this point. That said, massive models (175B+) may offer improved robustness. However, reiterate that much of
5

---

Under review as a conference paper at ICLR 2024
the excitement about Iustruction-tuming that this technique appears to allow much smaller models to achieve results competitive with massive alternatives. We report more details in Appendix H
4.4 ROBUSTNESS UNDER IN-CONTEXT LEARNING (ICL)
2023 showed that LLMs are less sensitive to prompt variation when provided "few-shot" examples in context. We are focused on zero-shot, instruction-followii capabilities; but for completeness we re-ran all experiments in few-shot setting. We followed the sub-sampling strategy from 4.2 and we evaluate the same examples in the few-shot settings we did for the zero shot case. Given space constraints, we report the full results in Appendix C but briefly summarize our findings here briefly. (I) On average, ICL examples improve both performance (avg. performance) and robustness (i.e., performance with novel compared to observed instructions) slightly for most models and tasks. (2) For each task, the variance of performance and robustness significantly increases when more shots are provided, suggesting a sensitivity to the specific ICL examples selected. In particular, model robustness varies significantly with choice of ICL examples.
Recent work (Gu et al.
In sum, ICI, does mitigate the robustness issue of medium-sized LLMS we have highlighted, but this is unstable and depends on the particular selection of examples.
4.5 CLOSER LOOK AT INSTRUCTION ROBUSTNESS
Above we used general instructions to induce LLMs to perform tasks (Table Here we delve further into the performance degradation observed when using novel instructions. We report curious result: Incorrect but observed instructions outperform appropriate but unobserved instructions (Figure 3).
We come to this observation by evaluating the performance of Flan-T5-XXL XXL (11B) using six in- struction types over seven datasets from BIG-BENNCH This includes (variants of) two instructions observed in training: Closest is the instruction from the most similar task in the instruction-tune teatiinnontnntnggggg set; Incorrect is an observed instruction for a completely differení and inappropriate task (but which has these should not yield the desired behavior; Negated
the same desired output format)-intuitee is the same as closest, but we negate the instruction to indicate that it should not perform the task. For unobserved instructions, we consider: Task designer, the instruction (task prefix) provided by the author of the task in BIG-BENCH, and; Newly collected, novel and appropriate instructions we collected as described above. For reference, we also consider Nonsensical, which is a random "instruction" completely irrelevant to any task.
Examples Instructions languago classification)
Nonsensical
street raccoons demands you respand thei inquisiiio...
Closest Classity given news commentary
language whinh it
Obsaned
linguagea dissely secternse
Incorrect Ghen not of four words, generate the catepory that the words bellng to.
Words are exparated by commas. The possible cateporis are
Negated Clesslly the
into lanquage it not sentences
wittion
languages
Unsbservad
Task designer Given the serdence: selert the connect verguage smorg the choloee
Newly collected Gheen sentence, salect the conrect unguage among the chotoes. The possible answers are....
0.0
0.2
0.4
0.6
Average accuracy
Figure Incorrect but observed instructions perform better on average than correct but unobserved instructions. We report averages over benchmarks, but show example instructions on the right for a specific, illustrative task. We provide all instructions in the Appendix.
Figure reports average results for these variants. Consistent with our findings, using instructions unobserved in training degrades performance. Strikingly, here we also find that using inappropriate but observed instructions outperforms using appropriate but unobserved instructions. This indicates that instruction-tun -ouneetttot-teeeddddeeeee models at least the modestly sized ones have evaluated here- here-may overly rely on having observed instructions in training, and not generalize to new instructions and phrasings we might hope. We provide full set of disaggregated results in Appendix B.2

---

Under review as a conference paper at ICLR 2024
4.6 ROBUSTNESS AND "SEMANTIC" DISTANCE
One observation in 4.2 is that performance on MMLU is less affected by using unobserved instruc- tions. MMLU is a benchmark with 57 QA tasks covering different domains; these tasks all share similar form of input-output (question, four choices answer). During instruction collection, we treated all tasks in MMLU as general QA task and asked NLP researchers to write general QA instructions. We therefore hypothesize that these instructions comparatively similar to observed instructions, and this in turn explains the relative robustness in this case.
Figure and Table 5 support this hypothesis. For each instance (instruction plus example), extract the representation at the penultimate layer for the first decoded token. We use tSNE (Van der Maaten Hinton| 2008) visualize these representations of observed and unobserved instructions instances in MMLU and BBL. Figure 4 shows that for MMLU, the unobserved instructions we collected are quite similar the observed, while there is a greater separation between unobserved and observed instructions in BBL. Table 5 further quantifies this phenomenon: We report the average (2 distance between representations of unobserved instructions and those of their nearest observed counterparts. MMLU unobserved instructions are, average, closer to the nearest observed instruction; this correlates with the smaller observed performance drop.
MML
BOL
Observed Unobserved
Figure 4: ISNE plots of representations for the first decoded tokens of 300 randomly sampled examples from MMLU and BBL with Flan-T5 (XXL). Embeddings of observed and unobserved instructions for MMLU are similar, while for BBL they are quite different. This result holds across most but all models considered: See Appendix ffor visualizations over all models.
We plot mean performance degradation (as %) as a function of average similarity between the similarity of the first decoded tokens (following unobserved instructions) and the same for the most similar observed instruction. The negative slope implies the intuitive relationship: Instructions that are dissimilar (in terms of model representations) correlate with poorer performance. However, the relationship is relatively weak, yielding slope estimate of -0.2 (p =0.08).
Dataset
Avg. A22
Avg. A Acc.
MMLU
19.8
-0.5
BBL-QU UBL-QA
37.9
-3,4
BBI-BC
25.3
-2.0
BBI-MC
26.1
-2.8
Figure 5: Average degradations in per- formance versus "semantic" distance while using unobserved instructions. In general, larger semantic distances correspond to increased degradation.
Table 5: Average degradations in performance for
four categories (Flan-T5-XXL)) MMLU has mini- mal average distance, indicating smaller distribu- tion shift; we hypothesize that this is what leads to the relatively small degradation performance.

---

Under review as a conference paper at ICLR 2024
5 ALIGNING REPRESENTATIONS OF EQUIVALENT INSTRUCTIONS
Based on the preceding analysis, we hypothesize that that to be robust instruction-folloving LLMs should induce close internal representations for instructions corresponding to similar tasks. Here we propose simple approach to explicitly realize this aim: We introduce a term in the objective that encourages the model to yield similar predictions (and, correspondingly. similar representations) for the same input when provided distinct but semantically equivalent instructions.
Specifically, we add soft embedding parameters with dimensions Rdx;;; this is equivalent to adding novel tokens (with embedding dimension d) as prefixes inputs (preceding instructions). The intuition is to push representations for semantically equivalent tasks close together: To this end, fine-tune the soft embeddings (Li & Liang 2021) under an additional loss term: The KL-divergence LKL of the output probabilities between a reference instruction for a given task and a paraphrased (effectively equivalent) version of the same. We combine this with the standard cross-entropy loss and fit only the introduced soft prompt parameters under this objective (Figure 6 Here À is loss- leighweig
o and of are the distributions over the vocabulary induced by the model with paraphrased instruction i and the reference instruction token position ;
CB
distiibtiinns vocabulary
L = (1- A)LCE + ALKI
LLM
Cu..
KL (L (1)
if
Bolect the best answer
Pick the conpec
0.0h Softmax (p.
).p)
RI
original (rafenenoa) Instruction
paraphrased instructon
parameters
Figure 6: Schematic depiction of the proposed instruction alignment method (left) and associated loss terms (right). Dotted (red) lines indicate packpropagttio; we update only the soft prompt parameters, which we show yields performance superior to fine-tuning all model parameters.
Optimizing for the above objective requires similar instructions for each task in the training data; we generate these automatically as follows. For the instruction-tunde dataset, we sample a small amount of training data to use for representation alignment. We paraphrase these reference instructions using GPT-4. For the Alpaca collection, we randomly sampled 1k tasks, paraphrased them with three prompts, and collected the top three candidates under temperature 0.5. For the Flan collection, we randomly sampled Ik instances from the mixture with 3 prompts with greedy decoding (see B.3)
For fine-tuning, we create instances for each example by pairing them with every distinct instruction available for the corresponding task. We then form batches by including one instance featuring the original instruction, and the rest comprising paraphrased instructions.
RESULTS
We implement the proposed method to push together the representations of instructions for two models: Flan-T5-XL. (3B) and Alpaca (7B). After training with the added prefix, we reproduce the main results from Section 4.2 Then, following the approach in Section 4.6] we take the average distance between observed and unobserved instructions before and after. Thols that this brings observed and unobserved instruction representations closer together. The similarity is most increased in the case of the biggest accuracy gain, supporting the hypothesized mechanism of improvement. We compare the canonical versions of these models trained in the usual way (the same evaluated in Table 3 to variants fine-tuned with the proposed approach. We ablate components of our method to tease out the contributions of the extra parameters and closer representation. Specifically, we
We pad instances such that the lengths in a given batch are effectively equal; the sum is therefore from to the length associated with the current batch, we omit this for simplicity.
8

---

Under review as a conference paper at ICLR 2024
Dataset
Closest Distance Before
Closest Distance After
A Acc. Improvement (%)
MMLU
22.2
21.3
0.3%
BBL QA
22.4
23.0
0.4%
BBL BC
30.1
27.9
4.2 %
BBL MC
26.0
24.6
+ 0.3%
Table 6: Average distances before and after soft prompt alignment with Flan-T5- T5-XL.
consider variants where we: Fine-tune all model parameters the additional, automatically generated instruction paraphrases (FT); impose the new KL loss term (again fine-tuning all model parameters; FT+KL); introduce the additional soft prompt parameters and fine-tune the paraphrase instances, but without KL (PT); and then the full proposed strategy, which introduces the soft prompt parameters and optimizes them for the loss augmented with the KL term (PT+KL).
MMLU
BBL
Model
OBS.
UNOBS.
Avg.
OBS.
UNOBS.
Avg.
FLAN-T5-33
48.1
47.5
47.8
56.1
51.9
54.0
39.4 (-8.7)
40.1 (-7.4)
(-8.0)
(-7.9)
42.3 (-9.2)
45.3 (-8.7)
FT+KL
41.8 (-6.3)
43.6 (-3.9)
45.9 (-1.9)
47.7 (-8.4)
43.1 (-8.8)
45.4 (-8.6)
PT
48.1 (+0.0)
47.6 (+0.1)
47.9 (+0.1)
55.9 (-0.2)
52.1 (40.2)
54,0 (+0.0)
PT+KL
48.1 (+0.1)
47.9 (+0.4)
48.0 (+0.2)
55.9 (-0.2)
53.7 (+1.8)
54.8 (+0.8)
ALPACA-7B
41.9
39.7
40.8
47.6
42.9
45.3
(-1.6)
39.1 (-0.6)
(-1.1)
44.4
42.1 (-0.8)
43.4 (-2.0)
FT+KL
39.7 (-2.2)
40.2 (+0.5)
40.0 (-0.8)
(+2.0)
42.8 (-0.1)
44.2 (-1.1)
PT
42.1 (+0.2)
40.0 (+0.3)
41.1 (+0.3)
47.5 (-0.1)
43,0 (+0.1)
45.3 (+0.0)
PT+KL"
42.4 (+0.5)
41.8 (+2.1)
42.1 (+1.3)
47,9 (40.3)
46.6 (+3.7)
(+2.0)
Table 7: Results and ablations of the proposed soft prompt alignment method. Ablated versions use the augmented set with paraphrased instructions. FT refers to fine-tuning (with teacher-forcing) on this additional data; PT denotes prefix tuning (i.e., introducing soft prompt parameters); KL refers to the alignment objective that we proposed above. Using all components together yields the best performance, especially unobserved instructions; this improvement is statistically significant (p < 0.05) for both models under separate paired t-tests. We report variances in Appendix
We report results in Table 7
The proposed soft prompt alignment strategy (PT+KL) yields consistent improvements across the tasks and models considered and especially improves performance on unobserved instructions, as anticipated.
Two observations: (1)
(2) Simply fine-tuning with extra parameters or data (e.g., the newly collected paraphrases) on the same training tasks does not improve overall performance and robustness.
CONCLUSIONS
Instruction-tunee LLMs have emerged as promising means of achieving zero-shot performance with smaller models that are competitive to that observed using much larger LLMs. We characterized the robustness of such models with respect instruction rephrasings. We collected manually composed instructions from 36 graduate students in NLP across 75 tasks, and we evaluated different families of instruction-tun LLMs (Flan, Alpaca, and TO) when provided observed and unobserved instructions (seen in training and not, respectively). We found that using the latter consistently degrades model performance, indicating that models are unduly sensitive instruction phrasings.
Our analysis suggested that an issue may be the internal representations of instructions: Equivalent instructions should be near each other in latent space. On this hypothesis, we introduced a simple mechanism intended to improve the robustness of instruction-tunne LLMs. This approach entails introducing an additional loss term that penalizes the model for inducing dissimilar distributions when using (a) paraphrased instructions opposed (b) a reference instruction for the same task. We found that training under this objective consistently improves results and mitigates the degradation observed when previously unobserved instructions used.

---

Under review as a conference paper at ICLR 2024
8 ETHICS STATEMENT
This work does not have an explicit ethical dimension, but we acknowledge that all LLMs are likely encode problematic biases; currently unclear how instruction-tuning might interact with these, and we think this is a topic worth pursuing in future work.
REPRODUCIBILITY STATEMENT
All the results reported in the paper are reproducible. We submit the code and include all the implementation details in Appendix We present the disaggregated results in Appendix K All the instructions used in the paper are formatted and listed in the supplementary materials.
REFERENCES
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algo- rithm in-context learning? investigations with linear models. arXiv preprint arXiv:22 15661, 2022.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0582
2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023//1 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani. Siddhartha Brahma, et al. Scaling instruction-iietuuee language models. arXiv preprint arXiv:2210. 1416, 2022.
Damai Dai, Yutao Sun, Li Dong. Yaru Hao, Zhifang Sui, and Furu Wei. Why can gpt learn in- context? language models secretly perform gradient descent as meta optimizers. arXiv preprint 1.. 2022.
Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment dialogue agents via targeted human judgements. arXiv preprint 22X1v:2209.44375 2022.
Jiasheng Gu, Hanzi Xu, Liangyu Nie, and Wenpeng Yin. Robustness of learning from task instructions. 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint N.XY:22009.03300. 2020.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasumuru, Todor Mihaylov, Dániel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. OPT-IMI: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint 22 2022.
Joel Jang, Seonghyeon Ye, and Minjoon Seo. Can large language models truly understand prompts? case study with negated prompts. In Transfer Learning for Natural Language Processing Workshop, pp. 52-62. PMLR, 2023.
10

---

Under review as a conference paper at ICLR 2024
Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. Unifying question answering, text classification, and regression via span extraction. arXiv preprint 0xXiv1904.009286, 2019.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. Unifiedqa: Crossing format boundaries with single qa system. arXiv preprint arXiv:2005.00700 2020.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint 1rXiv:1910.09700 2019.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101 10090000,, 2021.
Zekun Li, Baolin Peng, Pengcheng He, and Xifeng Yan. Do you really follow me? adversarial instruc- tions for evaluating the robustness of large language models. arXiv preprint arXiv:2308. 10819, 2023.
Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81,2 2004.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Tay, Denny Zhou, Quoc Le. Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301. 13688, 2023.
Ilya Loshchilov and Frank Hutter. arXiv:1711.05100 2017.
Decoupled weight decay regularization. arXiv preprint
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv preprint 1rXiv: 2018.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202. /2837, 2022.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. arXiv preprint drXiv:2104.0877, 2021.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint 12.0033,, 2021.
Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. Abstractive text summarization using seuuencc-to-eqquene rnns and beyond. arXiv preprint
2016.
Shashi Narayan, Shay Cohen, and Mirella Lapata. Don't give me the details, just the sum- mary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint 100.1880808745 2018
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, el. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 2022.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-555. 2020.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Amaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv: 10.02/0008000 2021.

---

Under review as a conference paper at ICLR 2024
Aarohi Srivastava, Abhinav Rastogi. Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.0460 2022.
et al. Beyond the
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang. and Tatsunori B. Hashimoto. Alpaca: A strong, replicable instruction-folloinng model, 2023. URL http:::/crtt..ttaatteee sttaffor..cu//222////1111//aaaa
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ama Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302. /3971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint urXiv:2307.092 092888 2023b.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(1 (), 2008.
Xinyi Wang. Wanrong Zhu, and William Yang Wang. Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning. arXiv preprint arXiv:2301. 1196,, 2023.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint 1.11:2212.11066 2022a.
Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv preprint arXiv:2204.07705, 2022b.
Albert Webson and Ellie Pavlick. Do prompt-based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.noo 167. URL http:///aaaaablllggggg//////////
Seattle, United
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprins (tt::22009.0655.. 2021.
Zhiyang Xu, Ying Shen, and Lifu Huang. Multinstrruc: Improving multi-modal zero-shot learning via instruction tuning. arXiv preprint arXiv:2212. 2022.
Seonghyeon Ye, Doyoung Kim, Joel Jang, Joongbo Shin, and Minjoon Seo. Guess the instruction! making language models stronger zero-shot learners. arXiv preprint arXiv:2210.02969, 2022.
Ping Yu, Tianlu Wang, Olga Golovneva, Badr Alkhamissy, Gargi Ghosh, Mona Diab, and Asli Ce- likyilmaz. Alert: Adapting language models to reasoning tasks. arXiv preprint arXiv:2212.08220 2022.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 130b: An open bilingual pre-trained model. arXiv preprins arXiv:2210.02414. 2022.
Kai Zhang, Bernal Jiménez Gutiérrez, and Yu Su. Aligning instruction tasks unlocks large language models as zero-shot relation extractors. arXiv preprint arXiv: /199, 2023a.
Tianjun Zhang, Fangchen Liu, Justin Wong, Pieter Abbeel, and Joseph E Gonzalez. The wisdom of hindsight makes language models better instruction followers. arXiv preprint arXiv:2302.05206. 2023b.
12

---

Under review as a conference paper at ICLR 2024
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint 1rXvv:190.00967, 2019.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910, 2022.
Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng. Mediasum: A large-scale media interview dataset for dialogue summarization. arXiv preprint arXiv:2103.06411 2021.
Kaijie Zhu, Jindong Wang. Jiaheng Zhou, Zichen Wang. Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, al. Promptbench: Towards evaluating the robustness of large language models on adversarial prompts. arXiv preprint 2rXiv:2306.0458,, 2023.
13

---

Under review as a conference paper at ICLR 2024
A LIMITATIONS
This work has important limitations: For example, we only evaluated "mid-sized" models (<20B parameters), is unclear our findings would generalize to much larger instruction-tuned models. (However, note that instruction tuning has been most promising for smaller models.) We also considered only a somewhat limited set of task types: QA, multi-class and binary classification, and then summarization and translation for generation tasks.
B EXPERIMENTS
To ensure reproducibility, we provide all details regarding our evaluation of the robustness of instruction-tuned LLMs.
B.1 MAIN RESULTS
LLMs sometimes generate outputs that are correct but different from a (natural language) target. Therefore, we predict answers according to "multiple-chiiee" grading suggested by BIG-BENCH, by which we take the logits score and argmax over all the possible choices to obtain the prediction. In most cases, this approach yields the same accuracy as using exact match for evaluation. Here are the configurations for all the models we evaluated.
Models
Node Type
Precision Batch Size
Hours CO2 emission (KG)
Inference
Flan-T5-Small
V100-SXM2-32G
FP16
128
64
4.0
Flan-T5-Bass
V100-SXM2-32G
FP16
128
128
8.1
Flan-T5-Large
V100-SXM2-32G
FP16
32
256
16.2
Flan-T5-XL.
V100-SXM2-32G
FP16
32
512
32.3
Flan-T5-XXI.
RTX-A6000-46G
BF16
8
600
37.8
T0++
RTX-A6600-46G
BF16
128
8.1
Alpaca-7B
1100-SXM-800G
BF16
16
160
13.4
Alpaca-13B
A A000-SXM4-00G
BF16
8
192
16.1
Training
Flan-T5-XL A A100-SXM4-80G
BF16
256
256
21.5
Alpaca-7B
A100-SXM4-80G
BF16
128
80
6.7
Estimated Total CO2 Emission (KG)
164.2
Table 8: The configurations for evaluating different instruction-tuned LMs. The CO2 emission is estimated by Lacoste el al. (2019) Total estimated emissions are about equivalent to a 679 KM drive by an average ICE car.
B.2 "CLOSER LOOK" EXPERIMENT RESULTS
We provide detailed results that correspond to the aggregated values we reported in Figure 3
Table shows that among the benchmark datasets that we experiment with, the "Incorrect" instructions outperform the "Collected" instructions. And then for the other 2 datasets the performance of "Incorrect" instructions is comparable (-1%) to the "collected" ones. This indicates that the instruction- -ooluoiion--ollowingg ability of the LLMs is not robust to the fine-grained semantics within the instructions, as it only about the high-level requirement of the prompt. (classification, QA, etc.)
B.2.1 MOTIVATION AND INSTRUCTION COI.LECTION
Here, we report the detailed procedure and motivation for collecting the instruction templates for the six settings experiment with.
14

---

Under review as a conference paper at ICLR 2024
Observed
Unobserved Task Designer
Control
Dataset
Closest
Incorrect
Collected
Negated
Nonsensical
Intent Recognition
93.6
93.1
94.1
28.0
40.7
94.7
10.3
11.0
10.6
16.5
17.2
Empirical
39.2
41.6
37.6
28.1
30.9
37.4
Judgments
+0.8
+6.3
#1.7
+3.5
+2.5
Conceptual
78.0
78.9
75.3
11.2
63.7
58.3
Combinations
116
+0.55
+3.3
422.1
42..6
Language
38.9
29.3
28.8
36.9
12.4
27.6
Identification
+0.3
+5.0
+5.8
+0.5
+0.5
Logical
56.9
49.4
52.8
11.8
34.4
Sequence
16.6
16.1
15.
16.9
+5.9
Crash
53.6
50.0
50.5
28.6
43.7
63.2
Blossom
+22.8
+5.3
2.2
+6.2
+1.4
Epistemic
62.8
59.3
58.1
65.5
49.5
60.2
Reasoning
1.29
13.4
11.7
14..6
11.3
60.5
57.4
56.8
30.0
39.3
Overall
56.4
-.1
+2.2
11.
+22.
+2.3
Table 9: The detailed results of "A Closer Look" experiment. Using "Incorrect" but observed instructions in most cases outperforms correct but unobserved instructions ("Closest" and "Collected", respectively).
Closest Instructions In the main results reported in 4.2 explicitly the observed instruction templates that are general enough to be used for all the task belongs to the same categories. We show that even with such generality, the observed instructions template still outperforms the unobserved ones by a large margin. Next we evaluate whether the observed instructions for similar trained tasks perform better than the general ones. For each task, we collected the closest instructions by manually going through the instruction tuning set, and selecting suitable instructions from the train tasks that are similar to the evaluation tasks, e.g., instructions from WMT16 (trained) for WMT19 (evaluate).
Incorrect Instructions As an extreme opposite setting to Closest Instruction, we investigate how the models perform when the given instructions from completely different tasks, but have similar input- format (e.g., language classification and movie genre classification). This setting should not work if the model "understands" instruction semantics. For each task, collect the closest instructions with the following procedure. We again manually go through the instruction tuning set and filter of tasks that belongs to the same category (QA, summarization, Multi-class classification, and so on). Among tasks in the same category, we identify instrctions from the most different task.
Negated Instructions Following the design of Jang al. (2023), we use negated instructions as an additional probe to assess whether the model understands instructions or mostly recalls back to the training data. For each task, we collect negated instructions by:
(1) Taking all of the Closest Instructions for this task.
(2) Insert a negation word (e.g., not) in the appropriate place of the instruction.
Task Designer Instruction For the benchmark BBL that we evaluate, for each task, it provides prompt template written by the task designer. This serves as gold unobserved instruction that guaranteed valid for this task.
Newly Collected Instruction This is the same the "Unobserved Instruction" that used throughout the paper. Following the exact same procedure Section 4.2 we collect 10 unobserved instructions for each task evaluate in this section.
15

---

Under review as a conference paper at ICLR 2024
Nonsensical Instruction This setting is used as baseline. We perform inference of the task with an instruction prepended to the input, that irrelevant and doesn 't provide any extra information. (e.g., "The council of street squirrel demands you response: Question: "..")
B.3 DETAILS ON ALIGNING REPRESENTATIONS OF EQUIVALENT INSTRUCTIONS
We conduct all training and ablation studies on A100s with 80GB memory. We kept the KL-Loss weight fixed 0.8, We trained both Flan-T5-XL. and Alpaca-7B with batch size of 4 and batch gradient accumulation of 4. We set the weight decay to 1e and the learning rate to 5e - 4 for all experiments. We use a prefix length of 10. We explore the choice of the prefix length and found no significant correlation with the performance and "semantic" distance between instructions.
MMLU
BBL
Model
OBS.
UNOBS.
Avg.
OBS.
UNOBS.
Avg.
FLAN-T5-3B
48.1 (+0.3)
47.5 (=0.9)
47.8 (+0.6)
56.1 (+5.8)
51.9 (49.6)
54.0 (+7.7)
FT
39.4 (10.5)
40.1 (=044
39.8 (#0.4)
48.2 (5566
42.3 (18.8)
45,3 (#7.2)
FT+KL
41.8 (:0.3)
43.6 (=0.2)
45.9 (10.3)
47.7 (14.2)
43.1 (17.3)
45.4 (45.8)
PT
48.1 (10.3)
47.6 (=0.8)
47.9 (10.6)
55.9 (16.0)
$2.1 (19.9)
54.0 (18.0)
PT+KL
48.1 (+:02)
47.9 (=0.4)
48.0 (+0.3)
55.9 (+4.4)
53.7 (+5.9)
54.8 (+5.1)
ALPACA-7B
(1.0.))
39.7 (12.2)
40.8 (11.4)
(12.8)
42.9 (15.5)
45.3 (14.2)
FT
40.3 (1.0.7)
39.1 (119)
39.7 (11.3)
44.4 (13.3)
42.1 (14.9)
43.4 43.4 (14.1)
FT
39.7 (1006)
40.2 (114)
40.0 (11.0)
45.6 (122)
42.8 (16.4)
44.2 (14.3)
PT
42.1 (+0.8)
40.0
22.6)
41.1 (+17
47.5
22.8)
43.0 (#6.1)
45.3 (+4.5
PT+KL
42.4 (+0.4)
41.8
(=1)
42.1 (#0)9)
47.9
22211
46.6 (#3.6)
47.3 -2.88
Table 10: For the simplicity of the main paper, we report the StD of the results in Table
To generate "equivalent" instructions for tasks from seed instructions, we prompted GPT-4 to produce paraphrases of given instructions. We detail our procedure for doing this for each model evaluated below. used different prompts these two instruction collections because, informal development, found them to produce better paraphrases (respectively). We emphasize that these are instructions from the training instruction-tuniin set, so this cannot construed as "cheating", although acknowledge that there is small amount of effort necessary ensure LLMs (GPT-4) produces "good" paraphrases at scale.
Alpaca To generate paraphrases of observed instructions in the Alpaca collection, sampled 1000 out of 52002 tasks at i.i.d. random and generated instruction paraphrases with GPT-4 via the following prompts.
"Paraphrase this sentence:nnnn CinitructioonPParapphit sentence:\n\n"
PPrraphrsee this instruction into longer sentenceellniin (instruction) J. New sentence:\n"
"You are given an instruction:Anknn (instrrutoo paraphrase into new instruction with equivalent meaning:\n'n"
Flan We first reproduced the held-in instruction-tuninn set of Flan-T5 with the pipeline. We randomly sampled 986 data samples from the generated data following the proportion reported in Chung et al. 2022). We generate paraphrases of the selected data with GPT-4 using the following prompts.
"Here's input utterance:\n'ni by only changing the instruction but leaving everything else the same.An Here's the new utterance:In'n"
laaree///nnninsuuuttttooo (instruction]ln In An Now, your task is to paraphrase the input
"You are given an utterance which is combination of task instruction and the actual input. Your job is to paraphrase the task instruction and leave the input unchanged. Here's the ut- terance to be paraphrased/InlN/ (iauiuninstuuttonn Antaln Now, generate the new utterance:Inlmln"
"You are provided with the utterance of a specific task and I need you to paraphrase it. The actual input, question, and examples in the task should not be changed. You
16

---

Under review as a conference paper at ICLR 2024
should only paraphrase the instructions. Task:\nnnnn (instruction iotinttiioonnnnnnnnnnnnnn The paraphrased utterance:Inlnln"
On informal and subjective assessment, these prompts yielded seemingly good paraphrases; we therefore did not invest time in further "prompt engineering" for this problem, given that our focus in this work not automated paraphrasing per se-rather, we aim only to collect "reasonable" paraphrases that allow us to realize the proposed objective which looks to push representations of semantically similar instructions relatively close together; GPT-4 readily provides outputs servicable in this regard.
B.3.1 QUALITY STUDY OF INSTRUCTION TUNING DATASETS AND PARAPHRASES
Here we investigate the quality and effectiveness of aligning instruction representations when para- phrases are generated under models aside from GPT-4. We do this to explore the sensitivity of approach to specific sources of instruction paraphrases, although we emphasize that these might also be manually composed (at an associated cost in effort, of course).
Training with More Instructions v.s. Inducing Closer Representations We show that the robustness of foollotioonfollooing LLMs could be improved by aligning the representations of semantically equivalent instructions. However, it remains unclear whether this approach is better than training new LLMs from scratch with the extra instructions. Here, we compare the performance and robustness of LLMs re-trained versus LLMs aligned with the new instructions.
Method
Avg Ace. (MMLU)
Avg Degr. (MMLU)
# Tokens
Parameters
Flan-T5-XL, (3B)
47.8
-0.6
Training w/ new paraphrases
47.7
-0.5
4.4B
3B
-Aligning new paraphrases
48.0
-0.2
390K
61K
Alpaca (7B)
40.8
-2.2
Trriinin w/ new paraphrases
41.3
-2.1
3.4M
Aligning w/ new paraphrases
42.1
-0.6
233K
122K
Table 11: Performance MMLU and BBL when the model is retrained from scratch (with the new instructions) and when the model representation is aligned with the new instructions. # Tokens refers to the total number of tokens used in the process; #Parameters refers to the total number of trainable parameters the process.
Table II shows that even with significantly larger training text and trainable parameters, re-training the model from scratch still underperforms, aligning the representations of equivalent instructions.
Aligning with Paraphrases from Different Sources In Section 5 we adopt GPT-4 as the source to generate semantically equivalent instructions. We agree that whether results hold with alternative sources of paraphrases a worthwhile scientific question. We have replicated the experiment from Section with two other reasonable sources of paraphrases: Alamm2-13B Touvron et al. (023b) and texc--aVinci-003 Ouyang et al. (022).. We reproduce the experiment in Table 7 with Flan- Flan-T5-XI. TI-
MMLU
BBL
Source Model
OBS.
UNOBS.
Avg.
OBS.
UNOBS.
Avg.
Text-DaVineci Llama2-13B
48.1
47.7
47.9
55.4
53.0
54.2
48.0
47.6
47.8
55.4
50.8
53.1
GPT-4
48.1
47.9
48.0
55.9
53.7
54.8
Table 12: The performance of the models after aligning representation with soft prefix (PT+KL in 7) GPT-4 generated paraphrases improve the performance and robustness of the cnstruction-followiin LLMs better than the other two sources.
We take a more in-depth look at the quality of paraphrased instructions generated by GPT-4, Davinci- 003, and Llama2- 13B. We observed that the performance improvement using the paraphrases as
17

---

Under review as a conference paper at ICLR 2024
equivalent instructions is correlated with the diversity and quality of the paraphrases. For each source of the paraphrases, we measure the average token length generated, as well as the similarity (ROUGE-2) between the seed instruction and its paraphrases.
Table 3 shows that GPT-4 and Da Vinci produce relatively similar paraphrases. The ROUGE-2 score compared with the original instruction is significantly higher than GPT-4 and Da DaVinci-003, implying a lower diversity and significance the equivalent instruction set.
Source Model
Avg. Token Length
Avg. ROUGE-2 w/ seed instruction
Text-DaVinci-003 Llama2-13B
154
(+76)
47.2 (#39.5)
286
(+:201)
55.3 (#:40.0)
GPT-4
165
(176)
45.8 (#:391)
Table 13: The statistics of the paraphrased instructions generated from different LLM sources
B.3.2 EFFICACY OF ALIGNMENT TERM
In Table conduct detailed ablation study to show the efficacy of the KL term migitating the robustness issue. Comparing with prefix tuming. the proposed method introduces extra term to cluster the hidden representation produced by the equivalent instructions together, which results in 1.0 and 3.6 improvement for the unobserved instructions on MMLU and BBL respectively.
20.0
37.5
35.0
12.5
10.0
Stanng
Dataset
Figure 7: The average 12 distance between the unobserved instructions to their closest observed instructions. The result shows that the semantic distance between equivalent instructions is smaller for PT+KL in 13 out of 14 datasets.
B.4 NLG EXPERIMENT RESULTS
We report the full results four NLG datasets. We ROUGE-2 (F1) Lin (2004) and BERTScore
Zhang et al. (201)) as the automatic metrics for summarization; we use BLEU Papineni et al. (2002) and BERTScon as the automatic metrics for translation. T0++ [annh et (021) not evaluated for
translation tasks because it is not trained on any non-Enge non-English tasks.
C INSTRUCTION ROBUSTNESS WITH IN-CONTEXT LEARNING
We have been focused on zero-shot settings, but here we also report results achieved under In-context Learning (ICL.). repeat experiments from Section 4.2 on MMLUU with the one-shot and three-shot ICL below. Additionally, compare the effect of few-shot examples to proposed representation alignment method:
We observe that while few-shot prompting does indeed mitigate robustness issues to some degree, does so with high variance, compared to the proposed alignment technique. Furthermore, the selection of in-context examples also affects the performance and robustness of k-shot prompting. We repeat the Alpaca-7B 7B (Three-shots) experiment more times with random examples sampled from the test set and the standard deviation raises from 3.9 to 5.8, indicating the instability of the
18

---

Under review as a conference paper at ICLR 2024
Held-in Datasets
Held-out Datasets
Model
CNNIDM
XSUM
MEDIASUM
WMITS
Rg Avg. Snd
Rg
Avg. Std.
Arg. Std.
Avg. Std.
Ang. Std.
Arg. Std.
Avg. Std.
Avg. Std.
Flan 13 Small
Ouservezo
(+6.1)
95.2 (#0.2)
246 (+0.3)
96.1
16.0 (+1.4)
95.9 (#0.1)
(+0.5)
94.2 (#0.2)
UNOESERVED
(=5.6)
(#0.2)
23.0 (#1.3)
(40.0)
186 (#2.3)
95.9 (+0.1)
(#1.6)
(+1.0)
Performance
0.5
0.9
0.0
L 0.0
0.8
0.6
Fan 15-Base OnsuRvinn
27.7 (=0.3)
(+0.1)
29.4 (#0.2)
(+0.0)
16.3
(+0.8)
95.8
(#0.1)
(+1.0)
(+0.0)
UNDISHRVED
28.7 (#0.G)
96.0 (#0.0)
29.1 (40.5)
90.0 (+0.00
17.1 (#4.0)
95.8 (bD.1)
11.8 (#3.9)
94.5 (#0.44
Performance A
1.0
0.1
1.0.4
4.0.0
0.0
1.1.7
4.2.2
Flm- TS-Lage
OBseavin
39.6 (=1.0)
98.4
(#0.2)
32.9 (21.1)
95.9
(+0.1) 1)
20.5 (#2.5)
95.9
(#0.2)
189 (#2.3)
04.5 (#0.1)
UNOuSMaVin
34.3
(=3.9)
95.2
(=0.3)
32.1 (#1.4)
95.9 (#0.1)
18.7
(+1.5)
95.9
(#0.1)
16.2 (#5.4)
94.5 (#0.3)
Peeformance
Flm- TS-3B
OBSERVED
41.5 (#0.1) 953
(20.0)
36,1 (#1.0)
95.9 (40.0)
234 (26
96.0
(#0.1)
1922 (#1.8)
945 (#0.1)
UNOBSERVED
(#2.4)
(20.1)
35.7 (d111) (10.00
(#1.4)
95.7 (400.1
(43300 (40.1)
Performance
19111
0.0
0.4
0.0
0.3
0.7
0.0
Alpuca 7B UNOBSERVED
(20.6)
(#1.3)
(1:02)
929 (#0.5)
16.8
(=0.9)
933 (#0.5)
8.3 (#0.4) 933 (40.9)
9.1 (+0.7)
92.9 (40.6)
16.2 (+6.9)
98.7 (+110)
Performance A
2.4
1 0.3
141
1.0
0.7
0.0
0.3
0.3
13-118
OBSERVED
336
(10.2)
93.1 (40.0)
39. (#0.1)
95.9 (10.00
222 (42.3)
95.7 (10.1)
222 (1:0.4)
9466 (10000)
UNOnSHaVED
32.0
(=2.3)
95.1 (#0.1)
39.7 (#0.3) 0.3
96.9 (+0.0)
18.0 (+2.2)
95.8 (#0.1)
22.0 (+113) 94.8 (#0.0)
Performance
1.8
0.0
0.2
T0-118 38.1
OBSERVED
(+0.1) 95. (#0.0)
40.2 (#0.3)
v5.9 (#0.0)
18. (+4.8)
95.6 (#:.2)
UNDISLRVID
(=0.2)
(=0.0)
40.1 (+0.5) (#0.0)
(+0.8)
95.4 (#:0.1
Performaace
0.1
Alpaca-138
OBSERVED
(20
(13.0)
(20.3)
94.2
(25.8)
UNonSHEVIn
(=0.))
94.1
(=0.1)
8.0
(20.6)
(#0.1)
6.9 (#0.3)
94.5 (#0.3)
(#6.1)
(#0.0)
Performaacce A
3.3
0.2
2.3
0.1
0.2
1 0.2
1.4
0.0
Table 14: Full results using observed and unobserved instructions across NLG tasks.
Models
Avg. Acc. (Performance)
(Obs Unobs) Acc. (Robustness)
Flan-T5-XL
O0-soo
47.8 (#0.6)
0.6
1-shot
47.5 (1.0.)
0.2
3-shot
48.1 (11
0.2
0-shot Aligned
48.0 (+0.3)
0.2
Alpaca-7B
00-hoo
40.8 (+1.4)
2.2
---hoo
41.3
22.
1.1
-3-shot
41.7
(339
1.4
-O-shot + Aligned
42.1
(+003)
0.6
Table 15: The ICL experiments on MMLU with one-shot and threc-shot examples provided.
With the above results, we argue that the effect of in-context learning on robustness is a hybrid research question, in which the effect of examples needs to be studied independently.
method.
We provide the disaggregated results for few-shot experiments on MMLU and BBL in Appendix
K.
19

---

Under review as a conference paper at ICLR 2024
D LLAMA2 V.S. LLAMA: DOES BETTER PRE-TRAINING PROVIDE STRONGER ROBUSTNESS?
To provide a better understanding of what factors (aside from Iestruction-tuning correlate with the ability of LLMS to robustly follow instructions, compare the performance and brittleness of Llama2 Touvron et al. (2023b) and Llama Touvron et al. (2023a) when instruction-tuned with Alpaca Taori al. (2023). Llama2 Touvron et al. (2023b) is an LLM trained on relatively new mix of publicly available data. This pre-training corpus is about 40% larger than that used to train the original Llama.
D.I TRAINING DETAILS
We trained both Llama-7B BB and Llama2-7B on 4 A100 80GB GPUs with the Fully Sharded Data Parallel (FSDP) strategy. We adopted the exact same hyperparameters for training: We use the AdamW optimizer Loshchilov Hutter (2017) with learning rate of 2 x 10-5, We use cosine learning rate scheduler with warmup ratio of 0.03. We trained for a total of 3 epochs over 52k instruction and answer pairs collected Alpaca, with batch size 128. Llama2 and Llama have different max context window sizes; to simplify things set both to fixed max length of 512.
D.2 EVALUATION DETAILS
reproduce experiments evaluate the performance of Llama2-Alpaca on observed and unobserved instructions. We omit two held-in datasets that we evaluated to avoid data leakage. For each category of tasks (MMLU, QA, MC, BC, Sum, Trans), the average performance calculated, as well as the percentage delta of the performance between observed and unobserved instructions each task.
Following the same procedure in Section 4.2
D.3 RESULTS
Figure shows that although Llama2-Alpaca is consistently superior across all categories of tasks compared to Llama-Alppe Alpaca, the robustness (performance difference between observed and unobserved instructions) of the model does always improve. On summarization datasets, binary classification, and MMLU, the robustness of I.lama2-Alpact is better than Llama-Alpace whereas, translation datasets, QA, and multi-class classification, the cieructton-folloinng robustness worse. Hence, cannot be concluded that better pre-training provides stronger instruction-folloiing robustness after instruction-tuning
1.35
Model Jamr Abacs 70 Jana Alacc 79
6.8
1.25
Petomancc
0.75
53
0.5
V90U
Vandatinn Summarkaan Ratianaiiin 8BL-0
B5UMC
BAL 9C
0.00
NYW
Tansbiion Sunmor dotonn OS
2BLMC
BBLBC
Figure The overall results of Llama-Alpaca v.s. Llaaa2-Alpaca on the six aforementioned categories of held-out datasets. It could be seen that although the overall average performance of Llama2-Alpaca is slightly better than Llama- Alpaca (Left), there is no clear pattern that the performance difference between observed and unobserved instructions are narrowed down (Right).
20

---

Under review as a conference paper at ICLR 2024
E DIFFERENT (SIMILAR) OBJECTIVES INDUCE SIMILAR REPRESENTATIONS
Besides the KL objective reported in the main paper, we explored two other objectives that similarly aim to align representations for practically equivalent instructions. Our goal with these experiments to establish that the primary mechanism to improve robustness has to do with encouraging the model internally represent equivalent instructions similarly; the particular choice of objective for realizing this is not critical.
E. CONTRASTIVE LEARNING ON PENULTIMATE LAYER LOGITS
We use an extra contrastive term to push the inequivalent instructions away while aligning the equivalent instructions. Following the same batching strategy in Section [ one training batch contains seed task instructions and n paraphrases of each. While the other terms and loss remain the same as 5 we introduce the contrastive KL loss LK as:
N Σ ifr,.
LKI. N-1
IIKL(ol - (L(?!")10?")
Where for vocabulary distribution of each paraphrase úi V, is the seed instruction it is equivalent to, and o the seed instruction it isn't equivalent to.
E.2 MSE LOSS THE PREFIX ENCODER HIDDEN STATE
Instead of having to optimize the logits to induce closer representations for equivalent instructions, we tried to directly align the hidden representation of the soft prefix from the last encoder layer we they are concatenated to equivalent instructions. With the same batching strategy in Section [ we optimize:
Σ ifr
LMsE(his Try))
argmax @4km
N
Where hú and hy are given by:
h
Encoder([e) EEm((x)))
This method is effective and fast but cannot be used with decoder-only architecture LLMs.
E.3 RESULTS COMPARISON
In some sense all of these objectives are equivalent in that we are trying to induce closer representa- tions for similar instructions. The experimental results are reproduced here (we omit variances for simplicity):
Methods
Avg. Obs. Acc. (MMLU) Avg. Unobs. Acc. (MMLU)
FT + KL.
41.8
43,6
PT + KL
48.1
47.9
FT + Contrastive KL
41.9
43,6
PT + Contrastive KL.
47.9
47.7
MSE on EncoderHS
48.1
48.0
Table 16:
The results of the representation alignment experiment on Flan-T5-3B with two alternative objectives that incent closer distance between similar instructions.
Using these objectives to attempt to align instructions performs comparably (in terms of robustness) to the proposed soft prompt alignment strategy, but they are slower to converge than this method. Furthermore, the second approach is only suitable for encoder-decoder LLMs like Flan-T5.
21

---

Under review as a conference paper at ICLR 2024
F "CLOSER LOOK" AT SEMANTIC DISTANCES
we reported a rather peculiar result: Over all the tasks we evaluated, incorrect but observed instructions perform better on average than correct but unobserved instructions. Does our proposed method for improving robustness (by aligning representations in latent space) change this particular result? We re-run the experiment reported in Figure again with a model fit using the objective proposed in Section S
In Section 4.5
we report non-trivial improvement in settings. Interestingly, with the proposed method (PT+ the average performance when using instructions that are observed and incorrect (but which belong to the same genre of task) still performs better than the task designer result; that said, the gap in performance is slightly narrowed compared to the baseline (7.1 difference for PT + KL vs 8).
In Table 17,
F.1 NEGATION
Previous studies Webson & Pavlick (002)); Jang al. (2023) suggest the insensitivity to negation as an indicator of unrobust instruction-folloinng ability of LLMs. The results reported in Table and Section 4.5 also support such finding. Additionally, Table 17 shows that despite generally improving model's performance and robustness in other settings, the proposed method has failed to bring significant improvement (drop in accuracy) in the Negated setting. Hence, we slight tilt the objective to see if training on "pushing opposite instructions away" would result in a more robust instructionfollowiin ability for recognizing negation.
Similar to Section 5 we take the same 1k seed instances from the flan collection and prompt (with prompts) the GPT-4 14 to rewrite them with completely opposite instruction. All the other parameters for generation stay the same.
Naturally, instead of clustering equivalent instructions together, we want to push the negated instruc- tions away from the original instruction:
L= (1+ ACKI,
LKL
1
(
We train the model with the exact same hyperparameters Section for PT KL and run it on the same benchmarks with the six different settings. We report the results along with the original method, denoted as Negation Additionlly, We also report the results of jointly optimizing with negated and paraphrased instructions, denoted as Joint.
Model
Closest
Incorrect
Negated
Task Designer
Collected
Nonsensical
XLXL
Baseline
58.1 +1.8
52.2
+8.6
48.3
+3.8
44.2
55.3 +4.5
30.6 +:4.4
PT + KI.
60.5 +2.4
54.2 +7.2
47.1
+4.9
47.1
56.7
+3.9
38.0 +4.1
Neeattion
58.6 +2.0
52.7
49.2
43.0
+4.9
46.5
55.6
+4.3
35.7 +5.6
Joint
59.3 +1.7
52.8
+8.7
44.2
+4.5
46.6
56.1
+3.5
36.1 +5.1
Table 17:
We report average result of six settings over benchmarks. For Negated setting, a negation is applied to the instruction lower accuracy is desired. Despite the fact that in Negation! setting the model performs comparably with the baseline, its ability recognize negation in the instruction is significantly improved. Training with the joint loss does realize some of the advantages of both negative and paraphrase alignment methods. Compared with the standard fine-tuned baseline (no KL), see consistent improvements across all settings. But stronger improvements in the respective settings when we optimize for only paraphrases or negations alone.
22

---

Under review as a conference paper at ICLR 2024
G CATASTROPHIC FORGETTING
In 7 can observe that simply fine-tuning (FT) on the augmented paraphrase data significantly degredes performance when using both observed and unobserved instructions. We hypothesize that this may be an instance of catastrophic forgetting, where when we fine-tune the instruction-tunce model this way (on a relatively small set of paraphrased instruction instances) it compromises its ability to follow instructions more general.
We design a simple study test this hypothesis. Following the same setting of FT in Section 5 we finetune FL-NNTT-XLL (which is instruction tuned) and T5-XL. (which is not) on the same augmented data (4k paraphrases of task instructions) for 25 epochs, and evaluate the performance
on (A) the validation set sampled from the flan collection Chung et al. (2022), (B) MMLU with observed instructions, and (C) MMLU U with unobserved instructions.
35:38 Obeerved returtonn Indt
Flan T5 38 Obceneed Inctuction Liccosered intrrdioon
22
Losa (085.0)
14
Exroh
Epooh
Spoch
Figure Loss as a function of training epoch for FLAN-T5-31 and T5-3B, fine-tuned on the paraphrased instruction set. The first (left) plot shows the validation loss on the flan collection for both models; the second (middle) plot shows the validation loss on MMLU for T5-3B: the third (right) plot shows the validation loss MMU for FLAN-T5-33 The loss for FLAN- T5-3B increases almost monotonically (Line and Lines 4); this strongly suggests catastrophic forgetting. Coult be that the collected paraphrased instructions indicated by Line and Lines - suggests otherwise, i.e., that the data does contain signal for instruction tuning (on base model), but the set somewhat small and degrades performance when the Flan checkpoint is used for initialization.
simply not informative? The decreasing loss for T5-3B
What is happening at the representational level in these cases? In Figure 10 we compare the average similarity (analogous to Table between observed and unobserved instructions after 25 epochs of fine-tuning. Rather than closing the "semantic gap", fine-tuning FLAN- ---LL (without the proposed KL term) increases
Finetiing Eppcr
Winowhy Language
Dialog
Know: Sww.grrnggee
Logcal
Dataset
Figure 10: The average Distance between the unobserved instructions to their closest observed instructions on each datasets epoch I and epoch 25. This result indicates that simply fine-tuning the model with paraphrases does not help to align equivalent instructions.
23

---

Under review as a conference paper at ICLR 2024
H CLOSER LOOK TO ROBUSTNESS AS FUNCTION OF SCALE?
While we maintain our stance that the success of instruction-tuning for boosting the performance of modestly-sized LLMs is the most exciting aspect of this method, and therefore worthy on its own of study, we nonetheless agree that studying the relationship between robustness and scale of such models interesting.
However, we can draw valid comparisons only if we keep all other variables (model architecture, pretraining data, and instruction tuning set) fixed. Therefore, we are not able study Black-boxed models (like GPT-3.5 or GPT-4) using this approach. To include additional scale in our work, we therefore conduct the full experiments on different sizes of Flan-T5 ranging from 70M 113 as well as FIa (20B), and we report the change of delta performance (observed instructions performance unobserved instructions performance) in the table below:
Model
Number of Parameters
MMLU
QA
MC
BC
Flan-T5-Small
70M
-2.6 + 8.3
4.1 3.1
-0.4 + 1.4
-7.9 + 12.0
Flan-T5-Base
250M
0,0 10 0.9
1.6
1.2
4.3
1.5 8.3
Flan-T5-Large
770M
0.5
3.1
11,4
1.6 6.1
TI-X-
3B
0.5 0.7
4.6
3.2
5.5
-5-X-
11B
5.7
2.8 3.1
2.0 9.1
Flan-T5-UL2
20B
0.2 + 0.5
3.3
0.6 = 0.3
1.9 1,9+7
Table 18: The aggregated result of the delta performance (bserved instructions performance unob- served instructions performance) as the Flan-T5 model scales. The results above (aggregated by task category) show that there is no apparent relationship between model size and the delta performance (our measure of robustness).
More scientifically, we perform linear regression on the size of the model v.s. (observed unobserved performance) over data points from all datasets, this yields slope of 4.45 x 10-13 with a p-value of 0.087 (R² is 0.002). This suggests that model size may weakly correlate with increased robustness, but this relationship not significant the p 0.05 level, and the relationship is tenuous. In other words, scale does seem to improve robustness, but only very moderately.
24

---

Under review as a conference paper at ICLR 2024
I REPRESENTATIONAL SIMILARITY AND MODEL PERFORMANCE
We re-generate the visualization from Figure for all T3 model sizes Figure. The qualitative resulttrrerrssenatiions of tokens following observed instructions are generally dissimilar from those following unobserved instructions-remain largely consistent, although it is less pronounced, e.g.. for XL (in particular here, the MMLU samples are not as entangled as we might expect).
MNLU Flan XXL
BBL Flan XXXXL
Observed Unobserved
MHLU Plan XL
BBL Plan T5
MHILU TIaN-TS-LNE
DOL TIAN-TS-LI9
MMLLU Ran-TS-Base
BBL Blae
MILU Har-T5--mal
8SL Har-T5-Smal
Figure We reproduce Figure from the main paper over all T5 model sizes.
25

---

Under review as a conference paper at ICLR 2024
J COLLECTION PROCEDURES AND SURVEYS
We recruited in total 36 annotators to contribute diverse human-written instructions for various NLP tasks. These were graduate students in NLP who volunteered write instructions following an open request. We explicitly asked volunteers not to spend more than 30 minutes on this task. We estimate that the maximum time spent by any one individual on this task was one hour, and most were (much) quicker. In light of the minimal workload per individual being done on strictly volunteer basis, we did not offer compensationl; we emphasize that it was made clear upfront to volunteers that this quick amount of work would be uncompensated.
The entire annotation process was done online (via shared Google Drive directory) with detailed instructions provided. We reproduce the template of the invitation letter, annotation instructions, and annotation form below:
XXX!
Thank you for helping me with my rescurch! may take up to 30 minutes of your time, and your partcipation deeply thanked and will be acknowledged the final research paper.
The lusks/settings/IDD of your instructions are:
MMLU General Zero-shot-
MML U General- Few-shot-
Hindu Knowledge- Zero-shot
Hindu Knowledge Few-shot-
Known Unknowns Zoro-shot-
Known Unknowns- Few-shot-
Novel Concepts Zero-shot
Novel Concepts Fow-shot
winowhy Zero-shot-
winowhy- Fow-shot 1
BBQ-Lite Zero-shot-1
BBQ-Lite - Few-sbot
Strang Storics Zero-shot-
Srange Stories Pow-shot-
Emoji Movic Zero-shot-
Emoji Movie Few-shot-
enter the google does, click on this link: http://oxx.coo//xs Be read the instructions dacx the front page for detailed instructions!
It is optimal that you can get dome before Iay. If you have any questions regurding any the procedures, please foel free
anylime for clarification!
Thank you!
Figure 12: Invitation note to participant
26

---

Under review as a conference paper at ICLR 2024
express my apprecistion belping my project again!
Background
This research aims to evaluate the sobustness of the instruction-tuna Language Models (LMs) with respect to the variation of instructions in zere-shot or few-shot sottings. commonly acknowledged that multinak instruction tuning on language model improves zero-shon and few-shot ahility. provide inforence time.
and genemlize unseen instructions
For instance, use this instruction (prompt) as the Prefix:
"Complete this code written in Java SE11
to the actual code, I want complete, the LM can understund the task and perform inference accordingly.
NLP practitioner and expert, you provide instructions that prompt the instruction-tunel L.Ms well for the given tasks, The models in which the instructions might be evalunted are:
GPT-4
Text Davinci
Flan-PaLM
Flan-T5
T0++
m
MetalICL
CharGLM
Alpuca
You are very well come use your experience on these models to come with the instruction you think will perform the best.
Tacks
The participation will take approximately 30 minutes, You will be given tasks/settings instruction the indieated by the order
For each fask/setting, you going number. For instance:
"Auto Debugging - Zero-Shot - 5"
meuns that you are ussigned to write an instruction on the task "Auto Debugging" with the setting "Zero-shot," and you are putting your answer in the row ID
Figure 13: The first page of the instruction given to the annotator
27

---

Under review as a conference paper at ICLR 2024
For each task, there folder with the exact the file, will see detailed information about the task, including input-output format, task description, and examples. There will be two main tables one to record instructions in Zero- shor and one to record instructions in Few-shots, In each table, there will be an example provided. Be sure to put your instruction under the correct tuble and follow the format of example!
the folder, there doex file. Open
exumples, the "task, joon" all the examples the test you may have better iden
you
Data source
The tasks given to you are sampled from benchmarkk BBH-Lite and MMLU.
Thank
Figure 14: The second page of the instruction given to the annotator
28

---

Under review as a conference paper at ICLR 2024
Thank you helping research project! goal gather instructions experienced NLP researchees on various downstream tasks incorporated in the benehmark BBH. Your
Write down the instraction (prompt) for task that you think will this task on instruction-tuned Seq2Sen LMs TS---T-XXI, Devinci-text-003, OPT- IML, etc.) zero-shot und shw-shots (in content learning).
for
Pleuse put instruction the corresponding one page below the zero-sbot table. Please use {...] to denote conresoonding information. Note: you de
tables. The few-shots table
all the information if think
distractions.
For multiple choice tasks, you cither formulate the instruction to let the model number/lttee
oonput
Same goes for classification
Information provides overview task, including input, outpor, have a better grasp Few-shots
description; Example provides an example the test set task; are in the following pages.
Zero-shot
Instend of using ""n" "*"",
directly enter or tab.
The given example also represents the average length of the input/ouiput for this task. You may the maximum token length of the LM is 4096
Task Information
Dataset
BIG-Bench
Code Line Description
Metric
Accuracy
description
Eaplish language description Python program, choiceA, choiceB, choicoC, choicel)
Input
Output
answer
Figure 15: The first page of the dataset information
29

---

Under review as a conference paper at ICLR 2024
Example:
program: for rin peint()
choiceA: prints values from
choicel:: computes first prime numbers
choiceC; prints values from to 10
choicelD: prints hello world' to the terminal
Output
answer: prints values from 22/A
Zero-shot Instruction:
You are given:
[pecgnm]:
input oode (cholceA). (cholceB), (chosceC), (cholceD); choices imerpreazzion
Instruction
Example
an English languge deserittio Python codelpmgaan) (choiock) [choiae8) [choiceC) D. (chooce()
Eiglish linrage deseription:
2
4
9
Figure 16: The second page of the dataset information
30

---

Under review as a conference paper at ICLR 2024
K DISAGGREGATED RESULTS
K.1 MAIN RESULTS AND SCALING RESULTS
In the main paper we reported aggregated results over benchmark corpora. Here we report results on individual datasets for BBL. For MMLU, we evaluate all 57 datasets together, because these are all QA tasks (and we would want QA model be capable of answering questions across diverse set of domains), We report means and stadard deviations of the accuracies achieved over all instructions Table 20 The numbers on the left of the setting suggest the number of instructions used. We also share even more granular repurtinrproring the performance for each instruction in CSV files provided in the supplemental material.
MMLU
Model
Flan-T5-XL
Flan-T5-XXL
TOpp-11B
Alpaca-7B
Alpaca-133 13B
MMLU
OBSERVED
48.1 (#: 0.3)
53.2 (# 0.2)
48.3 ( 0.9)
41.9 (+ 0.6)
47.8 (+ 0.5)
UNOBSERVED
47.5 0.9)
52.7 0.8)
48.5 (= 0.9)
39.7 (# 2.2)
47.0 (# 0.8)
Table 19; Granular results corresponding to the subset reported in Table 3 on each dataset of MMLU. We treat tasks MMLU equally as general QA and computed the overall accuracy.
MMLU
Size Variance
Small (80M)
Base (250M)
Large (780M)
XL (3B)
XXL (IIB)
MMLU
OBSERVED
29.4 (# 1.0)
34.1 (# 0.4)
41.1 (= 0.2)
48.1 (# 0.3)
53.2 (# 0.2)
UNOBSERVED
29.6 (=t 0.9)
33.8 (# 1.2)
40.7 (= 0.7)
47.5 (+ 0.9)
52.7 (# 0.8)
Table 20: Granular results corresponding the subset reported in Figure 2b each dataset of MMLU. We treat all tasks in MML.U equally as general QA and computed the overall accuracy.
31

---

Under review as a conference paper at ICLR 2024
BBL-QA
Model
Flan-TS-XL
Flan-T5-XXL
T0pp-11B
Alpaca-7B
Alpaca-133 1apaca-13B
BBQ Lite
OBSERVED
66.5 (#: 1.5)
77.4 (= 2.4)
5188 (1 5.3)
32.6 (1: 1.0)
43.5 (#: 1.4)
UNOBSERVED
67.0 (+: 7.0)
73.7 (#: 11.4)
51.6 (#: 3.0)
33.1 (#: 1.3)
45,5 (#: 2.9)
Code Desc.
OBSERVED
73.6 (+: 3.4)
83.6 (= 1.7)
70.3 (+: 3.0)
70.2 (+: 2.5)
85.2 (4: 2.4)
UNOBSERVED
69.7 (+ 12.4)
72.9 (# 22.2)
70.5 (# 3.7)
67.5 (# 11.3)
82.2 (# 8.5)
Hindu Know.
OBSERVED
52.4 (+ 1.6)
53.9 (= 1.8)
57.1 (+ 2.5)
50.9 (# 2.1)
63.8 (# 0.7)
UNOBSERVED
47.1 (#: 5.4)
56.5 3.5)
53.2 (# 3.0)
49.8 (#: 5.1)
63.9 ( 1.1)
Known Unk.
OBSERVED
79.3 (#: 2.5)
84.7 2.1)
70.9 (#: 10.2)
75.2 (#: 4.7)
81.9 ( 4.3)
UNOBSERVED
69.0 (1 6.7)
80.6 8.1)
76.1 (LL 5.9)
60.9 (1 11.2)
71.1 (1 16.3)
Logical Ded.
OBSERVED
52.5 (1: 1.0)
58.0 0.7)
45.5 (1L 0.8)
25.5 (1 1.1)
29.2 (1: 1.3)
UNOBSERVED
52.1 1.1)
57.8 (= 0.6)
45.3 1.2)
24.5 (#: 2.3)
28.0 (#: 1.6)
Novel Cone.
OBSERVED
29.8 2.4)
50.1 1.9)
28.8 2.9)
37.2 (+ 5.2)
20.0 (+: 3.1)
UNOBSERVED
31.2 5.0)
46.0 (= 5.4)
(+L 5.1)
36.1 (4: 7.6)
19.6 (1: 4.0)
Logic Grid
OBSERVED
41.8 1.1)
43.2 1.8)
37.6 1.7)
24.4 (# 1.9)
29.3 (#: 0.8)
UNOBSERVED
38.6 (+: 5.4)
39.6 (= 4.4)
36.4 (+ 3.6)
25.4 (+: 1.1)
28.7 (+ 1.2)
Cone. Com.
OBSERVED
75.9 (#: 1.8)
75.0 ( 2.6)
73.3 (+ 2.3)
58.7 (=t 4.0)
63.0 (#: 2.2)
UNOBSERVED
75.0 (+ 1.9)
73.6 (= 4.8)
74.2 (+: 2.6)
55.9 (+ 6.2)
61.1 (# 3.3)
Table 21: Granular results corresponding to the subset reported in Table 3 each dataset of category B BBLQQA.
32

---

Under review as a conference paper at ICLR 2024
BBI.QA
Size Variance
Small (80M)
Base (250M)
Large (780M)
XL (3B)
XXL (IIB)
BBQ Lite
OBSERVED
28.3 (# 1.3)
51.5 (= 1.4)
56.6 (# 2.0)
66.5 (# 1.5)
77.4 (# 2.4)
UNOBSERVED
28.6 (#: 4.3)
50.5 (#: 4.1)
56.7 (d: 4.7)
67.0 7.0)
73.7 (#: 11.4)
Code Desc.
OBSERVED
22.0 (# 4.0)
55.7 (# 3.3)
72.4 (#: 3.2)
73.6 (# 3.4)
83.6 (# 1.7)
UNOBSERVED
32.1 (# 7.2)
48.6 7.2)
63.3 (# 14.2)
69.7 (# 12.4)
72.9 (# 22.2)
Hindu Know.
OBSERVED
25.1 (+ 15.2)
30.7 (#: 2.6)
34.9 (+ 0.9)
52.4 (+ 1.6)
53.9 (+ 1.8)
UNOBSERVED 31.6 (#: 10.7)
26.9 4.4)
37.5 (# 7.0)
47.1 5.4)
56.5 (# 3.5)
Known Unk.
OBSERVED
49.9 (+ 1.9)
66.9 (+ 4.7)
76.2 (+ 3.5)
79.3 (+ 2.5)
84.7 (+ 2.1)
UNOBSERVED
52.8 (# 5.2)
63.8 7.3)
68.4 (# 11.1)
69.0 6.7)
80.6 (# 8.1)
Logical Ded.
OBSERVED
19.8 (+ 0.7)
27.1 (+ 1.3)
45.9 (+ 1.1)
52.5 (+ 1.0)
58.0 (+: 0.7)
UNOBSERVED
19.9 (# 0.4)
28.9
2.8)
46.4 (+: 2.6)
52.1 (#: 1.1)
57.8 (+ 0.6)
Novel Conc.
OBSERVED
22.9 (#: 9.2)
15.9 (= 5.4)
31.0 (#: 2.3)
29.8 (# 2.4)
50.1 (# 1.9)
UNOBSERVED
19.3 (+: 3.6)
16.8 4.0)
28.4 (+L 6.9)
31.2 (1 5.0)
46.0 (#: 5.4)
Logic Grid
OBSERVED
22.3 (# 4.0)
31.7 (= 0.8)
32.6 (#: 2.1)
41.8 (= 1.1)
43.2 (# 1.8)
UNOBSERVED
28.8 (# 3.1)
29.4 (# 5.1)
34.1 (# 2.8)
38.6 (# 5.4)
39.6 (# 4.4)
Conc. Com.
OBSERVED
30.4 (#: 10.6)
55.6 5.1)
64.2 (+ 1.9)
75.9 1.8)
75.0 (#: 2.6)
UNOBSERVED 32.2 (+: 16.7)
54.5 (# 9.4)
58.1 (# 11.6)
75.0 (# 1.9)
73.6 (+ 4.8)
Table 22: Granular results corresponding the subset reported in Figure 2b on each dataset of category BBI.QA.
33

---

Under review as a conference paper at ICLR 2024
BBL-BC
Model
Flan-T5-XI.
Flan-T5-XXI.
T0pp-11B
Alpaca-7B
Alpaca-133
Play Dialog
OBSERVED
61.6 (+: 5.8)
51.8 (+ 9.5)
62.7 (+ 0.4)
45.0 (+ 2.0)
53.4 (+ 5,8)
UNOBSERVED 53.0 (# 6.9)
58.1 (# 4.4)
55.2 (# 8.1)
42.9 (# 7.9)
42.9 (# 8.8)
Strat. QA
OBSERVED
58.7 (# 3.3)
64.2 (# 3.0)
51.0 (# 1.8)
53.0 (# 2.1)
56.7 (# 3.8)
UNOBSERVED 60.7 (=: 7.5)
59.3 (# 6.1)
54.5 (1 0.9)
53.3 (1 4.1)
61.0 (# 1.9)
Strange St.
OBSERVED
69.3 (= 4.4)
71.0 (# 7.3)
51.2 51 5.1)
67.0 (# 4.7)
69.8 ( 5.0)
UNOBSERVED
70.5 (+: 7.0)
77.4 (1 6.1)
48.4 (4 3.1)
59.9 (1 9.4)
57.5 (d: 5.6)
Winowhy
OBSERVED
76.5 (+: 1.9)
75.6 (# 4.0)
99.6 (# 1.0)
50.1 (1 4.8)
51.9 (1: 4.6)
UNORSERVED 60.2 (+ 6.2)
59.7 (+ 7.7)
60.9 (+ 5.1)
53.4 4.6)
55.2 (+ 6.0)
Table 23; Granular results corresponding to the subset reported in Table BBL-BC.
3 on each dataset of category
BBL-BC
Size Variance
Small (80M)
Base (250M)
Large (780M)
XL (3B)
XXL (11B)
Play Dialog
OBSERVED
51.6 (+ 13.3)
54.6 (+: 10.7)
59.0 (+: 6.7)
61.6 (+ 5.8)
51.8 (+ 9.5)
UNOBSERVED
61.6 (= 4.6)
56.3 (#: 10.9)
57.3 (# 7.8)
53.0 (# 6.9)
58.1 (= 4.4)
Strat. QA
OBSERVED
52.3 (+ 1.0)
48.9 (+: 2.1)
60.9 (+ 1.3)
58.7 (+: 3.3)
64,2 (+ 3.0)
UNOBSERVED
51.5 (# 2.7)
52.9 (# 1.3)
53.9 (# 3.8)
60.7 (# 7.5)
59.3 (# 6.1)
Strange St.
OBSERVED
41.3 (# 10.3)
43.1 (# 4.2)
54.4 (# 1.2)
69.3 (# 4.4)
71.0 (# 7.3)
UNOBSERVED
55.9 (= 18.5)
42.0 (#: 5.5)
67.9 (+: 8.0)
70.5 (1: 7.0)
77.4 (+: 6.1)
Winowhy
OBSERVED
54.8 (= 1.6)
55.9 (+ 7.6)
60.4 (#: 9.8)
76.5 (#: 1.9)
75.6 (# 4.0)
UNOBSERVED 53.7 (#: 5.1)
57.1 (+ 6.7)
53.5 (+: 4.9)
60.2 (1: 6.2)
59.7 (1: 7.7)
Table 24: Granular results for Figure 2b on each dataset of category BBL-BC
BBL-MC
Model
Flan-T5-XL
XIan-T5-XXL
T0pp-11B
Alpaca-7B
Alpaca- 13B
Language ID
OBSERVED
32.6 (=: 0.2)
38.9 (# 0.3)
15.7 (# 3.0)
12.9 (# 0.7)
18.5 (#: 0.7)
UNOBSERVED 25.5 (+: 7.3)
31.6 (# 9.4)
14.3 (# 2.4)
14.7 (# 1.7)
21.7 (#: 0.7)
Vitamin C
OBSERVED
78.6 + 1.1)
78.5 (#: 0.7)
68.3 (+ 1.1)
51.4 (# 3.6)
54.9 (# 2.9)
UNOBSERVED
78.6 (+ 3.6)
80.2 (+: 1.6)
68.5 (+ 2.4)
18.1 (# 5.3)
23.6 (#: 14.2)
Table 25: Granular results for Table on each dataset of category BBL-MC
34

---

Under review as a conference paper at ICLR 2024
BBL-MC
Size Variance
Small (80M)
Base (250M)
Large (780M)
XL (3B)
XXL (1IB)
Language ID
OBSERVED
11.9 (+: 0.2)
17.0 (#: 0.3)
25.8 (= 0.3)
32.6 (+: 0.2)
38.9 (#: 0.3)
UNORSERVED
9.5 (+ 0.2)
12.4 (+: 1.5)
19.2 (+ 4.4)
25.5 (+ 7.3)
31.6 (+ 9.4)
Vitamin C
OBSERVED
46.6 (#E 4.0)
60.7 (#: 5.6)
72.6 (# 1.5)
78.6 (# 1.1)
78.5 (#: 0.7)
UNOBSERVED
40.8 (+ 4.2)
63.0 (+ 4.6)
36.4 (+ 0.8)
78.6 (+ 3.6)
80.2 (+ 1.6)
Table 26: Granular results for Figure 2b on each dataset of category BBL-MC
35

---

Under review as a conference paper at ICLR 2024
K.2
ICL FEW EXPERIMENT RESULTS
MMLU (One-shot)
Flan-T5
Small (80M)
Base (250M)
Large (780M)
XL (3B)
XXL (1IB)
MMLU
OBSERVED
29.3 (1 0.9)
33.9 (1 0.5)
40.7 (1 0.2)
47.5 (:: 0.2)
52.7 (= 0.2)
UNOBSERVED 29.6 (#: 0.6)
33.8 (# 1.0)
40.4 (# 0.9)
47.5 (= 0.7)
52.8 (# 1.0)
Table 27: Granular results on each dataset MMLU for Flan-T5 models and one-shot ICL. We group all QA tasks in MMLU and report overall accuracy on these.
BBL-QA (One-shot)
Flan-T5
Small (80M)
Base (250M)
Large (780M)
(3B)
XXL (IIB)
BBQ Lite
OBSERVED
29.6 (+: 2.2)
50.5 (= 1.7)
57.0 (+: 2.0)
66.7 (+: 1.6)
77.2 2.7)
UNOBSERVED
28.9 (1: 1.4)
(: 3.6)
58.0 (1. 2.7)
69.0 (4: 5.9)
77.6 (1 5.5)
Code Desc.
OBSERVED
20.5 (# 3.1)
56.9 (# 5.1)
76.0 (+: 2.2)
73.6 (# 1.8)
85.3 1.6)
UNOBSERVED
25.6 (+: 8.7)
43.6 (+ 8.4)
53.5 (+: 16.0)
65.8 (+: 15.3)
75.0 (= 12.3)
Hindu Know.
OBSERVED
24.5 (+: 12.7)
30.5 (+ 3.6)
36.2 (+: 1.7)
50.9 (+ 1.3)
52.9 (+ 1.9)
UNOBSERVED
23.0 (#: 5.8)
22.4 (# 5.8)
37.7 (+: 4.7)
50,2 (#: 5.8)
54.6 (# 3.2)
Known Unk.
OBSERVED
49.2 (+: 3.8)
66.7 (= 8.3)
73.6 (#: 2.7)
76.3 (# 2.0)
84.7 (# 388)
UNOBSERVED
49.1 (1 4.1)
60.2 (1: 7.3)
67.7 (1: 12.9)
63.7 (1: 9.8)
76.0 (= 12.7)
Logical Ded.
OBSERVED
20.2 (1: 0.4)
26.8 (# 1.0)
45.9 (1 1.1)
53.0 (1: 0.7)
58.2 (1 0.5)
UNOBSERVED
20.2 (+: 0.8)
27.8 (# 3.5)
44.3 (+: 8.6)
48.6 (+: 10.1)
55.0 (= 10.5)
Novel Cone.
OBSERVED
21.3 (+: 4.3)
14.8 (# 4.9)
29.1 (+ 4.0)
31,2 (+: 2.0)
47.7 (#: 3.1)
UNOBSERVED
17.2 (+: 6.8)
14.7 (= 9.0)
24.7 (+ 6.2)
35.1 (+: 5.2)
42.5 (# 9.0)
Cone. Com.
OBSERVED
28.0 (+: 3.6)
38.5 (+ 2.3)
65.0 (+ 1.8)
77.7 (+ 1.6)
77.1 (+ 2,2)
UNOBSERVED
28.6 (+ 4.5)
36.3 (= 6.4)
58.2 (+ 10.7)
74.8 (# 11.9)
75.2 (# 2.8)
Table 28: Granular results on each dataset of category BBL-QA with one-shot in-context learning.
36

---

Under review as a conference paper at ICLR 2024
BBL-BC (One-shot)
Flan-T5
Small (80M)
Base (250M)
Large (780M)
XL (3B)
XXL (11B)
Play Dialog
OBSERVED
49.8 (+ 11.9)
54.4 (4: 10.3)
58.1 (+: 5.7)
57.8 (+: 3.3)
43.9 (# 4.2)
UNOBSERVED
55.8 (= 10.2)
52.5 (# 10.7)
48.6 (# 8.7)
46.3 (# 3.9)
51.3 (# 3.3)
Strat. QA
OBSERVED
52.5 (+: 0.8)
49.3 (4: 3.1)
60.6 (+: 1.5)
60.6 (+: 6.2)
66.2 (+ 2.6)
UNOBSERVED
53.2 (#: 0.0)
53.3 (#: 0.8)
55.9 (# 4.3)
61.2 (:: 6.0)
62.2 (# 5.5)
Strange St.
OBSERVED
40.7 (= 12.3)
41.8 (= 2.3)
51.7 (# 1.6)
74.4 (# 3.6)
78.5 (# 2.1)
UNOBSERVED
46.7 (#: 5.1)
37.9 (# 8.6)
56.3 (#: 3.0)
78.7 (d: 3.2)
83.2 (1 7.6)
Winowhy
OBSERVED
52.7 (= 2.8)
57.3 (# 6.2)
62.1 (# 5.9)
77.2 (# 0.6)
76.7 1.0)
UNOBSERVED 48.1 (#: 4.2)
58.3 (#: 8.1)
57.1 (# 8.8)
66.3 (# 8.8)
65.5 (= 9.9)
Table 29: Granular results on each dataset of category BBL-BC with one-shot in-context learning.
BBL-MC (One-shot)
Flan-T5
Small (80M)
Base (250M)
Large (780M)
XL (3B)
XXL (IIB)
Language ID
OBSERVED
11.7 (=: 0.2)
13.5 (# 2.8)
25.6 (# 0.4)
31.8 (# 0.3)
38.7 (#: 0.5)
UNOBSERVED
9.7 (# 0.9)
11.6 (+ 1.5)
16.9 (+: 5.4)
20.3 (+ 7.5)
28.4 (+: 10.3)
Vitamin C
OBSERVED
50.9 (+ 1.6)
60.9 (+: 5.4)
73.3 (+: 0.8)
78.6 (+: 1.4)
80.4 (+: 0.5)
UNOBSERVED
50.1 (# 0.9)
64.5 (+ 1.8)
73.2 (+ 1.5)
77.5 (+ 9.3)
80.7 (+ 3.6)
Table 30: Granular results on each dataset of category BBL-MMC with one-shot in-context learning.
37

---

Under review as a conference paper at ICLR 2024
K.3 REPRESENTATION ALIGNMENT RESULTS
In the main paper reported the aggregated results over benchmark corpora. We report means and stadard deviations of the accuracies achieved over all instructions in Table and 32 We also share granular results reporting the performance for each instruction in CSV files provided the supplemental material.
Flan-T5-XI
Alpaca-7B
MMLU
Dataset
OBS.
UNOBS.
Avg.
OBS.
UNOBS.
Avg.
MMLU
-Baseline
48.0 0.3
47.3 + 0.9
47.7 + 0.6
42.1 + 0.4
40.6 1.6
41.3 + 1.0
Aligned
48.4 0.2
47.9 + 0.5
48.2 + 0.6
42,9 + 1.4
41.7 2.0
42.3 + 1.0
Overall A
0.440.0
0.6+0.00
0.5
0.8
1.1:0.0
1.0+0.0
BBL-MC
Dataset
OBS.
UNOBS.
Avg.
OBS.
UNOBS.
Avg.
Language ID
Baseline
32.7 0.1
29.2 1.0
30.9 = 0.6
12.6 0.9
14.8 1.7
13.7 + 1.3
Aligned
32.6 + 0.2
29.4 + 0.8
31.0 + 0.6
12.9 0.5
16.7 + 1.0
14.8 + 1.3
Vitamin C
Basellne
77.6 + 0.0
80.9 + 1.3
79.3 + 0.7
45.3 + 0.5
13.3 + 0.1
29.3 + 0.3
Allined
78.3 0.1
81.4 1.5
79.8 - 0.7
45.6 0.9
18.5 + 0.4
32.1 + 0.3
Overall A
0.340.4
0.340.1
0.3
0.340.0
3.5+1.6
1.910.8
BBL-BC
Dataset
OBS.
UNOBS.
Avg.
OBS.
UNOBS.
Avg.
Strat. QA
-Baseline
58.4 1.6
59.9 4 8.0
59.2
+ 4.8
53.4 2.0
51.6 + 4.4
52.5 3.2
-Aligned
58.6 2.9
60.6 + 6.6
59.6
4.8
54.6 + 3.1
55.2 5.5
54.9 + 3.2
Strange St.
Basellne
69.1 2.9
69.4 + 4.1
69.2 + 3.5
69.6 + 1.3
56.1 4.3
62.9 2.8
Alignedd
59.0 4.1
69.8 + 5.0
64.4
+ 3.5
70.5 + 1.1
59.5 6.8
65.0 + 2.8
Play Dialog
Baseline
58.0 + 10.9
52.2 + 6.8
55.1
+ 8.9
45.0 + 2.0
45.6 10.8
45.3 + 6.4
Aligned
61.4 + 3,0
58.6 + 7.0
60.0 + 8.9
45.7
3.3
48.8 + 11.8
47.2 + 6.4
Winowhy
Basellne
76.2 1.9
54.3 + 11.8
65.3 6.9
49.5 + 8.9
55.7 0.1
52.6 + 4.5
Alligned
77.6 + 3.3
54.8 + 11.6
66.2 + 6.9
50.2 + 8.4
58.6 + 1.9
54.4 + 4.5
Overall A
-1.3+5.2
2.0+2..5
3.4
0.9+0.2
3.2++0.3
2.1+-0.2
Table 31: Granular results corresponding to the MMLU, BBL-MC, and BBL-BC subset reported in Baseline refers to the performance of the original Instruction-tunee model. We observe the significant improvement on following unobserved instructions consistently.
Table 7
38

---

Under review as a conference paper at ICLR 2024
Flan-T5 T5-XL
Alpaca-7B
BBL-QA
Dataset
OBS.
UNOBS.
Avg.
OBS.
UNOBS.
Avg.
Code Desc.
-Baseline
75.0 + 3.1
69.3 + 11.8
72.2 = 7.5
3.1
65.6
14.7
68.8 + 8.9
Aligged
76.4 + 1.8
74.9 1.7
75.7 + 7.5
72.8 + 3.5
71.7 + 17.3
72.2 + 8.9
Hindu Know.
Baseline
52.7 + 2.4
45.5 6.2
49.1 = 4.3
2.3
49.0 + 5.3
50.7 3.8
Aligned
54.0 = 2.6
49.6 = 1.2
51.8 = 4.3
53.8 + 0.2
55.9 + 1.5
54.8 + 3.8
BBQ Lite
Baseline
66.7 + 1.9
62.9 8.0
64.8 + 5.0
31.8 1.3
33.2 + 0.6
32.5 0.9
Aligned
66.0 + 2.3
69.0 = 2.2
67.5 = 5.0
32.7 + 2.0
36.3 + 2.2
34.5 0.9
Conc. Com.
Baseline
75.7 + 1.5
75.0 2.1
75.3 + 1.8
56.3 5.5
53.1 + 6.2
54.7 5.9
Aligned
81.4 + 1.1
79.0 + 1.1
80.2 + 1.8
56.4 + 5.7
56.1 + 6.9
56.3 + 5.9
Logical Ded.
Baseline
52.6 = 0.3
52.4 0.7
52.5 = 0.5
25.7 + 0.8
25.4 + 1.4
25.5 1.1
Aligned
52.6 + 0.7
52.8 0.6
52.7 + 0.5
25.9 + 0.5
28.4 + 1.2
27.2 + 1.1
Novel Conc.
Baseline
31.9 + 2.6
32.5 4.2
32.2 = 3,4
36.9 4.1
35,0 = 2.6
35.9 =
Aligned
28.7 + 2.0
31.6 + 4.3
30.2 + 3.4
37.5 + 3.9
39.2 + 2.4
38.3 + 3.3
Logic Grid
Baseline
42.0 = 0.9
40.0 4.4
41.0 = 2.7
0.9
25.6 1.5
25.7 + 1.2
Aligned
45.1 = 3.6
41.1 4.1
43.1 = 2.7
26.1 = 1.0
30.4 + 4.2
28.2 + 1.2
Known Unk.
Basellie
78.3 + 1.5
67.4 6.3
72.8 3.9
72.9
61.3 + 9.3
67.1 + 7.0
-Aligned
79.3 + 1.6
65.5 = 10.0
72.4
74.0
443
67.4 7.1
70.7 + 7.0
Overall A
1.1
2.342.8
1.742.1
0.7.0.44
4.6.1..5
2.740.9
Table 32:
Granular results corresponding to the BBL-QA subset reported in Table 7 Bascline refers the performance of the original instruction-tunee model. We observe the significant improvement on following unobserved instructions consistently.
39


Reviews:
[{"review_id": "P5UjYIQS75", "reviewer": "Reviewer_RNd8", "summary": "The paper critiques instruction tuning (IT) by showing that the IT models are not robust to rephrasing of the same prompt that were not seen in training. More surprisingly, using a prompt that 1) was seen in training, but 2) is incorrect for the context has a higher accuracy than an unseen but correct prompt. They propose a mitigation strategy by adding a soft prompt that is explicitly optimized to make the model see the different rephrases as similar to each other. They also release a dataset of 319 prompt rephrasings.", "questions": "- It looks like just fine tuning (or even fine tuning + KL) severely drops performance. Do you know why? This seems very surprising\n- Why use NLP grad students to generate rephrasing of the prompts, when you could just use an LLM (which it seems like you do later for the eval set)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This seems like a significant result overall, and explicitly shows an important weakness in IT models\n- In general, this kind of work (exploring what \"success on a benchmark\" really means) is extremely important for the field. You are raising a really important issue with fine tuning, and this is impactful.\n- The experiments in the paper are well-executed, well-presented, and broad across models and datasets.\n- It's interesting to explicitly introduce the rephrasing invariance into the fine tuning data-- it's sort of an  equivalent of regularization, I think? \n- Thanks for including figure 4! It helped a lot with the intuition in that section.", "weaknesses": "- nit: \"instruction-tuned models are not especially robust to instruction rephrasings\" → this is very clear, say it first. Better yet, make it the title :) \n- I don't know if just releasing prompt rephrasing is a significant enough contribution-- https://openreview.net/forum?id=9Vrb9D0WI4 and others also have datasets of different rephrasing of prompts for the same task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cU1wpQX4pv", "reviewer": "Reviewer_cGUu", "summary": "The authors analyse how sensitive various instruction-tuned models are to rephrased instructions, especially in zero-shot settings. They find that using novel instructions previously unseen by the model results in degraded performance across different tasks, and does not seem to improve with model scale. Further analysis suggests that instructions that are more dissimilar to instructions seen during training generally perform (somewhat) poorer. Finally, they introduce a prompt-tuning method that utilises a KL penalty to punish representations that drift from paraphrased versions of instructions. They show this performs better than simply training on the paraphrased instructions alone.", "questions": "- How does your proposed method compare to introducing the model paraphrases during initial instruction finetuning? Training on the paraphrased versions alone after the instruction tuning may cause forgetting in an undesirable way.\n- Is the degradation of performance on unobserved prompts statistically significant? The variance seems very high.\n- Doing prompt tuning alone seems to often perform very similarly to PT+KL, and within the reported standard error in Appendix B.3. Do you have statistical tests for significance for your results?\n- It would be interesting to see the results in Table 7 broken down into the BBL QA/BC/MC categories shown in Table 6 to see how much the difference in closest distance affects the performance of these different categories.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The experiments are clear and well-thought-out, and the results are solid and interesting, affecting many current popular models (Llama finetunes), and holding for different model families.\n- The observed correlations between performance and instruction similarity (in representation space) are interesting (even if the correlation is somewhat weak).\n- The proposed fix method is well-motivated, and the results seem reasonable, especially for the Alpaca-7B model.\n- The robustness of models to different prompts is an important and interesting area of study, especially considering current difficulties with LM evaluations.", "weaknesses": "- The novelty of the robustness experiments is somewhat limited, as there is prior work that has observed variance based on prompt formatting (discussed in the work itself). Considering this work seems to examine more modern models such as Alpaca, and considers a different set of tasks, I think this is a minor weakness - the work does add a number of interesting insights, especially around representation distance.\n- The claim that models perform worse at novel phrasings of tasks seems a bit strong compared to the results: examining Table 3, the variance of performance on unobserved phrasings often is greater than the difference in average performance, suggesting that the difference in performance may not be significant (although the large increase in variance is certainly not desirable and interesting to see).\n- While the proposed method seems strong, examining the results in Appendix B.3 seems to show that in many cases, prompt-tuning alone performs within one standard deviation of the PT+KL results (e.g., for BBL unobserved and average). As such, the proposed method might not be significantly different from just prompt tuning.\n\nOverall, I think this paper is okay, and the robustness results are interesting and useful to see, along with the representation alignment results. However, the novelty of the robustness experiments is a bit limited, and it’s not clear the proposed method for improving this is significantly better than baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7iw6UIIDCQ", "reviewer": "Reviewer_hcVs", "summary": "This paper observes that current instruction-tuned LLMs are sensitive to surface forms of instructions by showing that they underperform on unseen instructions compared to seen instructions. To mitigate this issue, the paper proposes soft prompt alignment technique.", "questions": "- Do you think that RLHF after instruction tuning might mitigate this issue?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper performs experiments on various tasks including reasoning benchmarks (MMLU and BBL) and NLG benchmarks.\n- The paper provides the experimental setting of evaluating the robustness of LLMs on different instructions by releasing various instructions collected from NLP researchers, which might facilitate further research.", "weaknesses": "- In Tables 3 and 7, compared to the performance difference, the variance is very large, questioning the significance of the difference between observed and unobserved instructions. A significance test should be conducted to test the significance of the difference.\n- Although the title, abstract, and motivation of the paper mention that the paper tries to evaluate the zero-shot robustness of instruction-tuned language models generally, it only focuses on models up to 11B. The tendency is likely to be unpredictable for state-of-the-art LLMs such as GPT-3.5 or GPT-4. \n- How does PT+KL on Table 7 perform on negated instructions? The alignment stage might make the LLM insensitive to most instructions even though the instructions are not relevant or negated, which is a similar finding for instruction-tuned LLMs for Webson & Pavlick (2022) \n\nWebson & Pavlick (2022) - Do Prompt-Based Models Really Understand the Meaning of their Prompts?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iOWmzrnXnG", "reviewer": "Reviewer_KbYU", "summary": "The study examines the robustness of LLMs in zero-shot prompting. It was observed that the model's performance varies based on the instructions used, particularly with unfamiliar instructions. To mitigate this decline in performance, the study proposes a method that incorporates soft-prompts and trains the model with an additional loss function designed to align the embedding representation of the instructions.", "questions": "- \"We then form batches by including one instance featuring the original instruction, and the rest comprising paraphrased instructions.\" -> what's the batch size, and consequently, what's the paraphrase size and the final dataset size altogether? \n- Significant degradation on FT on Table 7, why is that so?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "This paper provides a thorough comparison of LLM's instruction robustness across various models and sizes. Although the finding—that LLMs are not robust to varying instructions—is anticipated, this paper presents compelling empirical evidence. Additional analysis in the embedding space of the instruction is interesting. Furthermore, the study suggests an elegant solution to narrow the performance disparity between familiar and unfamiliar instructions.", "weaknesses": "The proposed solution could benefit from more details. Exploring the soft-prompt size (i.e., the variable N) would be nice. The N used is not specified in the main text either, which I believe is an important detail. \n\nThey provide synthetic instruction, and one of the baselines is to fine-tune normally with this new instruction. Interestingly, there is a noticable degradation in performance, yet there's no explanation as to why. Usually, continual fine-tuning on the same data should not damage performance (significantly so). My guess is that the quality of the synthetic prompt is not very good. If so, this raises a question: what if we spend more effort on generating better synthetic instruction beforehand?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)