Instructive Decoding: Instruction-Tuned Large Language Models are Self-Refiner from Noisy Instructions
Authors: Taehyeon Kim
Joonkee Kim
Gihun Lee
Se-Young Yun
Venue: ICLR 2024 spotlight (2024)
Keywords: Instruction Following
Language Model
Decoding
Primary area: generative models
TL;DR: This paper proposes a decoding method that using noisy instructions to help instruction-tuned models concentrate on the given instructions.

Abstract: While instruction-tuned language models have demonstrated impressive zero-shot generalization, these models often struggle to generate accurate responses when faced with instructions that fall outside their training set. This paper presents Instructive Decoding (ID), a simple yet effective approach that augments the efficacy of instruction-tuned models. Specifically, ID adjusts the logits for next-token prediction in a contrastive manner, utilizing predictions generated from a manipulated version of the original instruction, referred to as a noisy instruction. This noisy instruction aims to elicit responses that could diverge from the intended instruction yet remain plausible. We conduct experiments across a spectrum of such noisy instructions, ranging from those that insert semantic noise via random words to others like 'opposite' that elicit the deviated responses. Our approach achieves considerable performance gains across various instruction-tuned models and tasks without necessitating any additional parameter updates. Notably, utilizing 'opposite' as the noisy instruction in ID, which shows the maximum divergence from the original instruction, consistently produces the most significant performance gains across multiple models and tasks.

Published as a conference paper at ICLR 2024
INSTRUCTIVE DECODING: IISTRUCTION-TUNEL LARGE LANGUAGE MODELS ARE SELF-REFINER FROM NOISY INSTRUCTIONS
Tachyeon Kim", Joonkee Kim", Gihun Lee', and Se-Young Yun
KAIST AI
ABSTRACT
While instruction-tuned language models have demonstrated impressive zero-shot generalization, these models often struggle generate accurate responses when faced with instructions that fall outside their training set. This paper presents Instructive Decoding (ID), simple yet effective approach that augments the efficacy of instruction-tun -uuned models. Specifically, ID adjusts the logits for next- token prediction in contrastive manner, utilizing predictions generated from a manipulated version of the original instruction, referred to as a noisy instruction. This noisy instruction aims elicit responses that could diverge from the intended instruction yet remain plausible. We conduct experiments across spectrum of such noisy instructions, ranging from those that insert semantic noise via random words to others like 'opposite" that elicit the deviated responses. Our approach achieves considerable performance gains across various instruction-tunet models and tasks without necessitating any additional parameter updates. Notably, utilizing 'opposite" as the noisy instruction in ID, which exhibits the maximum divergence from the original instruction, consistently produces the most significant performance gains across multiple models and tasks.
INTRODUCTION
Language Models (LMs) have opened up new era in Natural Language Processing (NLP) by leveraging extensive datasets and billions of parameters (Zhao al., 2023; OpenAl, 2023; Kaplan et al., 2020). These LMs excel at In-Context Learning (ICL), generating responses based on a few demonstrations without needing further parameter adjustments (Wei et al., 2022: Brown et al., 2020; Dong al., 2022). The rise of instruction-tuning has further enhanced this capability, optimizing LMS to align their outputs closely with human-specified instructions (Wei et al., 2021 Sanh et al., 2021; Brown et al., 2020; Radford et al., 2019). This approach has demonstrated a significant improvement in zero-shot scenarios, underscoring its importance for tackling diverse tasks.
However, instruction-tuned models often struggle with unfamiliar tasks due to limitations in their training datasets, whether the datasets are human- annotated (Mishra et al., 2021; Wang et al., 2022c) or model-generated (Wang et al., 2022b: Honovich et al., 2022). Refining these datasets essential but requires substantial effort and computational resources, highlighting the need for more efficient approaches (Chung et al., 2022: Zhou al., 2023). Moreover, the depth of a model's understanding of and how they respond to instructions remains an area of active research. While recent studies have provided some insights (Kung& & Peng, 2023; Yin et al., 2023), many questions remain unanswered. Techniques such as prompt-engineering (Wei et al., 2021 and utilizing diversified outputs (Wang et al., 2022a) aim to increase the quality of outputs. However, the effectiveness of these techniques often depends on the fortuitous alignment of prompts initial conditions, making them labor- intensive since the tuning process must be tailored for each task.
In pursuit of refining the behavior of LMs, some researchers have begun explore the anchoring effect (Kahneman et al., 1982)- well-known cognitive bias where initial information exerts dis- proportionate influence demonstrated to extend to LMs. For example, through effective prompting, the outputs generated by LMs can be steered towards specific intent (Jones & Steinhardt, 2022). Similarly, emphasiz- ing the first few sentences of a long context enhances the model's 's overall comprehension of the
subsequent judgments. Intriguingly, this cognitive principle has been
Equul contribution

---

Published as a conference paper at ICLR 2024
Base Instruction
[Tack Typee Reweiting
Logils
Instruction
generate paraphraee of that question without changing
meening of It. ... .Skkpp.....
Base or Noisy
Instructive Decoding
Noley Instruction
Instrudioo-turee
Inpot
LLM
Distracted
Opposite
Question: What USA population?
Always respond with the
Z
opposile of what you're asked. Y ou never get it right.
Finall Logits
Base Response
X
Our Response
Zq
The United States America has population of 127 million people.
What's the total population the United States?
Overview of Instructive Decoding (ID). The example in this figure is from task4 mt_.cmm.a.paaaphrraee. -eeeuettttteeeeeeeeeeee weweestiinggeneeeeeeeeeeee on.generation in SUPNATINST (Wang al., 2022c). The original response not only fails to meet the task requirements (Question Rewriting) but also contains incorrect information' In contrast, ID generates a more relevant response by refining its next-token predictions based on the noisy instruction (here, opposite prompting is used for ID).
Figure 1:
content (Malkin et al., 2021). Given these observations on LMs parallels that mirror human tenden- cies-and the influential role of initial prompts, we hypothesize that the strategic application of the anchoring effect could substantially improve LMs' fidelity to instructions.
In this work, we propose Instructive Decoding (ID) (Figure 1), a novel method that enhances the attention of instruction-tumed LMs towards provided instructions during the generation phase without any parameter updates. The essence of ID lies in the introduction of noisy instruction variants. These designed to anchor the model's s output in a specific direction, potentially away from the most optimal predictions. This deliberate steering enables a clear anchoring effect within the language models, facilitating contrastive approach in decoding process. Our range of variants spans from simple strategies such as instruction truncation and more aggressive alterations, the most extreme of which is the opposile instruction. By intentionally introducing such deviations, ID capitalizes on the resulting disparities. Within contrastive framework, next-token prediction logits that are influenced by the noisy instructions are systematically compared those derived from the original instruction. This process refines the model's 's responses to align more closely with the intended instruction.
Experiments on unseen task generalization with SUPNATINST (Wang et al., 2022c) and UNNATINST (Honovich al., 2022) held- out datasets show that instruction- suune models enhanced by ID consistently out- perform bascline models across various tups. Intriguingly, Tk-XL combined with method outperforms its larger version, TK-XXL, with standard inference (Figure 2).
41:00
47.43
BMG
Plyputtee Daloort Waryet J000
Roage- g
SBL 1t
25.30
Models not previously trained the SUP- NATINST dataset, including Alpaca (7B) and TO (3B), also show marked enhance- ments in performance. Additionally, the overall Rouge-L score of the GPT3 (175B) is strikingly competitive, closely mirroring the performance of OpenSNI (7B) when augmented with method. We further observed that ID's generation exhibits in- creased both adherence to the instruction and an improvement in semantic quality. To provide comprehensive understanding, we investigated the anchoring effect of noisy instructions. Our findings suggest that as the model's comprehension of the noisy instruction intensifies, the anchoring effect becomes more potent, making ID more effective. Our main contributions are as follows:
20
15AE: 11-13 -1hm
TTADOL
Openr Ope
Trstraction-lu SapPoal
Sup Nat
Figure 2: Zero-shot Rouge-L comparison on the SUP- NATINST heldout dataset (Wang et al., 2022c). Models not instruction-tume on SUPNATINST are in blue dot- ted boxes, while those instruction-tun tuned are in green.
According the 2022 U.N. Revision, the population of USA is approximately 338.3 million as of 2022.
2

---

Published as a conference paper at ICLR 2024
We introduce Instructive Decoding (ID), a novel method to enhance the instruction following capabilities in instruction tuneo LMs. By using distorted versions of the original instruction, ID directs the model bring its attention to the instruction during generation (Section 2).
We show that steering the noisy instruction towards more degrading predictions leads to improved decoding performance. Remarkably, the opposite variant, which designed for the most significant deviation from the original instruction yet plausible, consistently shows notable performance gains across various models and tasks (Section 3).
We provide a comprehensive analysis of the behavior of ID, demonstrating its efficacy from various perspectives. The generated responses via ID also improve in terms of label adherence and coherence, and contribute to mitigate the typical imbalances observed in the standard decoding process. (Section 4)
INSTRUCTIVE DECODING
In this section, we present instructive decoding, method designed to enhance the response generation of instruction-ture tuned models. By leveraging the responses derived from noisy instructions, our approach employs contrastive technique to refine generated responses, ensuring they are more closely aligned with provided instructions.
2.1 PRELIMINARY
In the context of an auto-regressive language model, denoted as MM parameterized by 0, the primary 3/.) Within the ICL framework, a specific demonstration, represented as I, is supplied in conjunction with the context x. The language model Ma then computes the logit for the th token, symbolized as 2, E RIVI equal to Mo(uelT, Vct). wherein V is the vocabulary set. Consequently, the probability of output sequence can be formally expressed as:
goal is to generate an output sequence Vc+++ = . when presented with an input sequence
(o(y)
II pa (o(ve 2,,1)
where po(yel T, Vee) is the probability for the next token prediction derived from the softmax function applied to 2, It can either be the token with the highest probability (i.e., greedy decoding) or sampled from its distribution (e.g., nucleus sampling (Holtzman et al., 2019)). In the broader scope of task generalization with previously unobserved instructions, the demonstration / takes the form of the guiding instruction. Depending on the specific context or setting, a few examples can be incorporated to enhance the learning process. Generally, predictions of the instruction tuned models are derived from both the context x and the given instruction 7, which play pivotal roles (Eq. 1).
2.2 MOTIVATION AND OVERVIEW INSTRUCTIVE DECODING
A significant challenge in instruction following is ensuring that the generated tokens intrinsically adhere to the instruction J. While the dominant strategy involves enriching the dataset with numerous, diverse, and creatively curated high-quality tasks, this approach is both labor-intensive and compu- tationally expensive. It requires new training cycles and does not always produce improvements commensurate with the effort invested. Consequently, there is growing interest in exploring more sustainable and effective alternative strategies for enhancing instruction-tuned models.
Drawing inspiration from cognitive science, we highlight the anchoring effect, a well-known cog- nitive bias in which initial information exerts a disproportionate influence on subsequent judg- ments (Kahneman et al., 1982). Recent studies have hinted at this principle being relevant LMs. where the LM's 's predictions are significantly conditioned (i.e., anchored) on the given context (Jones Steinhardt, 2022; Malkin et al., 2021). Based on these findings, we hypothesize that the strategic of the anchoring effect could refine the responses of instruction-tuned models by leveraging the discrepancies between the predictions that are anchored on different instructions.
Contrastive Decoding (CD) is straightforward technique that improves the performance of LMs by comparing two sets of predictions (Li et al., 2022; Liu et al., 2021). In this approach, predictions from performifn primary model contrasted against those from less accurate 'amateur' model. The goal is to differentiate the primary model's outputs against the less reliable outputs from the amateur model during the decoding process. Despite its simplicity, the need for two
3

---

Published as a conference paper at ICLR 2024
Algorithm 1: Instructive Decoding
INPUT Language model Ma, base instruction sequence I, noisy instruction sequence 7, initial prompt sequence z and target sequence length T, smoothing coefficient c.
1: Initialize & 4-1
2: while | < T do
3: 2z, 2t Mo(ye yet),,
4: = arg (OF c* 2,))
5: set t+1
6: end while
models limits its broad applicability, and its utility in cioruction-folloiing scenarios remains largely unexplored. To this end, we propose Instructive Decoding (ID), a novel method ensure that the model's output closely aligns with the given instruction. Leveraging the anchoring effect, ID incorporates these principles into the Contrastive Decoding framework by introducing noisy variants of the original instruction. These variants are designed to subtly mislead the model into generating deviated responses based the noisy instruction yet plausible. The comparison between the original instruction and the noisy version helps the model identify and correct biases (c.g.. inherent model bias and input bias), resulting in outputs better aligned with the intended purpose. To delve deeper into the mechanics, during decoding, the model contrasts the logits originating from the original instruction, with the logits I, originating from the noisy instructions, as described in Algorithm 1.
2.3
A COLLECTION OF NOISY INSTRUCTIONS FOR INSTRUCTIVE DECODING
We aim to design collection of noisy instructions that harness the anchoring effect while maintaining task fidelity. Key guiding principles for our noisy instruction design include:
Automated Perturbations: To ensure scalability and minimize manual intervention across diverse tasks, we inject perturbations into the instructions. These perturbations include deletion, shuffling, random word insertion.
We systematically create prompts that elicit counter-intuitee yet plausible responses, thereby producing a deviation from the expected responses.
Contrastive Elicitation:
In line with the principles outlined above, we employ the following noisy instruction variants. Full-text examples of these variants are displayed in Figure 3.
Trunc-Shuf: Words from the instruction are randomly truncated and then shuffled. This challenges the model to deal with both missing words and altered word sequences.
Null: The model receives only onput-output pairs. This evaluates its inherent ability to comprehend text and identify biases without any guiding instruction.
3. Rand Words:
Random words from the Natural Language Toolkit (NLTK) (Loper & Bird, 2002) replace the original instruction. This places the model an environment filled with semantic noise, requiring it to distinguish meaningful signals.
Opposite: a contrarian approach, the instructions contain misleading directives like " "Awways respond with the opposite of what you're asked. You never get right. An),n".. Such directives confront the model with conflicting guidance, helping it better align with the base instruction.
Unless specified, in the Experiment Section, we configure the noisy instructions include one random word (Rand Words) and set the truncation ratio to 0.6 (Trunc-Shuf).
Nail
Ranl Truns
Traxn-haf
Rand Words
Sapa: Quoliie whal the popuiaton?
New eompletee tol owrni enzpple
Defiiition genende perpprmn azzner sboud
Defrstion: quedior greenss withus devid Tour the it la daille informsioon Now comples the fallowing stample tngatt Quvetton whitls toe pepelntoo?? Outzat
unbeleel bnddgstoness wamin cxlaniic cosotatccnees jangl Amattel Sepbodoun uxneastning prodisiaatato Now coepleto the tolowingg comple- Ortpet
Ontpot
th: clergtec gown
Oppxnite
Qunines aluata pap dutinn?
supord with appoata wius.
the quequuedee..
edeve night
Now comp ete the tellownng exargle- lxput Quaitiie: viait dee populatio? Ouzut:
New comulkta the dalowing esanilee
tauu Questic what the popuazon? Orlgat
Other Noisy Templates...
Figure 3: Full-text examples for collection of noisy instructions for instructive decoding on
task4 42 com qa wwapphaase.qusstiiee gew.ratiinn.o.nn orr.gga...raaasseeepeeeeeeeeeeoa
4

---

Published as a conference paper at ICLR 2024
3 EXPERIMENTS
3.1 EXPERIMENTAL SETUP
Datasets For our experiments, two datasets are utilized: SUPNATINST (Wang et al., 2022c) and UNNATINST (Honovich et al., 2022). Both datasets feature diverse collection of crowd-sourced NLP tasks. In SUPNATINST, each task formatted as a 'Definition' prompt that acts as the instruction. For zero-shot evaluations, only the 'Definition' is utilized, whereas two positive demonstration examples are incorporated for few-shot evaluations. Our experiments focus solely on the English segment of the dataset, and 100 instances per tasks are used for evaluation following Wang et al. (2022c). This subset comprises 119 evaluation tasks, grouped into 12 categories:
AC: Answerability Classification
DAR: Dialogue Act Recognition
QR: Question Rewriting
CEC: Cause- Effect Classification
GEC: Grammar Error Correction
TE: Textual Entailment
CR: Coherence Resolution
KT: Keyword Tagging
TG: Title Generation
DT: Data-to-Tex
OE: Overlap Extraction
WA: Word Analogy
The UNNATINST dataset features L.M-generated instructions based an initial set of 15 seed samples. From its 64,000 samples, we evaluate subset of 10,000.
Models We use the Tk-instruct models (Wang et al., 2022c), instruction-tunet from T5-LM (Lester et al., 2021). These models are trained across 757 english tasks from the SUPNATINST training split over epochs, with each task comprising 100 samples. Our evaluation primarily involves three sizes of Ik-Insrutt models: Large (770M), XL (3B), and XXL (11B). While Tk-XL and Tk-XXL come from publicly available checkpoints, the 770M model is manually trained under the same settings as the other Tk-instruct models. Additionally, TO (3B), Alpaca (7B), and Open-instruct-SNN (OpenSNI) also used for further evaluations. TO model also fine-tunes T5-LM (Lester et al., 2021) using task prompts sourced from PromptSource (Bach et al., 2022). Alpaca (Taori et al., 2023) fine-tunes the LLaMA (Touvron et al., 2023) based on a style outlined by Wang et al. (2022b), whereas OpenSNI (Wang et al., 2023b) is a fine-tuned version of LLaMA on SUPNATINST, marking distinct way of use from Alpaca. In our experiments, greedy decoding primarily employed for these models.
We examine the outputs of instruction tuned LMS on unseen tasks. Unless specified, all evaluations conducted in zero-shot setting, where the models perform tasks based solely on instructions, without any demonstration examples. Task performance is measured using the Rouge-L. score (Lin, 2004), which measures the overlap between generated and reference sequences, and is often used for open-ended tasks as Wang et al. (2022c). Adding to the Rouge-I. score, classification tasks further use the Exact Match (EM) metric, which measures whether the response precisely matches a pre-defined label. To better evaluate semantics not captured by metrics like EM or Rouge-L, we introduce two additional metries: Label Adherence and Label Coherence. These metrics offer insights into how closely the generated responses adhere to the provided instructions. Detailed explanations of our evaluation metrics are as follows:
Evaluation Metrics
Label Adherence (LA): LA checks if the response stays within the label space defined by the instruction. regardless of its agreement with the golden label. For example, if the instruction specifies answers as 'True' or 'False', any response within this set is deemed conforming.
Label Coherence (LC): This metric evaluates the semantic alignment of the response with the gold label, allowing for near-equivalent answers. For example, responses like 'Correct' may align with a gold label of 'True'. We compare responses against an expanded set of gold labels with semantically equivalent expressions.
For more comprehensive evaluation, LA and LC are primarily measured on classification tasks identifying 58 tasks among the 119 unseen tasks in SUPNATINST, which contains the predefined labels. Although adherence and coherence are valuable for open-ended generation, focusing on classification ensures thorough evaluation. For clarity, an example illustrating the relationship between EM, LA, and LC is provided with further details on evaluation in Appendix D.
3.2 PERFORMANCE ON UNSEEN TASK GENERALIZATIOON
Result Overview Table I displays the results when applying Instructive Decoding (ID) to the Tk-Instruct models and OpenSNI-7B model. ID consistently outperforms the baseline model, which
5

---

Published as a conference paper at ICLR 2024
Table 1: Zero-shot Rouge-L score on unseen tasks in the held-out set of SUPNATINST (Wang et al., 2022c) is evaluated with Tk-instruct families and OpenSNI-7B. Green circles indicate improvement over the Baseline with the sample model, while red eircles denote no improvement.
Medel
Method
Overall
AC
CBC
CR
DT
DAR
GEC
KT
OE
QR
TE
TG
WA
Baseline
41.10
55.95
$4.33
36.32
30.55
40.72
W6.0n
51.16
21.30
55.19
42.15
31.51
12.21
Trinc-huf
50662
55.56
86.62
46
Th: Large
Nall
41.79 0
50.92 e
55.05
42,00
20.12
41.10
$6.62
47.28
25.84
56.26 0
36.16 0
31.83
11.90
Rand Words
41.77
50.54
55.66
42,09
29.57
41.06
86,30
47.92
25.42
56.14
45.97
32.24
12.15
Opposite
42.21
52.74
56.14
42.66
32.34
Biseline
45.36
50,00
99.73
43.94
34,01
58.15
87:07
50.08
17.09
$4.01
46.45
36.24
27.29
Trunc-huff
46.37
48,80
62,13
45.85
33.03
$7.76
86.66
$4.21
15.50
51.61
50.55
36,69
52.46
TI-XI
Nuil
46.35
45.75
62.01
32.42
59.52
95.79
$2.43
50.96
36.41
Rand Woods
46
40.05
62.28
45.85
$2.50
54.71
86.45
53.53
14.86
$2.01
$1.24
56,
52.21
Opposite
46.69
50.73
61.93
45.89
33,63
87.56
55,00
16.32
51.51
50.47
37.33
33,05
Biseline
46.01
59.25
33,91
59.05
HUI
43.53
26.78
57.70
35.06
19.13
Trinc-shuf
46.96 0
61.28
59.5
56.03
33.52
60.76
82.77
49.14
25.00
$2.66
56
21.37
TkXXLL
Nail
47.29
60.69
59,75
36,07
33.44
61.83
83.15
48.1
27.35
53.36
56 99
36.32
22,91
Rand Words
47.25
61.
36,59
39.57
82.67
47.92
25.77
53.54
56
23,10
Opposite
47.43
60.77
60.01
35.91
33,79
60.51
$1.06
48,66
25.16
52.98
58.56
36.11 @
22.43
Baseline
48,05
54.36
60.87
51.83
36.34
54,00
81.85
29.60
22.15
48,51
52.50
34.56
43,33
Trunc-suff
48.45 0
61.03
65.83
43.31
37.63
57
82.57
46.1
27.33
51.94 e
54.35
34.00
Ope4SNI-78
Nail
49.04
61.64
66.19
42.75
38.90
$7.48
83.58
48.50
24.20
51.99
56.17
33.44
34,50
Rand Woods
65.90
83.11
35.21
Opposite
49.47 6
62.26 e
66.53
42.51
39.32
57.41
83,85
51.96
23.60
54,03 0
33.68
36.30
34.56
employs only the standard instruction, as indicated by higher overall Rouge-L scores. This per- formance advantage is evident across all types of noisy instructions. Notably, while larger models generally yield higher scores, the improvements are not uniformly distributed across task categories. For instance, the 'OE (Overlap Extraction) task shows a slight performance decline, which hints at possible architectural limitations for learning in this specific task Nevertheless, the 'opposite' variant consistently results in the most significant improvements in Rouge-L scores across all model sizes, thus affirming the robustness of our method.
From Degradation to Enhancement: The Two-fold Impact of Noisy Instructions When used in a standard decoding process, noisy instructions lead to a significant decline in performance for generated responses. However, when integrated into ID, these instructions actually enhance performance. We attempt to unveil the relationship between such degradation and its enhancement with ID (Figure 4 (a)). When replacing the original instruction with noisy variant during the decoding process, a noticeable drop in Rouge-l. scores occurs, as shown on the x-axis labeled 'degradation'. The y-axis displays the performance improvement gained through ID when using these noisy instructions. Interestingly, we find a strong positive correlation between the initial drop in performance and the subsequent improvement when using ID. This correlation is quantified using the Pearson Correlation Coefficient (R in Figure 4 (a): Cohen et al. (2009)). The more substantial the initial drop caused by noisy instruction, the greater the performance gain when it is integrated into ID. Notably, the 'opposite" instruction, which causes the most significant initial decline, results in the largest performance boost when used with ID.
Comparative Winning Rates of Base vs. Ours Figure 4 (b) illustrates tasks where ID outperforms the baseline, as measured by the Rouge-L score. This improvement is consistent across a range of tasks, regardless of model size. Although the overall Rouge-L score for Tk-XXL is on par with that
Tk-Large
Tk-XL
Tk-XXL
OpenSNI-7B
Base vs Ours (96)
100
O
OpurSW 75 (ow) 5780)
Improvemont
rand words
rid traee
2.2
Hend t
0.6-0
6.1
Degradation
(a) Degradation vs. ID Boost
(b) Comparative winning rates of Base vs. Ours
Figure 4: (a) Correlation between performance degradation with noisy instructions and improvement with those used in ID; (b) comparative winning rates of Base vs. Ours. on held-out tasks. The blue bars show base method wins, while the green bars indicate our method's dominance.

---

Published as a conference paper at ICLR 2024
of Tk-Large and Tk-XL, distinct improvements observed across tasks when ID is used with larger models. This synergy appears to optimize the potential of the larger models.
Exact Match
Label Adherence
Label Coherence
55
88
60
baseline trunc shur
Metrics
50
56
null
52
rand words posite
80
48
Tk-Large
Tk-XL
Tk-XXL
Tk-Large
Tk-XL
Tk-XXL
Tk-Large
Tk-XL
Tk-XXL
Figure 5: Evaluation three different scales of Tk-Instruct models (i,e., Large, XI., XXI. with different noisy instructions for instructive decoding over classification tasks in heldout dataset of SUPNATINST. Each figure shows the performance changes from applying ID.
Granular Performance Analysis on the Classification Tasks We conduct an in-depth analysis of 58 classification tasks from SUPNATINST to scrutinize the shifts in their response outcomes in detail (Figure 5). The analysis segmented into three metrics: EM, LA, and LC. A clear trend emerges: the model size increases, EM scores also rise. However, when examining the LA and LC metrics based on baseline responses, the Tk-XL model outperforms the Tk-XXL model. This suggests that while larger models excel at strictly adhering to provided instructions, smaller models are more effective at generating semantically congruent outputs within the given instructional context. With the incorporation of ID. performance patterns remain largely consistent across different model sizes and noisy instructions. Specifically, as model sizes increase, the 'opposite' variant significantly improves the performances, particularly in the LC metrics for the TK-XXL model. The random 'trunc-shuffle' variant exhibits significant boost in LA scores as model size grows, highlighting the complex interplay between model sizes and their responsiveness instructions.
Table 2: Rouge-L scores cross-evaluated
Table 3: Rouge-L scores under a few-shot sce- nario across different models. We set to 0.2.
across different models and datasets.
Dataset
UNNATINST
SUPNATINST
Model
Tk- Tk-Large
Tk- XL
Alpaca- 7B
Model
Tk-Large
TO-3B
Alpaca-7B
baseline
47.63
$4.34
37.06
baseline
43.25
26.58
23.61
null
47.94
54.78
38.75
null
44.57
29.33
31.21
null (2 shots)
46.95
54.41
38.07
rand words
44.44
29.49
30.93
opposite
48.08
54.80
37.79
opposite
43.42
29.46
31.38
opposite shots)
47.01
54.51
37.55
3.3 ABLATION STUDY
Generalization Capabilities of ID To further assess the adaptability and effectiveness of ID. cross-evaluate models in the following way: models trained on SUPNATINST are tested UNNATINST and models not trained on SUPNATINST are assessed using the SUPNATINST test set. Table 2 shows the results, measured through the overall Rouge-L score. For the Tk-Large model evaluated on the UNNATINST training set, ID consistently outperforms the baseline, even if the 'opposite" variant isn't the top performer. Models trained other datasets, such TO-3B and Alpaca-7B, also perform better with ID. Notably, there is a significant performance boost, especially for Alpaca-7B. This indicates that ID effectively addresses the shift between training and test distributions, highlighting its versatility and robustness broadly applicable solution.
Sensitivity of Smoothing Coefficient Figure shows the influ- our method's performance. This
ence of the hyperparameter parameter adjusts the smoothness of logits derived from noisy instructions. Although our typical choice for was 0.3, we eval-
Rouge-L
uated ID-null across a range of e values, spanning from -0.5 to
0.5 0.01 intervals. Performance tends to decline with negative values, as the model becomes increasingly biased toward the noisy instruction. Conversely, excessively positive values (above 0.4) lead to deterioration in performance. Interestingly, the model's s performance stabilizes between 0.I and 0.4, indicating certain level of robustness to variations in within this range.
Strength
Figure 6: Overall Rouge-L, scores scores across varying values with 'null" instruction in ID.

---

Published as a conference paper at ICLR 2024
task39]_causst relntioship
task1529 schalll 1 cassiiction
Keyword Tegging
Word Analogy
Yia ute
Probubiiity
Probabiiiity (True)
Probabiiity (Truc)
Oppacterig
Oaganiile and Sagget 1XD1
(a) Distribution shift in classification
(b) Visualization of input embeddings
Figure 7: (a) Shift in responses for binary classification tasks using Tk-XL, comparing baseline, ID with 'Opposite', and ID combining 'Opposite Base Inst'. (b) t-SNE visualization of embeddings for 'Keyword Tagging (KT)) and 'Word Analogy (WA)', extracted from the Tk-XXL encoder by concatenating the instruction and input.
Few-Shot Generalization Here, we evaluate how ID performs in the presence of a few positive demonstration examples (i.e., few-shot evaluation). The results are presented in Table 3. In this table, the terms 'null' and 'opposite' refer to the use of noisy instructions without examples, while '(2 shots)' indicates the incorporation of two positive demonstration examples. The table shows that ID's performance gains are more modest in the few-shot context than in the zero-shot context. This is likely because the baseline performance is already improved by the inclusion of examples, thereby diminishing the benefits of perturbations from ă. Nevertheless, we find that the negative impact of noisy instructions is relatively minor, as the provided examples help to clarify the task's intent.
DISCUSSION
Qualitative Analysis on ID with Opposite As Figure (a) demonstrates, the baseline shows strong label adherence but often settles on a single label. The introduction of the 'Opposite" technique diversifies these responses, as evidenced by tasks previously biased toward "True' now yielding more balanced outcomes. Specifically, there is marked increase in the prediction probabilities for tokens that are not the top-ranked predictions guided by the original instruction. This not only expands the instruction-guiided output space but also emphasizes the increased likelihood for alternative tokens. This observation is evident when the data points in the figure gravitate closer to the origin. Intriguingly, combining the original instruction with the noisy instruction prompt does not lead to improved performance. Although there is a shift away from distinct 'True' or 'False' medictions-iinicalle
smoothing effect- this shift does not reverse the predictions. We conjecture that including the original instruction in the contrastive prediction may inadvertently anchor the model's responses, altering their magnitudes but not their directions.
Visualization of Embeddings: Evidence of Anchoring Effect Figure (b) provides t- SNE (Van der Maaten & Hinton, 2008) visualization of input embeddings from category KT and WA, extracted from the TX-XXL encoder. This visualization serves empirical evidence for the impact of various noisy instruction variants. Notably, unique clusters form for each type of instruc- tion embedding, indicating that the encoder interprets these noisy instructions differently, thereby exerting different anchoring effects- beneficial for ID. This phenomenon is clearly reflected in the WA category, consistent with the improvements by our method. In contrast, some embeddings in the KT category overlap, suggesting a limited distinction between the original and noisy instructions. This weakens the anchoring effect and results a decline in Rouge-L. scores for KT. This observation suggests that as the model gets better at understanding noisy instructions, the performance of ID) usually improves as well. This is often the case when using higher-performinn models.
On the Utility of ID over Contrastive Decoding We ex- amine the synergistic effects of integrating ID with the use of amateur models (ID- ---amtu for across various In-Instruct model families Figure 8. More precisely, we feed a smaller amateur model with the noisy 'opposite" instruction in the ID-amateur method. This approach is compared with the stan- dard Contrastive Decoding (CD, Li et al. (2022)) with the original instruction for analysis, where T is temperature for amateur. Using Tk-small with XL-XLL1 in CD modestly sur- passes ID-amateur due to smaller models' limited grasp of noisy instructions. As the 'amateur' model size grows, CD's
Rougest
Telarce
(plla Wales
wea1
Ling:
Amateur Model
Figure 8: CD Am-teur perfor- mances across Tk-instruct models.
8

---

Published as a conference paper at ICLR 2024
performance diminishes, highlighting its size sensitivity. Conversely, ID- amateuu maintains consistent adaptability across diverse model sizes. To sum up, D-amateer method maintains performance across model scales while mitigating issues inherent in standard contrastive decoding.
5 RELATED WORK
Instruction-tuned Language Models Instruction tunning is method to fine-tune pre-trained LMs to better follow natural language instructions (Wei et al., 202 Sanh et al., 2021). This fine-tuning process has demonstrated consistent enhancements in the model's ability to generalize to unseen tasks, particularly in zero-shot scenarios (Taori et al., 2023; Xu et al., 2023; Wang et al., 2022c; Peng et al., 2023; Ouyang et al., 2022: Chung et al., 2022). Previous studies indicate that expanding the breadth, volume, and ingenuity of tasks for training improves instruction-tuning even further (Wang al., 2022c;b; 2023c). While some efforts also use human feedback (Ouyang et al., 2022; Wu et al., 2023a; Ziegler et al.. 2019; Song et al., 2023), this paper focuses on the instruction-tuned models that are trained on task datasets in a supervised manner.
Impact of Instructions on Generated Responses Understanding how generative LMs interpret instructions remains an active area of discussion. It has been suggested only the essential tokens directly related to the expected response influence the performance (Yin et al., 2023). However, instruction-tunee LMs are so heavily conditioned on pre-trained knowledge that it is difficult to override such conditioning through the prompted instructions, et al., 2023; Wu et al., 2023b). Recent research indicates that the success of instruction-tung -unnng is contingent upon the familiarity of instructions LMs encounter during their training phase (Chia et al., 2023; Liang al., 2023). More specifically, LMs trained with certain instructions can exhibit improved generalization on unseen tasks, even when presented with misleading instructions during evaluation (Sun et al., 2023; Kung & Peng, 2023). In zero-shot scenarios. this sensitivity to instruction variations becomes especially evident (Sun al., 2023; Gu et al., 2022). In this work, we suggest this sensitivity can be leveraged by contrasting responses generated from noisy instructions.
Contrast in Text Generation The concept of using contrast to improve generative models (Ho Salimans, 2022; Li et al., 2015) has been studied in various ways in text generation (Shen et al., 2019; Yona et al., 2023; Liu et al., 2021; Li et al., 2022). For example, Contrastive Decoding (Li et al., 2022) aims to maximize the output probability by contrasting less proficient model with expert-level model. Meanwhile, Coherence Boosting enhances long-range contextual understanding by giving more weight distant words,(Malkinn et al., 2021). This contrastive approach has demonstrated its effectiveness in diverse areas through its variants, such as text detoxification (Liu et al., 2021), resolving knowledge confticts (Shi et al., 2023), mitigating bias in input text (Yona et al., 2023) and boosting response truthfulness (Chuang et al., 2023). Our study extends this line of work but places emphasis on the role of instructions the input text. Also, unlike previous studies, we present findings that it is possible to utilize inputs that cause severe performance degradation, experiments show that contrasting predictions based on noisy instructions can significantly improve the generalization of instruction-tunee LMs on unseen tasks.
CONCLUSIOO
This paper explores the challenges faced by instruction-tuned language models, especially when dealing with unfamiliar instructions, termed as unseen task generalization. Our approach is inspired by the anchoring effect, a cognitive bias where initial information significantly influences subsequent decisions. Based on this concept, we introduce Instructive Decoding (ID), a method that adjusts next-token predictions by contrasting them with those generated from manipulated version of the original instruction, termed the "noisy" instruction. Designed to counterbalance inherent model biases and potential input biases, these 'noisy" instructions guide the model's outputs towards contextually relevant but deviating paths. Our empirical results across multiple tasks confirm the method's 's efficacy. Notably, the 'opposite" noisy instruction, which offers the highest degree of deviation, emerges as the most effective variant for improving model performance. This highlights the significant role the anchoring effect can play in shaping the model's behavior. The simplicity of the ID approach, which necessitates no additional parameter updates, renders it compelling option to enhance instruction following of the generated responses. As the field of instruction-tuned models continues evolve, we expect that methods like ID will become crucial in extending their capabilities.

---

Published as a conference paper at ICLR 2024
Ethics Statement This work primarily presents no direct ethical concerns. However, from broader impact perspective, there are some potential implications related to systematic impact and possible misuse. These concerns are detailed further in the Appendix A.
Reproducibility Statement To ensure reproducibility, the main paper offers an in-depth exposition of the materials and experimental configurations. The organization is follows:
Section 2 This section provides the details of the noisy instructions employed throughout
our experiments. The accompanying pseudocode offers a more technical breakdown.
Section This section elaborates on the implementation specifies, including the pre-trained models, datasets, and evaluation metrics used. Additionally, the appendix furnishes the input format for models in the SUPNATINST dataset, offering further clarity on the dataset description.
Appendix
This section delves into the origins and specifications of the instruction- models employed in our study.
Appendix D Detailed methodologies for evaluating models using our proposed metrics, LA and LC, are elaborated upon in this section.
Appendix Comprehensive experimental configurations for integrating ID with other decoding techniques are documented. The expected responses from instances utilizing ID are also enumerated in this section.
REFERENCES
Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafcai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang. Jason Fries, Maged Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang. Dragomir Radev, Mike Tian-jian Jiang, and Alexander Rush. PromptSource: An integrated development environment and repository for natural language prompts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 93-104, Dublin, Ireland, URL
May 2022. Association for Computational Linguistics. doi: 10. 12118665//112222222leeeeee https: ac www.cannnhhoogggo org/ 2c1. ac1-demoo 9.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Yew Ken Chia, Pengfei Hong, Lidong Bing, and Soujanya Poria. Instructeval: Towards holistic evaluation of instruction-tuned large language models. arXiv preprint 230.:23300004757, 2023.
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models, 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling caaruction-finetunee language models. arXiv preprini arXiv:22 10. /416, 2022.
Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Benesty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. Pearson correlation coefficient. Noise reduction in speech processing,
pp. 1-4, 2009.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning. arXiv preprint 2.Xix:2301.00234 2022.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.
Jiasheng Gu, Hanzi Xu, Liangyu Nie, and Wenpeng Yin. Robustness of learning from task instructions. arXiv preprint 22. 2022.
10

---

Published as a conference paper at ICLR 2024
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. 2020.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207112996 2022.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint urXiv:I904. 09751, 2019.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:22/2.09689 2022.
Erik Jones and Jacob Steinhardt. Capturing failures of large language models via human cognitive biases. Advances in Neural Information Processing Systems, 35:11785-11799, 2022.
Daniel Kahneman, Paul Slovic, and Amos Tversky. Judgment under uncertainty: Heuristics and biases. Cambridge university press, 1982.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprini arXiv:2001.066 .08361, 2020.
Po-Nien Kung and Nanyun Peng. Do models really learn to follow instructions? an empirical study of instruction tuning. arXiv preprint arXiv:2305.1 11122335511388 11383, 2023.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficien prompt tuning. In Proceedings of the 202/ Conference Empirical Methods in Natural Language Processing, pp. 3045-3059 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10. 18151881/2222111... 12202....mmmmmmiin222.. URL https: //aclanthology org/ 2021. emnlp-mee ma inwain...
243.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity- promoting objective function for neural conversation models. arXiv preprint arXiv:/510.03055, 2015.
Shiyang Li, Jun Yan, Hai Wang. Zheng Tang. Xiang Ren, Vijay Srinivasan, and Hongxia Jin. Instruction-following evaluation through verbalizer manipulation. arXiv preprint arXiv:2307.10558, 2023.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint ar 10. 5097,, 2022.
Shihao Liang, Kunlun Zhu, Runchu Tian, Yujia Qin, Huadong Wang, Xin Cong, Zhiyuan Liu, Xiaojiang Liu, and Maosong Sun. Exploring format consistency for instruction tuning. arXiv preprint arXiv:2307.15504 2023.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Alisa Liu, Maarten Sap. Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. Dexperts: Decoding-time controlled text generation with experts and anti-experts. arXiv preprini arXiv: 105.03033, 2021.
Edward Loper and Steven Bird. Nltk: The natural language toolkit. arXiv preprint cs/0205028, 2002.
Nikolay Malkin, Zhen Wang, and Nebojsa Jojie. Coherence boosting: When your pretrained language model is not paying enough attention. arXiv preprint arXiv:2110.08294 2021.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. arXiv preprint arXiv:2104.0877, 2021.
OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
URL
https: https://apl. /api.
semanticsche ar. .or. :otp://ooouuuu:::55:5:::

---

Published as a conference paper at ICLR 2024
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprini urXiv:2304.03277 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv: 2tf::://000000000 10.08207, 2021.
Sheng Shen, Daniel Fried, Jacob Andreas, and Dan Klein. Pragmatically informative text generation. arXiv preprint arXiv: 000..090440300 2019.
Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Scott Wen-tau Yih. Trusting your evidence: Hallucinate less with context-aware decoding. arXiv preprint arXiv:2305.147733 2023.
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. arXiv preprint arXiv:2306. 7742, 2023.
Jiuding Sun, Chantal Shaib, and Byron C Wallace. Evaluating the zero-shot robustness of instruction- tuned language models. arXiv preprint arXiv:2306. 1270, 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang and Tatsunori B. Hashimoto. Stanford alpaca: An enstructionfoolloiing Ilama model. 2023.
https: //gttubb// com/
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ann Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: open and efficient foundation language models, 2023. URL https://arxiv. org/abs/23302. 1397,, 2023.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint ar.iv:2203. 2022a.
Yidong Wang, Zhuohao Yu, Zhengran Zeng. Linyi Yang. Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. Pandalm: An automatic evaluation benchmark for IIm instruction tuning optimization, 2023a.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint 1:Xi::2212.10560 2022b.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabsshi, Yeganeh Kordi, Amirreza Mirzaci, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, al. Sunprrnnaturalinsttucioos::: Generalization via declarative instructions on 1600+ nip tasks. arXiv preprint arXiv:2204.07700: 2022c.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. How far can camels go? exploring the state of instruction tuning on open resources, 2023b.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. arXiv preprint 2306:2306.0441 2023c.
12

---

Published as a conference paper at ICLR 2024
Colin Wei, Sang Michael Xie, and Tengyu Ma. Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning. Advances in Neural Information Processing Systems, 34:16158-16111 2021a.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. arXiv preprini SEti:2200.006522. 2021b.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint urXiv:2206. 07682, 2022.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grainee human feedback gives better rewards for language model training. arXiv preprint arXiv:2306.0106 2023a.
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. arXiv preprint arXiv:2307.02477, 2023b.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint ar WXX.:23304.122444 2023.
Seonghyeon Ye, Doyoung Kim, Joel Jang, Joongbo Shin, and Minjoon Seo. Guess the instruction! flipped learning makes language models stronger zero-shot learners. In The Eleventh International Conference on Learning Representations, 2023. URL https: /pwp//pppweeee.e////// net forum?
Fan Yin, Jesse Vig, Philippe Laban, Shafiq Joty, Caiming Xiong, and Chien-Sheng Jason Wu. Did you read the instructions? rethinking the effectiveness of task definitions instruction learning. arXiv preprint wrXiv:2306.01150 2023.
Gal Yona, Or Honovich, Itay Laish, and Roee Aharoni. Surfacing biases in large language models using contrastive input decoding. arXiv preprint arXiv:2305.073378, 2023.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303. 18223, 2023.
Lianmin Zheng, Wn Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li. Eric Xing, et al. Judging Ilm-as fim-as-a-jus aa--as----ude with mt-bench and chatbot arena. arXiv preprint arXiv:2306. 200.0:2300006888,, 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint TrXiv:2305.11206 2023,
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint
2019.
13

---

Published as a conference paper at ICLR 2024
Appendices
A
BROADER IMPACT
In advancing the domain of instraction-acherencce for Language Models (LLM), we introduce an innovative technique, Instructive Decoding (ID). Recognizing the potential paradigm shifts this method might instigate, we find imperative discuss broader implications, especially concerning efficiency, scalability, accessibility, systematic impact, and potential misuse.
Efficiency and Scalability Optimizing instruction adherence at the decoding level, as underscored by ID, presents pronounced advantages in both efficiency and scalability. Traditional endeavors to fine-tune instructions often Iean exhaustive training, entailing considerable resource commitments. This not only poses challenges for real-world applicability, especially with behemoth models or voluminous datasets, but also limits scalability. Our decoding-centrie method, the other hand, augments instruction adherence without extensive retraining. This reduction in computational overhead paired with the method's adaptability diverse tasks signifies pivotal step towards ensuring future large language models are both instruction-espponiiee and deployment-efficien..
Accessibility ID inherently fosters increased accessibility of instruction-tuned models to wider spectrum of users. A salient attribute of our methodology its efficacy in amplifying instruction adherence, even for models with more modest parameter count (up to 3B). This democratization is potent, especially when considering that our method eschews dependencies on vast datasets, high-end computational resources, specialized engineering teams. In machine learning landscape often characterized by escalating computational needs and intricacies, ID emerges as a beacon, rendering top-tier, istruction-adheren models accessible more expansive audience. This broad-based accessibility is poised to catalyze novel applications across sectors, enriching both the research community and the general populace.
Systematic Impact
The introduction our Instructive Decoding (ID) methodology offers a promis- ing avenue for democratizing advanced instruction-following capabilities in contemporary language models. Independent of their operational scale, organizations and researchers can leverage the en- hanced proficiency of LLMs without the typical burdens of exhaustive tuning. This democratization holds the potential to streamline and standardize Al implementations across multifarious industries. Nevertheless, with widespread adoption comes the imperative of rigorous monitoring to identify, mitigate, and rectify unforeseen biases or unintended consequences that may emerge.
The amplification of instruction-adhereen in models, while laudable, introduces vulnerabilities that may be exploited for malevolent purposes, such as disseminating misleading narratives or manipulating public discourse. It is our responsibility, as proponents of this technology. to instate robust safeguards, advocate for ethical deployment standards, and formulate stringent usage guidelines. Continuous emphasis should be placed on responsible application, vigilant oversight, and cultivating a user ecosystem that cognizant of both the potential benefits and inherent risks such advanced systems.
Potential Misuses
B LIMITATION & FUTURE WORK
B. LIMITATION
Generalization While our method has shown promising results in specific tasks and datasets, it remains uncertain how universally it can be applied across various cnatuuction-folloinng scenarios. languages, and cultures. Future research is essential to validate its effectiveness in diverse contexts and ensure it doesn't inadvertently introduce biases or inaccuracies in untested situations.
Robustness and Stability Our approach, though effective under the conditions tested, may exhibit sensitivity to slight variations in instructions other input parameters. This sensitivity might manifest
14

---

Published as a conference paper at ICLR 2024
as inconsistent outputs or varied model performance, emphasizing the need for comprehensive testing across a range of inputs to ensure stable and robust operation.
Resources To produce single output, our method necessitates two separate inferences. This inherent design, while facilitating the desired model behaviors, leads to increased computational overhead. As a consequence, there could a tangible impact on speed, particularly resource- constrained environments, and a potential increment in storage requirements due to the need to maintain intermediate representations or states.
Error Propagation Given our method's two-step inference process, there's an inherent risk of error propagation: inaccuracies or biases introduced in the initial inference might not only persist but could also be exacerbated in the subsequent inference. Addressing this challenge requires meticulous design and evaluation to ensure that initial errors don't compromise the quality of final outputs.
B.2 FUTURE WORK
Resource-Efficient ID As we explore deeper into the behaviors of instruction sertuuttioo-tueedd models and the efficacy of ID, clear trajectory emerges for future exploration: enhancing the resource efficiency of the Il) process. While our current methodology has showcased promising results, the computational overhead and time complexity associated with it remain areas of improvement. In future iterations, we aim to refine our algorithms to make the ID process not just effective but also leaner in terms of computational resources. This could involve optimizing the perturbation computations, streamlining the sampling process, introducing lightweight heuristics to guide the decoding. Such enhancements would make our approach more amenable to real-world applications, where both accuracy and efficiency are paramount.
Robustness on More Diverse Tasks Another direction for future research lies in testing the robustness of instruction-tuned models, especially with ID, across a broader spectrum tasks. While initial investigations are mainly focused on the analysis of SUPNATINST dataset, the potential of this approach could be unearthed by exposing the model to a gamut of diverse challenges from tasks multi-modal problem settings. Such expanded evaluation would provide deeper insights into the model's versatility and its adaptability to various task nuances. Furthermore, would be intriguing observe how the model, anchored by its initial instruction, fares in tasks that exhibit high levels of ambiguity or where the boundaries between classes are not starkly defined. Pushing the boundaries in this manner will not only test the model's resilience but also its capability to generalize from one context to another seamlessly.
intricate
ID for RLHF Enhanced-LLMSs Instruction tuning in a supervised manner equips models to respond precisely clear-cut tasks queries, but its prowess diminishes when faced with ambiguous or vague questions. Herein lies the significance of Reinforcement Learning from Human Feedback (RLHF). By integrating human feedback into the model's learning process, RLHF ensures that models can interpret and respond to less defined queries in a manner that aligns closely with human intentions. Introducing ID into RLHF-enhanced LLMs emerges as an intriguing avenue to further enhance this capability. While RLHF provides the foundation for models to comprehend and align with human intent, ID can be instrumental in refining the model's adaptability to instructions and user preferences. The amal amalgamall gamation of RLHF's continuous learning approach with ID's anchoring capabilities may lead to a more contextually adept and user-aligned model. In essence, this synergy could result in LLMs that not only grasp the intricacies of human intent but also consistently generate outputs that are both accurate and contextually relevant, regardless of the clarity or vagueness of the incoming query.
Theoretical Analysis for ID ID stands as distinct mechanism that aligns responses more to- ward a goal-oriented direction without the need for additional training; it augments the provided instruction to elicit more pertinent outputs from the model. Yet, while its practical benefits are becoming increasingly evident, a deeper theoretical understanding remains a pressing requirement. Specifically, understanding the interplay between the input that's 's's inarucctioo-augmenteed and how it influences the model's prediction is of paramount importance. A rigorous analysis should explore the level of perturbation this augmented instruction introduces into the model's decision-making
15

---

Published as a conference paper at ICLR 2024
process. Furthermore, the inherent trade-offs between the exact match, Rouge-L scores, and semantic coherence in relation to these perturbations need to be delineated. Establishing such a theoretical foundation would provide invaluable insights into how ID effectively alters model behavior, paving the way for more predictable and controlled outcomes. Future research endeavors focusing on these aspects unveil the precise mechanics at play, allowing for further refinement and optimization of the ID approach.
C EXPERIMENTAL SETUP DETAILS
Tk-Instruct Tk-Instruct is an instruction-tune model trained using SUPNATINST on the T5-LM within encoder-decode architecture. As previously mentioned, we employ the publicly available checkpoints [Tk-Instruct public checkpoints) for Tk-Instruct, specifically models such as 3b-def. 3b-def-pos, and / Ib-def, which -def models are zero-shot tuned model and -def-pos model is tuned with additional positive demonstration examples for few-shot generalization. For model sizes not publicly disclosed, we adhere to the training setup provided by Wang et al. (2022c) to perform fine- tuning, Only the definition and input are used for training the Tk-Small (60M), Base (220M), Large models (770M), whereas the TK-Large-def-pee deg---de-pos is trained with both definition and two positivie demonstration examples, each adapted from their corresponding T5-LM (Lester et al., 2021). The models are trained with a batch size of 16, for 2 epochs, using a learning rate of 5e-5. Due to the absence of an official validation task, training is conducted without splits and the last checkpoint used for experiments. The number of training instances utilized is 67, 825. For both training and evaluation, the combined maximum length of demonstrations and contexts is set to 1,024, while the maximum generation length limited to 128.
OpenSNI, TO, and Alpaca OpenSNI represents model trained the SUPNATINST for compari- son among instruction datasets as depicted by Wang et al. (2023b), following the methods of Touvron al. (2023). It has been fine-tuned with 96,913 training instances over epochs using a learning rate of 2e-5. Two publicly available variants of this model exists: 7B and 13B, with our experiments using the 7B variant from OpenSNI-7B public checkpoint. We observe a superior performance in the 7B model compared to the 11B variant of Tk-Instruct (i.e., Tk-XXI.). We attribute this not only to LLaMA's potent pre-trained capability but also the increased number of instances used in training. In the methodology proposed by Wang et al. (2023b), the fine-tuning is conducted with fixed template for both input and output to facilitate comparisons across instruction datasets. Notably, this differs slightly from the template of SUPNATINS" In experiments, we employ the SUPNATINSS template with the OPenSNI model. As seen Table 4, there is a significant performance difference when using the SUPNATINST template compared to the one used in training.
Table 4: Rouge-L score of OpenSNI-7B SUPNATINST with different input format
Method Format
SupNatInst
Open-instruct
baseline null opposite
47.85
46.20
49.04
48.70 49.47
48.94
We also use the TO-3B (Sanh et al., 2021) and Alpaca-7B (Taori et al., 2023) checkpoints from TO-3B public checkpoint, and Reproduced Alpaca-7B public checkpoint (Wang al., 2023a) in our experiments, repectively. We set maximum length of inputs and generation length to 1,024 and 128, respectively.
D METRIC DETAILS
Rouge-L Rouge-L (Recall-Oriented Understudy for Gisting Evaluation with Longest Common Subsequence) is one of the metrics under the ROUGE framework (Lin, 2004), used predominantly for evaluating the quality of summaries by comparing them to reference summaries. Rouge-L specifically utilizes the Longest Common Subsequence (LCS) approach. LCS captures the longest co-occurring in-sequence n-grams, words, bytes between the system-generated summary and of reference
16

---

Published as a conference paper at ICLR 2024
Table 5: Examples of expanded label space for evaluating Label Coherence (LC).
Task
Label
Keywords
entailment
entailment, entail, entails, entailing, Valid, en-
task1385 anli rl eentailment
tailments
neutral
neutral, neutrality, neutrally, neutrals, Unknown contradiction, contradictions, contradicts, con- tradict, contradicting, Disagree
contradiction
(e1.935dddcccibble.oom anoom ic.classification
weakener
weakener, weakens, weak, weaken, weakening,
a weak strengthener, strengthens, strong, strengthen, strengthening, a strong, stronger, strongest, strongly
strengthener
kusk392.inverse www.weeeeeeecsslll..eeee.a. rela- tionship
plausible not plausible
plausible, Yes
not plausible, No
The advantage of Rouge-L is that does not require predefined n-gram length like other ROUGE metrics (e.g., ROUGE-N), making it more adaptive to varying lengths of summaries and capturing fluent sequences more effectively. Given candidate summary and reference summary R, the precision P and recall R for Rouge-L are calculated as:
summaries.
Ptcs LCS(C, R)
|C|
LCS(C, R)
RLCS=
|R|
where IC| and | the lengths of the candidate and reference summaries, respectively, and LCS(C, R) denotes the length of the longest common subsequence between the two summaries. The Fl score for Rouge-L is then computed as the harmonic mean of the precision and recall:
PLCS X RiCs
Flacs = PLCS + RLCs
Due to its measurement efficiency, we choose Rouge-L as our main metric for zero-shot instruction following ability.
We opt for Rouge-L, as our primary metric for zero-shot instruction following capability. Other studies (Hendrycks et al., 2020; Ye al., 2023) have utilized methods such as ranking options by likelihood for possible labels to assess instruction following abilities. However, these methods not only fail to reflect the efficacy our ID but, when considering more practical instruction follow- ing scenario ceeeario-seccficcally open-ended text generation corresponding the provided instruction and context- context-Rouge-L L emerges as the more appropriate metric for representing the overall task performance.
While there exist frameworks, such as Alpaca Farm (Dubois et al., 2023) and Chatbot Arena (Zheng al., 2023), that evaluate the generation capabilities of Dnstruction-tuned models, they predominantly focus on assessing dialogue formats. As result, they are not ideally suited for evaluating IDs that aim improve zero-shot task generalization.
Label Adherence & Label Coherence For an in-depth analysis of ID, we measure LA and LC in addition to EM (Exact Match) across 58 classification tasks. The illustration of Label Adherence and Coherence is in Figure 9. To measure LA, we construct the space of all ground truth outputs for each task's instances and evaluate whether the generated answer resided within this space. Conversely, to comprehensively evaluate the LC of instruction-tuned LMs, we take scrupulous approach. Rather than solely relying on the default labels provided in the SUPNATINST dataset for classification tasks (Table 15), go step further. We manually select all classification tasks and deliberately extend their label space. By doing so, we aim to capture a broader range of potential responses the precise assessment of its semantic coherence.
model generates, ensuring
17

---

Published as a conference paper at ICLR 2024
Task Type
Answerability Classification
Definition
In this task you are given story and question regarding that story. You must judge whether the question is answerable based on the info given to you. Label the instances as "Answerable" or "Not Answerahle" based your judgment, the story and the question separated by new line character." "
Instance
Adler tried to place a fire ant in his brother's bed. He got bit by the ant several times. In addition, he caught an infection. Adler's dad was angry from what happened. He decided to send Adler summer camp. 'n Why did He decide to send Adler summer camp?
Coherence
Adherence
Outputs
(a) Only Coherence
Only Adherence
Expected Response
Answer:
Not Answerablle
Answerable
Because he was anery
Figure 9: The example of Adhernece and Coherence in task290.te Lmewhy -quest in Lanswerabil lity. classification tasks, the definition (i.e., instruction) contains not only seman- tic information about the task but also hints on how to respond. If an instruction-tunel model solely pursues adherence and conforms only to the label format (i.e., Only Adherence), may produce incorrect answers. Conversely, if it tries to align only semantically (i.e., Only Coherence), it deviates from the predetermined format.
Table 5 presents an example of an extended label space. For tasks like entailment classification, we expanded the label space by collating responses from experiments that semantically matched the ground truth labels such as 'entailment', 'neutral', and 'contradiction'. Additionally, we underwent further processing, such as removing special characters like "., "\n', "?", "!' and conducting com- parisons without upper case sensitivity, to ultimately create the extended label space used the LC evaluation. This manual label space enhancement not only increases the quality our evaluation but also provides deeper insights into how the model interprets and aligns its outputs with the intended semantics. Figure 9 shows the example of adherence and coherence for the needs of LA and LC.
E ADDITIONAL EXPERIMENTS
Table 6: Comparison between sampling-
srapeinn-bbased based decoding and greedy decoding. Top-k and temperature scaling adopted. Mean and standard deviation of seeds experiments reported.
Method
Top-k 40 & Temp 0.7
Greedy
original instruction
43.17 = 0.26
45.36
null rand words
41.61 = 0.20
46.35
41.60 = 0.26
46.46
Decode by Sampling We conduct experiments using greedy decoding. This is necessary because SUPNATINST comprises 1 119 tasks, which encompass not only generation but also classification and dnestion-answerrig tasks. Although sampling-bait based decoding aids in increasing diversity, it operates stochastically, which is not beneficial for classification examine whether ID has benefits from sampling, and the results are presented in Table 6. From the outset, observe performance degradation across all methods, including the baseline, with ID experiencing a particularly significant decline. As described in Section 4, we demonstrate that this outcome stems from the smoothing effect from the characteristics of ID. Because ID reduces the top1 probability by increasing the probabilities for other tokens, sub-optimal tokens can be easily sampled, leading generalization far worse than that of the greedy decoding.
Nevertheless,
CD ablations In Section 4, we discuss the application of Contrastive Decoding (CD) to ID for unseen task generalization. The comprehensive results for the hyperparameters that demonstrates the highest performance during experiments can be found in Figure 10. As previously mentioned,
18

---

Published as a conference paper at ICLR 2024
ID-amatuer
Small
--1
-1.
-.2
4..9
-2.6
Expert Large
Barc
-1.
4.2
0.29
-1.7
-3.1
Improvement
0.54
0.62
5.2
0.002
0.16
0.48
0.99
0.37
0.34
1.6
Small
Base
Large
XL
Small
Base
Large
XL
Small
Base
Large
Amatuer
Figure 10: Performance results after applying CD SUPNATINST in comparison the expert model's original score. We explore various values for т, representing the temperature parameter т for the amateur model, within the range [0.5,10.0]. The parameter a, which constrains the model's confidence, is set to 0.1 as in Li et al. (2022). For the ID-amateur approach, which introduces noisy instructions the 'amateur' model, examine the optimal value for among 0.1, 0.2, and 0.3.
while CD experiences significant sensitivity concerning model selection. the simultaneous use of ID's opposite instruction with CD (i.e. ID>amatuer) reduces this sensitivity. Even when the expert model is smaller than the amateur model, it displays more robust results, and the degradation is considerably less when compared to the standard CD. This can be attributed the fact that as the amateur model grows in size, it better understands the meaning of the opposite instruction, thereby producing significant noisy logits.
Table Performance degradation with increasing number random words in the noisy instruction for Tk-Large and Tk-XL models. This table highlights the trade-offs when introducing randomness instructions.
The number of random words
Model
1
3
5
10
30
50
100
Tk- Tk-Large
41.77
41.74
41.73
41.54
41.40
41.36
41.35
Tk-XL
46.46
46.39
46.34
46.30
46.29
46.25
46.11
Ablations on the Number of Random Words for Noisy Instruction To understand the influence of the number of random words in the noisy instruction, we conduct ablation experiments varying their count. In Table 7, performance metrics for Tk-Large and Tk-XL models across different random word counts are presented. As the number of random words increases, there a marginal decline in performance for both models. This suggests potential saturation point beyond which additional random words might not offer significant noise benefits. The results underscore the importance of adjusting the randomness level in the noisy instruction achieve optimal performance.
Table 8: Variation in performance with different truncation ratios in the Truc-Shuf approach for Tk- and Tk-XL models. The table showcases the resilience and adaptability of the models to varying degrees of truncation in the instructions
Trucation Ratio
Model
0.1
0.2
0.3
0.4
0.5
0.7
0.8
0.9
Tk-Large Tk-XL
41.68
41.67
41.57
41.87
41.60
41.70
41.61
41.73
41.66
46.37
46.31
46.39
46.60
46.21
46.45
46.30
46.26
46.67
Ablations on the Ration of Truc-Shuf performance, perform ablation studies varying the truncation ratio. As illustrated in Table
To ascertain the impact truncation on the model's
19

---

Published as a conference paper at ICLR 2024
Task ID Definition
task133, winowhy reason t plausibility detection
"In this task you need to indicate the plausibility reasoning for the pronoun coreference relations. Each of the provided inputs contains a sentence with a target /Skipp You should answer 'Correct' if the reasoning made sense, otherwise,
should answer "rong""
Instance
"Sentence: Thomson visited Cooper's grave in 1765. At that date he had been dead for five years.In Reason: The 'he' refers to cooper because dead people are the graves. Question: Is the above reasoning correct or wrong?" Golden Label: Correct
Original vs. Contrast Logits
Instructive Decoding Logits
Original
6-
Arsmmi
Gor
Siginall
lokens
Takens
Figure 11: An example on logits correction by ID with Tk-XL model.
we report the performance for Tk-Large and Tk-XL models across different truncation ratios. The table indicates that the models exhibit varying sensitivity to the truncation level. Notably, neither extreme truncation nor minimal truncation consistently maximizes the performance, suggesting an intermediate truncation ratio might be optimal. The results underline the significance of adjusting the truncation ratio to optimize the balance between the retention of task-relevant information and the introduction of noise.
Example on Logits Correction by ID The baseline response guided by the original instruction displays slight ambiguity, predicting tokens like 'True' and 'Correct' at relatively high levels, but also showing minor confusion with 'Anwwer', 'Fal', and "Yes". However, for the given task, the correct response should be 'Correct'. When using the 'null' with ID, the prediction scores across these tokens generally increase. By contrasting these outcomes, the model is further reinforced to adhere to the 'Correct' response, underlining the strength of ID guiding models towards the appropriate response.
F QUANTITATIVE ANALYSIS ON TOKENS GENERATED BY ID
Relationship between Token Probability and Consistency In the context of ID, noisy predictions derived from noisy instructions are used in contrast. These noisy predictions typically exhibit different tendency compared to those originating from base instructions, as illustrated in Table 9, Specifically, when decoding in 'opposite', classification tasks (CLS) often demonstrate mismatch in the max token index between base and noisy predictions. However, the contrasting in ID is modulated through the use of c, which significantly reduces the number of altered predictions. In generation tasks (GEN), there is lower proportion of differences between noisy and base predictions compared CL.S tasks. This phenomenon attributed to the progressive influence ID refined tokens on noisy predictions more tokens generated. This is particularly pertinent in generation tasks, which involve creating longer texts. Also, the inherent characteristics of the language model frequently result in natural predictions within the responses being generated.
Figure 12 shows the density of maximum probability from the token distribution of base predictions, distinguishing between tokens that are altered remain unchanged by noisy predictions. Setting 0.3, we observe that base predictions that are confident tend to remain unchanged, while those that unconfident may alter due to contrasting. Thus, predictions generated from noisy instructions
20

---

Published as a conference paper at ICLR 2024
do not always perfectly contrast with those from base instructions. If the prediction from the base instruction is confident, remains consistent. However, in cases where the instruction-tuna model poorly understands the base instruction, leading unconfident predictions, the noisy instruction can be beneficial.
Table 9: A comparison of predictions from base instruction and predictions from noisy instruction (Opposite) across 58 classification tasks (CLS) and 61 generation tasks (GEN) in the heldout dataset of "SPPERNATINS the argmax values of each prediction match, and 'Changed' denotes the number tokens that have altered in the base prediction when using 0.3
using II) with OpenSNI-7B. "Consistent" and 'Inconsistent' refer whether
Task
Consistent
Inconsistent
Changed
CLS
6392
9665 17807
1071
GEN
61164
5907
KDE of Classification tasks
KDE of Generation tasks
10
8-
Density
2
0.2
0.4
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Maximum Probability
Maximum Probability
Changed
Unchanged
Figure 12: Kernel density estimation of predictions of base instruction from ID 'opposite" with OPenSNI-7B in SUPERNATINST dataset. Maximum Probability refers to the maximum the token distribution derived from base instructions. 'Changed' denotes the tokens that have been altered in the base prediction when applying e = 0.3 and 'Unchanged' represents the tokens that remain unaltered.
Distributions on the Number of Generated Tokens Figure 13 compares the distribution of the number of tokens per response between baseline models and models enhanced with ID using the 'opposite' instruction, across various tasks in the heldout dataset of SUPERNATINST The results highlight that the improvements offered by ID consistent regardless of the response length required by the tasks.
420
(a) Tk-Instruct XL
(b) Ik-Instruct XXL
(c) OpenSNI
Figure 13: Kernel density estimation of response lengths for Tk-Instruct XL (3B), XXL (1IB), and OpenSNI-7B Models over a held-out set of SUTERT
21

---

Published as a conference paper at ICLR 2024
G QUALITATIVE ANALYSIS ON ID RESPONSES
In this section, we present a comprehensive qualitative analysis of the responses generated by both the Baseline and our ID (Opposite). We aim to understand the cases in which our ID fails show performance enhancement in the evaluation.
G.1 WRONG RESPONSE PATTERNS FROM ID
We observe that wrong responses from ID often contain words from the given instructions, while wrong responses from the Baseline typically include words from the instance sentences. In Table 10, we showcase responses in the OE (Overlap Extraction) category from both the 3B & 1IB Baseline and the 3B & 11B ID (Opposite). In this category, as indicated in Table 1, ID does not show improvement over the Baseline. In Table 10, the Baseline typically just repeats the given phrase from the instance, whereas the ID (Opposite), while not completely following the instructions, includes some overlapping words from the provided sentences. Although the Baseline's 's repetitive responses might score well on the Rouge-L evaluation, we suggest that the ID (Opposite)'s s responses, though incorrect, are more ciarrcctionfoouseee
This pattern is also evident in Table 13, where we provide OpenSNI-7B responses on CR (Coreference Resolution) category. Here, both the Baseline and ID models generate wrong responses. However, the Baseline attempts to respond with the words in the instance sentence, while ID uses words from the given instructions. This suggests that while ID generally improves responses, it may reduce performance in tasks that require the use of exact words from the provided instance. This is consistent with our observation in Table 1, where drop in performance is primarily seen in the OE, CR, and KT categories, which often demand such specific word usage.
G.2 How ID REFINES RESPONSE WHEN MODEL SIZE INCREASES
In Table II and Table 12, we showcase responses from two categories: QR (Question Rewriting) and KT (Keyword Tagging). In these categories, the ID initially shows improvement over the Tk-XL Baseline but demonstrates progress in the larger TK-XXL model. Specifically in the QR category, detailed in Table 12, the ID (Opposite) does not successfully enhance the Baseline's 's performance in the XL model. However, it does modify the Baseline response to more closely align with the given instructions, even though the overall performance does not improve. For example, the first example in Table 12. from task035, the Bascline response both in the Tk-XL and Tk-XXL model fails to rewrite the question. In contrast, ID (Opposite) attempts specify the question, even it is incorrectly changed the original intention in Tk-XL model. In the Tk-XXL model, ID (Opposite) successfully adheres to the instructions by specifying the question while maintain the original intention. We observe a similar enhancement pattern in the KT category, shown in Table 11.
G.3 LIMITATIONS OF USING FIXED REFERENCES IN ROUGE-L EVALUATION
In Table 14, we present responses from the WA (Word Analogy) category, comparing the Baseline and ID from the OpenSNI-7B model as shown Table Despite decrease Rouge-L performacce in this category. our qualitative analysis indicates that the ID responses more effectively follow the given instructions, refining the Baseline responses. For instance, in the first example, modifies the response to a Unicode emoji, which is contextually appropriate. In second example, the II) response (' ('spatula') differs from the reference ('knife", ), yet remains contextually valid. An interesting observation is how ID responds when the Baseline output is degeneration response. In third example, where the Baseline simply gives the number '100000 ... ID changes this to '1000 degree oven', better adhering to the instruction. This indicates that using fixed reference response for evaluations can sometimes lead to misleading comparisons between Baseline and ID responses. It underscores the difficulty in accurately assessing open-ended generation tasks.
22

---

Published as a conference paper at ICLR 2024
Table 10: Tk-XL & Tk-XXL Baseline Opposite (OE: Overlap Extraction)
Task
task281 task281 pppinntssffc.orrsspooeeeeee
Instance
You will be given three sentences. Read them, then identify a noun phrase (person, place, or thing) or event that is shared between all three sentences. As the output, write the span of the text corresponding to that phrase in each sentence. Keep the order of the sentences, that is, your answer should look like: 1: 7a phras from sentence le*
Definition
2: $a phras from sentence 2 3: *a phrase from sentence 3
"I: It may help scientists better understand why Yellowstone 's previous eruptions in prehistoric times were some of Earth 's largest explosions in the last few million years 2: Volcanic activity triggers 2,000 to 3,000 small earthquakes per year, and the shake and shock waves travel at different speeds through molten, hot and other rock. Scientist measured the thousands small earthquakes in Yellowstone to sean the earth underneath it
Instance
["I: Yellowstone 's previous eruptions 2: 2,000 to 3,000 small carthquakes per year 3: small earthquakes in Yellowstone") ]
Reference Outputs
Response (XL-Baseline)
The response just repeated the inpul instance.
Response (XXI.-Baseline)
"Voleanic activity triggers 2,000 3,000 small earthquakes per year"
Response (XL-Opposite)
"volcano"
Response (XXL-Oppositee)
"volcan activity"
Response (GPT-4)
"I: Yellowstone in 2: small earthquakes \nn 3: Yellowstone"
Table 11: Tk-XI& & Tk-XXI. Baseline vs Opposite (KT: Keyword Tagging)
Task
Instance ID
In this task, you need to write topic word from the given fact. The topic word must have at least one word overlap with the given fact. The topic word often involves adding new word from a related concept. In your topic word, use at least one word from the given fact. Topic words with two or more words work best.
Definition
Fact: Earth 's magnetic patterns used for finding locations by animals that migrate.
Instance
"annimals migrate.", "magnetic patterns.", "migrate animals north fly.", "migrate bird.", "migrate birds.". "migrating animals.", "migrating ensures species survival."
Reference Outputs
Response (XL-Baseline)
"earth"
Response (XXI-Baseline)
"magnetometry"
Response (XL-Opposite)
"magnetic"
Response (XXI-Oppposite)
"magnetotaxis"
Response (GPT-4)
Maaneticc Navigation in Animals"
23

---

Published as a conference paper at ICLR 2024
Table 12: Tk-XL & Tk-XXL Baseline vs Opposite (QR: Question Rewriting)
Task ID Instance ID
task luxx670.ambgguuuqusiiooonnnaainn
In this task, you re given an ambiguous question (which can be answered in more than one way). Your task to write a question that elarifies the given question such a way that the generated question has one unique answer.
Definition
Instance
What is the brightest star seen from earth?
["What is the brightest star the night sky seen from Earth?",
Reference Outputs
"What is the brightest star, seen anytime from Earth?"]
Response (XL-Baseline)
"What is the brightest star seen from earth?"
Response (XXL-Baseline)
"what star is the brightest as seen from earth?"
Response (XL-Opposite)
"Which constellation is the brightest seen from earth?"
Response (XXI--Opposite)
"what star can be seen by the naked eye as the brightest seen from earth?"
Response (GPT-4)
"Which star appears as the brightest in the night sky when viewed from Earth?"
Task ID Instance ID
task035 winogrande question modification perpaao
You're given a fil-in-the-bbank question where the answer PersonX. You need to minimally change the given question that the answer flips PersonY. This task typically involves replacing one word i.e., the 'trigger word' with its antonym (c.g., changing from "sympathetic" to "stern") You should not change any content in the given question beyond a word or two i.e. the trigger word/phrase. PersonX and PersonY should not be equally likely to fill the blank. For your question, PersonY should be a well-agreed answer to fill in the blank. {Skip)... Although there are many correct answers, you only need to write one them.
Definition
Context Word: pet. \nQuestion: PersonX sneezed more than PersonY did because had kept their pet in the house. \nAnswer: PersonX
Instance
["PersonXX sneezed more than Person'Y did because had kept their pet out of the house."]
Reference Outputs
Response (XL-Baseline)
"Person' sneezed more than PersonX did because had kept their pet in the house."
Response (XXL-Baseline)
Response (XL-Opposite)
"PersonY sneezed less than PersonX did because
Response (XXL-Opposite)
had not kept their pet the house. Answer: Person
"PersonX sneezed less than PersonY did because had not kept their pet in the house. \n Answer: Person
Response (GPT-4)
24

---

Published as a conference paper at ICLR 2024
Table 13: OpenSNI-7B Baseline vs Opposite (CR: Coreference Resolution)
Task
task033 winogrande answer generation
Instance ID
You need to answer a given question containing a blank (-). Your answer must be one of the two objects mentioned the question, for example "trophy" and "suitcase". Your answer must contain word that is not present in the question. Please don't t use articles (e.g., the, before the answer.
Definition
Jenny's doctor told her add more carbohydrates her diet and exercise more, because the would help her process insulin better.
Instance
Reference Outputs
["exercise"]
Response (Baseline)
"diet"
Response (Opposite)
"trophy"
Task ID
task033 winoorane answer www.wwwewweeeeetiio..... 2akk0331172220017772288800011111
Instance ID
You need to answer given question containing a blank (L). Your answer must be one of the two objects mentioned the question, for example "trophy" and "suitcase". Your answer must not contain word that is not present in the question. Please don't 't use articles (e.g., the, before the answer.
Definition
The computer would not be put my old desk because the was heavy be there.
Instance
Reference Outputs
["computer"]
Response (Baseline)
"desk"
Response (Opposite)
"suitcase"
Task ID
Instance ID
task401
In this task, you will use your knowledge about language (and common sense) to determine what element the marked number refers to. The numbers marked with two underlines around them, like: number Your answer should be chosen from the given text, and should not contain other words.
Definition
Ron Woodroof: got one one life right Mine But I want someone else 's sometimes.
Instance
Reference Outputs ["life"]
Response (Baseline)
"1"
Response (Opposite)
" number
Task ID
http:o0onmmrrriieiii fused head reference task401 htt.://00c.ccaa7/////00000001113
Instance ID
In this task, you will use your knowledge about language (and common sense) to determine what element the marked number refers to. The numbers are marked with two underlines around them, like: number Your answer should be chosen from the given text, and should not contain other words.
Definition
The Punisher: still talk to God sometimes I ask him if what I 'm doing is right or wrong I 'm still waiting for an answer and until I get one I '1l waiting, watching, THE GUILTY WILL BE PUNISHED
Instance
Reference Outputs
[]life"]
Response (Baseline) "1"
Response (Opposite) "number"
25

---

Published as a conference paper at ICLR 2024
Table 14: OpenSNI-7B Baseline vs Opposite (WA: Word Analogy)
Task Instance ID
task1153 bard analogical reasoning affordance
Two analogies that signify affordances are given in the form "A B. : Affordance is the possibility of an action being done on an object, for example book is affordance of writing. The phrase "A B" implies that B is affordance of A. Your task is to replace the question mark (?) with the appropriate affordance of the given action C, following the "A B" relation. Your answer should be single object without further explanation.
Definition
Instance
climb: ladder. stir:
Reference Outputs
["soup"]
Response (Baseline)
"spoon"
Response (Opposite)
"Iu2615"
### Corresponds to Unicode emoji
Task ID
waw..aamalloiccll.acaonnnnnnnnn tools
Instance ID
Two analogies that relate actions to the tools used to perform the action given in the form "A B. C : ?", "A B" relates action A to tool B. Your task is replace the question mark (?) with the appropriate tool for the given action C, following the "A B" relation.
Definition
Instance
stir spoon. spread
Reference Outputs
["Knife"]
Response (Baseline)
"spread: : knife"
Response (Opposite)
"spatula"
### Semantically Valid Response
Task ID
Instance ID
task 11611666.6ccceee0504455555000100
Two analogies that relate actions to the tools used to perform the action given in the form "A B. C: ?", "A B" relates action tool B. Your task is replace the question mark (?) with the appropriate tool for the given action C, following the "A B" relation.
Definition
Instance
iron iron. cook
Reference Outputs
["pan", "stove"]
10000000000000000000000000000000
Response (Baseline)
00000000000000000000000000000000 00000000000000000000
Response (Opposite)
"1000 degree oven"
### Revised the Degeneration Response
Task ID
Instance ID
Two analogies that relate objects to the associated rooms is given in the form "A B. C: 2". "A B" relates object A to room B. Your task is to replace the question mark (?) with the appropriate room for the given object C, following the "A B" relation.
Definition
Instance
bookshelf library. table :
Reference Outputs
["kitchen"]
Response (Baseline)
library
Response (Opposite)
dining room
### Semantically Valid Response
26

---

Published as a conference paper at ICLR 2024
Table 15: List of Task IDs in the SUPNATINST used to evaluate Adherence and Coherence of instruction following.
Task IDs
task893 gap fill the blank coreference resolution
task641 eenli classification
musk1529_scinn 15529sscitall1, I. Lelassification
task1393_sun wttp:/3eeegglueecoo wttpeecpt.....opppllll.
taskI344_glue.entaall www.geeeenaalmeet.cleee teaasfinent.lassffittttinn
task1387 anli r3 entailment
task880 schema guided dstc8 classification
task738 perspectrum. classification
task1439 www.ddoaaccooo cooking isanswerable
task242_.weetqa__e wtw.ueeetuucccasffcaaaionn
uusk890.ccwd.classifccct
task1612.sick htt.sscck.abee.llsssiiccaccaa classiffccati
task task1442_doqa_movv moviees wianssebbbl
task233_iir www.iire_iink... beciikexistsscchhe
ciissttclasiiiiationn
task936.defcasible. _ww._ssliiii...
celiisnni_cllssse classificcatinn
www..eeellmewh-gueesttitoee wwweestoon.nswerablle
Lask391_causal_rratttiio relationhiipp
task201 mnli neutral classification
task520 aquamuse answer given in passage
task892 gap reverse coreference resolution
task828 copa commonsense cause effect
task wa____arr__mallgiceee reasoning_trane trash oor treasure
task 11385_anl_rl__ettaal
taskly3l_daily siillyddaplllpppeee satallogtyyeecclssfffiiiic _classification
task1516 ww1166.mmppe_mseme www.raallaaguugggnneeee....
task tisskk.ssikk _b _urelation__
1.44770sherliic.oo caaiiiccccsalllaacccc relationshs
l3sk htl.3/..wwxxeeeccoeefe.eee coreference
task199 1141199_mni.cc/e clasfillasiiico
lask 1 wwnowhy www.eeeaannaaaaae wwwpl/pplllutydllleeeeae detecction
task226 english language answer relevance classification
task935 defeasible atomic classification
task020 metaco span based question
task937 defeasible nli social classification
task 1a511_8cheettallmme
task050_multia http:550mmuiii..nsseaabili ansswrrbbilit
task362 spolin yesand prompt response.suus sub classification
task220 rocstories title classification
task232 lirc link number classification
wwinogrneee easy classsification
generation
laskl533ddaily diiadidoooyydiillgg..ooo wtt../allllaasiiccalllaaalaaaaaa classificattion
laskl624_disllqqa.a.a hww.llaaaaaatooyeeeeeeeee
task827.com task827.copa.com
reasoning
swwema.guideeddde..c guided_ classification
195 cllicclassffication
task200 100000mmmi..ctailmentt www.ennttmmentt.mmmmmmmm claccnttclssiiication
task1534_daily
lallllddlallaeeeeel -uesttuuunnooooo quustiin.classiffcattiiee classsification
task392_inverse _acatcalcl
task640.esnli.com
1aak640.ssii.ccassffcatiioo celassificcatoon
mask633.ohsumed.u
task1640.anm
w...awawaeeee.ee nnw.erable.unanswerabbll
1444349ssquad.00
anwweerbleeeeerblee...... anserrableuunnssseeellee unanswerable
question.classifiino
_question.clasiiiccale classificattion
27

---

Published as a conference paper at ICLR 2024
H EVALUATION ON MMLU BENCHMARK
Overview of MMLU Evaluation Our evaluation the MMLU dataset (Hendrycks et al., 2020), a comprehensive question-answeringg framework encompassing a wide array of subjects across humanities, social sciences, and STEM fields, provides rigorous testbed for our ID method under zero-shot scenarios. The dataset includes a diverse array of questions, sourced from educational materials like GRE, USMLE, and AP exams. Each question is structured with a query and four possible answers, where the task is to identify the single correct choice. We format the tasks as multiple-choice, following Hendrycks al. (2020).
Result Table 16 shows the performance of various models using our ID) technique with epsilon value set to 0.3. We specifically analyze the impact of "Opposite', "Opposite" - "Opposite and 'Null' strategies three distinct models: Tk-Large, Tk-XL, and OpenSNI. The use of these strategies on MMLU dataset are briefly described as follows:
Opposite': : involves using an opposite instruction to replace the original instruction (includes input and answer candidates).
"Opposite combines the opposite instruction with answer options A, B, etc., but excludes the others.
'Oppositet"; involves putting the opposite instruction in front of the given instruction.
'Null": presents only the answer options, removing others.
These strategies are contrasted against the baseline to showcase their effectiveness enhancing EM scores in zero-shot scenario. This consistent performance of the 'Opposite' strategy is particularly noteworthy, as it highlights the strength of our ID approach in effectively leveraging the anchoring effect enhance model response accuracy.
Table 16:
EM scores under a zero-shot scenario across different models on 57 tasks in MMLU dataset (Hendrycks et al., 2020). We set the = 0.3.
Method
Tk-Large
Tk-XL
OpenSNI-7B
Bascline
32.16
43.53
42.22
Opposite
33.79
46.85
43.17
Opposite
32.20
45.13
43.48
Opposite
31.83
43.88
43.25
Null
33.36
45.81
43.69
Null-
33.07
45.16
42.73
28


Reviews:
[{"review_id": "AQh59VGkXL", "reviewer": "Reviewer_K2iN", "summary": "This paper introduces an approach for improving the performance of instruction-tuned LLMs on classification and generation tasks. The key idea is to contrast the token generation probabilities conditioned on the true instruction with probabilities conditioned on noised instructions. The paper performs experiments across four model families (Tk, T0, Alpaca, and OpenSNI) on the UnNatInst and SupNatInst datasets, finding that the proposed approach generally produces improvements across tasks and model families over a standard decoding baseline.", "questions": "Q1) How was the \\epsilon value chosen for the important experiments in the paper (e.g. Table 1 - Table 3)?\n\nQ2) I was confused by the discussion of the smoothing coefficient in 3.3. It says that the typical choice was 0.3, but performance tends to decline with positive \\epsilon values. And the equation for decoding is z - \\epsilon \\tilde z, which makes it seem like \\epsilon should be positive in order to get a contrast.\n\nQ3) Can any intuition be given for why the approach seems to work best in the generalization setting (3.3 \"Generalization Capabiltiies\")?\n\nQ4) What method is used to choose the random words from NLTK (i.e. what word list is used? something involving antonyms?)\n\nOther comments:\n- The discussion in the \"ID with Opposite\" section hints that some of the baseline models might be mode-splitting, and that (at least on these classification tasks), approaches might benefit from a decoding approach that uses min-Bayes risk or consensus decoding: group together words based on their similarities and choose to generate a word from a large group.\n- I wasn't really clear on the support for the conclusions in \"Visualization of Embeddings\", that as models get better at understanding instructions (e.g. are larger), performance of ID increases\n- \"aims to steer the models toward particular\" in Section 1. Should \"toward\" be \"away from\"?\n- \"pragmatics\" in the motivation for Label Adherence and Label Coherence is a bit misleading; this is much closer to \"semantics\".\n- it would help to make some of the table/figure captions more self-contained, e.g. defining * in Table 3 caption.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "S1) The approach is simple but well-motivated: it doesn't require access to additional models (as Contrastive Decoding does) or access to model internals beyond token probabilities. \n\nS2) The experimentation was comprehensive, and I appreciated the analysis and ablation experiments, especially the \"degradation to enhancement\" part, the generalization experiments, and the analysis of sensitivity to the \\epsilon parameter.\n\nS3) The results were convincing to me: the approach produces consistent improvements, with substantial benefits in cross-dataset generalization.\n\nS4) The paper was overall very-well written.", "weaknesses": "W1) While I appreciated the analysis of sensitivity to the \\epsilon parameter, given that there is only a relatively small region of \\epsilon where the approach outperforms the baseline, I'd appreciate more discussion of how \\epsilon was chosen and whether it was the same across all tasks in each experiment.\n\nW2) There's a bit of room to improve the analysis and discussion of it, see questions and comments below. \n\nW3) This is a minor weakness, but I didn't really understand the connection to \"anchoring\". Anchoring seems to be about sensitivity to the first thing appearing in the text, while this method doesn't privilege any particular positions in the text but contrasts against the negative prompt at all positions.\n\nW4) While the related work was generally good, there is room to mention a few other related papers (below) on contrastive methods for generation. But I also don't think this is a major weakness, as the application to instruction following is new.\n\n- Li et al. A Diversity-Prompting Objective Function for Neural Conversational Models. NAACL 2016\n- Shen et al. Pragmatically Informative Text Generation. NAACL 2019\n- Ho and Salimans. Classifier-Free Diffusion Guidance. 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yCA38jIHsp", "reviewer": "Reviewer_yaLP", "summary": "This paper proposes a simple yet effective method named instructive decoding (ID) to boost the performance of instruction-tuned models. First, an noisy instruction is crafted to elicit undesired response from instruction-tuned models; then ID refines the next-word logits based on the logits from the noisy instruction. The method is training-free and demonstrates improvement in many instruction-following tasks.", "questions": "- is it possible to further fine-tune the model via the new logits (i.e.,  $z_t - \\tilde z_t$ in Algorithm1)\n- I wonder if the improvement from ID correlates with the response length required for a task. E.g., is ID more useful in tasks that require long response generation?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- the method is simple and effective.\n- extensive experiments and analysis", "weaknesses": "-it’s still unclear (for a practitioner or researcher) to what extent the noisy instruction would be helpful for a new task. Maybe some qualitative analysis in those no-improvement tasks would provide some intuition on why the noisy instruction is harmful in the proposed instructive decoding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kTsnlypPzm", "reviewer": "Reviewer_sigH", "summary": "The paper introduces a method called Instructive Decoding (ID) designed to enhance the performance of instruction-tuned LLMs when they encounter unfamiliar tasks due to the lack of diverse annotated prompts during training. Unlike other methods that rely on expanding the set of instructions, ID manipulates the model by adding 'noisy instructions' that alter the logits of the next-token prediction contrastively. These 'noisy instructions' are essentially counter-instructions that generate contrasting logits, which are then applied to the original logits to refine the prediction distribution.\n\nEvaluations show that models augmented with ID have an edge over basic models in zero-shot generation and better adherence to instructions for certain tasks. The main metric used for evaluation is the zero-shot Rouge-L score on unencountered tasks.", "questions": "Can the authors shed light on the underlying rationale behind the proposed method, especially concerning the prediction of logits? How can it be ensured that the noisy responses will always lead to contrasting predictions at every word level?\n\nHow is the \"anchoring effect\" relevant to the proposed method? The paper seems to lack a detailed explanation connecting the two.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The Instructive Decoding (ID) approach is interesting. It offers an alternative to enhancing LLMs using a broader set of instructions and doesn't necessitate further fine-tuning.\n\n2) The methodology is straightforward and the evaluation metrics, especially those in the table, offer a comprehensive view of the results.", "weaknesses": "The results, although promising, are mixed. From table 1, only about half of the tasks demonstrated improvements, while the rest experienced a decline in performance.\n\nThe paper's rationale needs clarity, especially from a machine learning perspective. It seems there's an assumption that noise-based responses will always produce contrasting predictions at every word level. For instance, if the base prediction is 'xxxx', and the noisy prediction is 'xxxx is wrong', the 'xxxx' part remains consistent, which can be misleading.\n\nThe paper references the \"anchoring effect\" but doesn't provide a clear link between it and the proposed method. The model doesn't directly witness the output from the noisy instruction. Instead, this output is indirectly factored into the next-word prediction logits computation. This differs from the anchoring effect, where individuals receive specific information before arriving at a final judgment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yOgPrwCoS4", "reviewer": "Reviewer_wghP", "summary": "The paper proposes a new decoding method, instructive decoding, which generates tokens with logit scores constrasted with the logits under perturbed instructions. Extensive experiments show that such a technique is effective across a wide range of tasks,", "questions": "- Would there a better way to present the results in Table 1? Currently it feels incremental since the improvement is only 1-2 points, but I could see other arguments against this (e.g., increasing from 3b to 11B also only does not improve much). Would there be a better way to help the readers better contextualize the improvement (or find datasets that are more discriminative between weaker and stronger systems)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The presented idea is intuitive and interesting.\n- Extensive experiments and ablations show that the idea works.\n- The paper is generally well-written.", "weaknesses": "- The empirical gain in Table 1 seems incremental (e.g. only 1-2 points of overall improvement) \n- The idea is not particularly groundbreaking", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)