NEFTune: Noisy Embeddings Improve Instruction Finetuning
Authors: Neel Jain
Ping-yeh Chiang
Yuxin Wen
John Kirchenbauer
Hong-Min Chu
Gowthami Somepalli
Brian R. Bartoldson
Bhavya Kailkhura
Avi Schwarzschild
Aniruddha Saha
Micah Goldblum
Jonas Geiping
Tom Goldstein
Venue: ICLR 2024 poster (2024)
Keywords: Instruction Finetuning
Primary area: generative models

Abstract: We show that language model finetuning can be improved, sometimes dramatically, with a simple augmentation. 
NEFTune adds noise to the embedding vectors during training.
Standard finetuning of LLaMA-2-7B using Alpaca achieves $29.79$\% on AlpacaEval, which rises to $64.69$\% using noisy embeddings. NEFTune also improves over strong baselines on modern instruction datasets.
Models trained with Evol-Instruct see a $10$\% improvement, with ShareGPT an $8$\% improvement, and with OpenPlatypus an $8$\% improvement. 
Even powerful models further refined with RLHF such as LLaMA-2-Chat benefit from additional training with NEFTune. Particularly, we see these improvements on the conversational abilities of the instruction model and not on traditional tasks like those on the OpenLLM Leaderboard, where performance is the same.

Published as a conference paper at ICLR 2024
NEFTUNE: NOISY EMBEDDINGS IMPROVE INSTRUC- TION FINETUNING
Neel Jain' Ping-yeh Chiang"', Yuxin , Kirchenbauer', -oong-Min Chu', Gowthami Somepalli' , Brian R. Bartoldson', Bhavya Kailkhura", Avi Schwarzschild', Aniruddha Saha', Micah Goldblum', Jonas Geiping', Tom Goldstein' 1 University of Maryland, 2 Lawrence Livermore National Laboratory, New York University
ABSTRACT
We show that language model finetuning can be improved, sometimes dramat- ically, with a simple augmentation. NEFTune adds noise to the embedding vectors during training. Standard finetuning of LLaMA-2-7B using Alpaca achieves 29. 79.%%% on AlpacaEval, which rises to 64.69% using noisy embeddings. NEFTune also improves over strong baselines on modern instruction datasets. Models trained with Evol-Instruct see a 10% improvement, with ShareGPT an 8% improvement, and with OpenPlatypus an 8% improvement. Even powerful models further refined with RLHF such as LLaMA-2-Chir benefit from addi- tional training with NEFTune. Particularly, we see these improvements on the conversational abilities of the instruction model and not on traditional tasks like those on the OpenI.LM Leaderboard, where performance the same.
INTRODUCTION
Generative language models are typically trained on raw web data, and then subsequently fine-tuned on comparatively small but carefully curated set of instruction data. Instruction fine-tuning is crucial to taming the power of LLMs, and the usefulness a model is largely determined by our ability get the most out of small instruction datasets.
In this paper, we propose to add random noise to the embedding vectors of the training data during the forward pass of fine-tuning. We show that this simple trick improve the outcome of instruction fine-tuning, often by a large margin, with no additional compute or data overhead. Noisy Embedding Instruction Fine Tuning (NEF Tune), while simple, has a strong impact on downstream conversational quality. When a raw LLM like LLaMA-2-1 2-B-- is finetuned with noisy embeddings, its performance on AlpacaEval improves from 29.8% 64.7% (Figure of around 35 percentage points (Touvron et al., 2023b; Dubois et al., 2023). NEFTune leads to this surprising and large jump in performance on conversational tasks, maintaining performance on factual question answering baselines.
an impressive boost
Alpaca
LLaMAA2(7BB)
+NEFT
Evst-nnstract
70.1
79. 79.66.19.3)
SharGPT
(-7.5)
OpenPlatypus
k2..0
No.(48.))
$7.7
Average
72.6+15.1)
20
80
100
AlpacaEval Win Rate
Figure 1: Alpaca Win Rate measurements for LLaMA-2-7B models finetuned on various datasets with and without NEFTune. NEFTune leads performance boosts across all of these datasets, showcasing the increased conversational quality of the generated answers.
Equal contribution. Code is available on Github: httpp:///githhb..co//eeett/// hww. NEFTune. Correspondence to Neel Jain: <njain 17 @umddedu>>.

---

Published as a conference paper at ICLR 2024
1.1 RELATED WORK
The earliest forms of instruction finetuning such as FLAN and TO (Sanh et al., 2021; Wei et al., 2021) focused on cross-task generalization in language models. Encoder-decoder language models were finetuned on broad range of NLP tasks (about 100) and then evaluated
set of different tasks. This was later scaled up to include thousands of tasks, seeing further improvement over the original FLAN (Chung et al., 2022; Xu al., 2022). Although these works showed that LLMs could be easily adapted to solve simple and classical NLP tasks, real-word scenarios require LLMs to provide free-form answers to open-ended queries.
InstructGPT (Ouyang et al., 2022) was the first model to tackle open-ended queries with impressive performance. OpenAl further trained GPT-3 (Brown et al., 2020) using reinforcement learning from human feedback (RLHF) to align the model. This procedure gave rise to highly popular models like ChatGPT (OpenAI, 2022) that captured the imagination of the general public and generated longer coherent text than its InstructGPT predecessor.
This led to the work of Wang et al. (2022) (Self-Instruct), which used InstructGPT (Text-Davinci- 003) to produce instruction-output pairs which could be used to finetune the foundation models like LLaMA into instruction following variants like Alpaca (Taori et al., 2023). Through the rise in popularity of distilled models Taori et al. (2023), the community has constructed other datasets distilling in particular ways from other models like ChatGPT, including Xu et al. (2023). In another approach, ShareGPT (Chiang et al., 2023) was constructed by crowd sourcing real user conversations from ChatGPT. Other datasets like Lee et al. (2023) construct a dataset to improve specific aspects of the model like STEM question answering and logical reasoning. AlpaGasus (Chen et al., 2023) filters data by quality (according to GPT-4) to improve performance.
should be noted that noisy inputs have been used to improve models in various ways. The first in- stance of noise being used to improve language models was the FreeLB method by Zhu et al. (2019), who observed that adversarial perturbations boosted the performance of MLM models. The noise in this case is not random, but is rather computed by first adding a small Gaussian perturbation to the embeddings and then using a gradient step to find the perturbation that maximally alters model performance. This adversarial augmentation approach also improves model performance graphs Kong et al. (2022). While our proposed scheme is non-adversarial, we adopt the noise scaling rules from these works. Training noisy inputs has also been done for other applications, such to improve image captioning systems (Nukrai et al., 2022; Roth et al., 2023), natural language under- standing tasks (Wu et al., 2022; Yuan et al., 2023; Roth al., 2023), and common component of early differential privacy mechanisms (Dwork et al., 2014).
NEFTUNE: NOISY EMBEDDING INSTRUCTION FINETUNING
Instruction models are trained on datasets comprising pairs of instructions and responses. Each step of NEFTune begins by sampling an instruction from the dataset, and converting its tokens to embedding vectors. NEFTune then departs from standard training by adding a random noise vector to the embeddings. The noise is generated by sampling iid uniform entries, each in the range [-1, 1]. and then scaling the entire noise vector by factor of a/ Ld, where L is the sequence length, is the embedding dimension, and is tunable parameter. This scaling rule was borrowed from the adversarial ML literature (Zhu et al., 2019; Kong et al., 2022), and results in a random vector with an expected Euclidean magnitude of approximately a/ /3. Algorithm describes our method in detail.
3 EXPERIMENTAL SET-UP
3.1 MODELS
We conduct the majority of our experiments using 7B parameter LLMs. Particularly, we use LLaMA-I, LLaMA-2, and OPT-6.7B (Touvron et al., 2023a;b; Zhang et al., 2022). These similarly shaped transformers mostly differ in tokens seen during training. OPT, LLaMA-1, and LLaMA-2 were trained using 180B, 1T. and 2T tokens respectively. This difference is to be reflected in model performance standard benchmarks like MMLU, with LLaMA-2 performing the best and OPT performing the worst. For the 13B parameter models, we train LLaMA-2. Additionally, we improve RLHF models by finetuning the highly refined LLaMA-2-Chat (7B) model.
2

---

Published as a conference paper at ICLR 2024
Algorithm NEFTune: Noisy Embedding Instruction Finetuning
Input: D = { Ji tokenized dataset, embedding layer emb(-), rest of model f/emb (-), model parameters 0, loss(-), optimizer opt(.)
NEFT Hyperparameter: base noise scale R
Initialize from a pretrained model.
repeat (X,,Y, Y,) D
sample minibatch of data and labels b batch size B, seq. length L, embedding dimension D sample noise vector add scaled noise to embeds
Xemb f emb( (X;), R BxLxd
e~ Uniform( -1, 111R)RBB.... BxLxd
X & Xemb + VLd
Y, + f/emmb(Xxx mmb
> make prediction at noised embeddings D train step, e.g., grad descent
t- opt(O, loss(Y,, Y:))
until Stopping criteria met/max iterations.
"If sequence lengths in batch equivalent, then L is vector E 2 and the scaling factor (a/ VLd) computed independently for each sequence in batch.
3.2 INSTRUCTION FINETUNING DATASETS
We focus on the following finetuning datasets either because of their wide popularity, or because they have yielded state-of-the-ee art results in the recent past. Note that only single-turn datasets because of the memory constraints of our hardware setup.
Alpaca (Taori et al., 2023) was constructed using the Self- Instructt method of Wang et al. (2022), and the next--avinci-003 model (Ouyang et al., 2022). Self-Instruct uses a small seed set of tasks to construct new instruction tuning tasks and filter out bad ones.
Evol-Instruct (Xu et al., 2023) contains 70k single-turn instructions that are considered more complex than Alpaca. This dataset was derived from the Alpaca dataset by using ChatGPT to evolve the initial instructions.
Open-Platypus (Lee et al., 2023) is curated dataset amalgamated from 11 open-source datasets, curated specifically towards improving LLM performance in STEM and logical domains. This set contains 25k questions where 10% are -enerreeedt and the remainder human-writter ChatGPT conversations (ShareGPT, 2023). Although ShareGPT is multiturn, we use the dataset version from Vicuna- v1.1 and split the multi-turn conversations closer to single-turn format.
ShareGPT (Chiang et al., 2023) is dataset of 70K voluntarily
Additionally, we finetune all models with the Alpaca system prompt, except for ShareGPT, where we use the Vicuna system prompt. The hyperparameters can be found in Appendix A.1. We set our hyperparameters through a coarse sweep LLaMA-1 (7B) trained on the Alpaca dataset, where see 6% improvement over the standard Alpaca model. We use these as the defaults on all models.
3.3 EVALUATION
Since we train using largely single-turn data, we evaluate the model's conversational abilities using AlpacaEval. We also evaluate on the tasks from the OpenLLM Leaderboard determine if the NEFTune augmentation causes any loss in performance on standard multiple choice tasks.
AlpacaEval. The AlpacaEval dataset released by Dubois et al. (2023) is used evaluate the overall quality of generations. AlpacaEval is an automatic model-based evaluation that compares Text-Davinci-003 generations to the model generations over 805 instructions with the Win Rate reported. The Win Rate is the rate at which the model in question is preferred to Text-Davinci- 003 as determined by model evaluator (GPT-4). The 805 test prompts are scraped from vicuna, koala, Anthropic's 's hh-rlhf, and other sources, making it fairly comprehensive and diverse test. Additionally, AlpacaEval has high agreement with humans (Dubois et al., 2023) (validated on 20K annotations). We believe at the 7B and 13B scale this evaluation is still quite reasonable. We use both GPT-4 and ChatGPT as evaluators. We use ChatGPT as a precursor test to determine which models to evaluate GPT-4, This is due to the cost and API restrictions of GPT-4.
3

---

Published as a conference paper at ICLR 2024
Table 1: AlpacaEval Win Rate versus Text-Davinci-003 for ILAMA-2 trained different datasets, using GPT-4 as the evaluator, showing an average improvement of 15% across all datasets over three seeds.
Alpaca
Evol-Instruct
ShareGPT
OpenPlatypus
Average
Llama-2 7B +NEFTune
+0.91+0.40 +0.40
70. +0.00
70.37 76.22 20.67
60.95 69.96
57.91 73.06
65.70 10.52
80.36
A AlpacaEval Win Rate
6*
Win Rate (ChatGPT Eval.
60-
.23 515
50
40
30-
OPT-6.7B
LLaMA-1-7B
LLaMA-2-7B
+NEFT
+NEFT
NEEF
Alpaca
Evol-Instruct
OpenPlatypus
ShareGPT
AlpacaEval Win Rate with and without NEF e on LLaMA-2, LLaMA-I, and OPT across Alpaca, Evol-Instruct, ShareGPT and OpenPlatypus datasets. Performance improves across different datasets and models with ChatGPT as the evaluator.
Figure 2:
Hugging Face OpenLLM Leaderboard. The evaluation datasets used for leaderboard ranking are the verbalized multiclass classification datasets ARC (Clark et al., 2018), HellaSwag (Zellers al., 2019), MMLU (Hendrycks et al., 2020), and TruthfulQA (Lin et al., 2022). This combination of datasets broadly evaluates the ability of LLM to respond to factual questions and reasoning challenges, and we evaluate on these datasets confirm that model capabilities are not negatively impacted by NEFTune.
RESULTS
NEFTune Improves Text Quality. From Table 1, we can see an increase across all datasets for the 7B scale with an average increase 15.1%, showing that training with NEFT significantly improves conversational ability and answer quality, as measured via Alval.Eval. Additionally, we can see from Figure that we also see improvements on older models, such as LLaMA-1 and OPT. Interestingly, we see less improvement on ShareGPT than on other datasets according ChatGPT. However, this is not reflected in the GPT-4 evaluation. From Table 2, we see the Win Rate climbs from 75.03 75.03% to 88.81% % 13)788) after adding NEFTune to the 70B parameter model trained on Evol-Instruct (hyperparameters in Appendix A.1).
NEFTune Can Improve Chat Models. From Table 2, we see that further instruction finetuning LLaMA-2 Chat (7B) on Evol-Instruct can boost the performance of LLaMA-2-Chal by 3%. This model was already extensively tuned, using multiple rounds of RLHF. Yet, with NEFTune, see a sizable, additional performance increase of 10%, although we note that some capabilities of this checkpoint model may be affected like its ability refrain from outputting toxic behavior. Nevertheless, it is surprising that the conversation quality such refined chat model can be so dramatically improved.
4

---

Published as a conference paper at ICLR 2024
100
100
Accuracy
58.4
35.4
60-
47.9
47.
60
43.9
45.3
40
40
20
2 7B (Alpaca)
Lluma (OgeaPlatyuu)
+NEFT
1NE2T
0
HelbSwang
MMLU
HellaSwag
MMLU
TruthfuQQA
100
100
17.9
78.0
Accuracy
60
53.7
31.
49.0
8913
48
46
A5
18.3
28
40-
20
LAuns22 78 (Evol-lastruc))
Llame- 78 (Evok-nciisc))
0
HellaSwag
TruthfuQQA
HellSSaag
MMLU
TruthuuQA
Figure 3: OpenLLM Leaderboard tasks with and without NEFTune on LLaMA-2 across Alpaca, Evol-Instruct, and OpenPlatypus datasets and LLaMA-1 trained on Evol-Instruct. We observe that performance does not change across datasets and models.
Effect on Capabilities. A potential concern is that NEFTune improves conversational ability only at the cost other classical skills. We evaluate on the OpenLLM Leaderboard tasks, using the LM- Eval Harness (Gao et al., 2021) implementation of MMLU, ARC, HellaSwag, waag, and TruthfulQA. These benchmarks give us glimpse into model knowledge, reasoning, and truthfulness. Figure shows that scores remain stable and that NEFTune preserves model capabilities.
Table 2: LLaMA-2-Chat (7B), LLaMA-2 (13B), and LLaMA-2 (70B) can be finetuned further to improve performance.
LLaMA-2 (7B)
LLaMA-2-Chat (7B)
LLaMA-2 (13B)
LLaMA-2 (70B)
Base
71.37*
Evol-Instruct 70.34
74.44
72.61
75.03
+NEFT
79.60
81.74
82.04
88.81
NEFTune Works with QLORA. We show that NEFTune also improves performance in con- strained resource environments by training with Quantized Low Rank Adapters (QLORA) (Dettmers al., 2023). We use the implementation from Dettmers et al. (2023), and the default training hy- perparameters for all model weights, training for only one epoch. Table shows that when training with QLORA, Al paccavall performance increases across all model sizes and datasets studied. However, performance gains are less stark than those seen in full scale finetuning. This may be because different hyperparameters (i.e, number finetuning epochs) are needed, or because we are heavily quantizing to 4-bits.
A Qualitative Example. Here we show qualitative example from LLaMA-2 trained on Alpaca with and without NEFT. We choose the quantum computing prompt from the Vicuna Eval subset of the AlpacaEval instructions. We from this example that Alpaca produces response that shorter, giving only a very basic definition of quantum computing, mentioning qubits, superposi- tion, entanglement, and increased complex computation. In the Alpaca-NEFT response, the model
5

---

Published as a conference paper at ICLR 2024
Table 3: A AlpacaEval Win Rate (ChatGPT Eval.) reported across different datasets and model
sizes, Even training with QLORA, we can see performance increases across the board, although they are milder than regular finetuning.
Model
LLaMA2(7B)
LLaMA2(7B) Evolve70k
LLaMA2(13B)
LLaMA2(13B) Evolve70k
LLaMA1(30R)
Dataset
Alpaca
Alpaca
Alpaca
Base
26.71
52.80
31.99
56.96
31.99
+NEFT 5
31.43
56,96
38.20
60.99
35,16
+NEFT 10
28.57
54.84
41.55
61.30
36.89
+NEFT 15
29.01
57.27
42.17
57.39
35.03
provides a more fluid answer, expounding on these topics with clearer explanations of superposition and quantum entanglement, and including a mention of potential applications. We feel this example is representative of the kinds of changes induced by NEFT. See the appendix for more examples using different finetuning datasets.
AlpacaEval (Vicuna Subset) Prompt: Can you explain the basics of quantum comput- ing?
Alpaca-2-7B Response:
Quantum computing is based on the principles of quantum mechanics, which allow for phenomena such as superposition and entanglement between particles. This means that information can be processed in parallel instead of sequentially like traditional computers. Quantum bits (qubits) are used to store data instead of classical bits, allowing more complex computations to take place with fewer resources than would otherwise be necessary.
Alpaca-2-7B 2-r-a-2-7--NFT Response:
Quantum computing is based the principles of quantum mechanics, which describe how particles such as electrons and photons behave at the subatomic level. Unlike traditional computers, which process information in binary digits (bits) that can be either 0 1, quantum bits (qubíts) can exist in multiple states between 0 and simultaneously. This phenomenon is known superposition and allows for much to be performed than would otherwise be possible with classical computers. Quantum algorithms are designed to take advantage of this property by encoding data into qubit states so that it can be processed exponentially faster than conventional methods. Additionally, quantum entanglement-the ability for two particles to remain connected even over great aitances-allowe for large amounts of data to be sent across networks quickly and securely. The potential applications of quantum computing range from cryptography and artificial in- telligence drug discovery and material science. With continued research and development, quantum technology has the potential to revolutionize many industries.
complex calculations
5 ANALYSIS
We hypothesize that by adding noise to the embeddings at train time, the model overfits less to the specifics of the instruction-tuning dataset, such formatting details, exact wording, and length. Instead of collapsing to the exact instruction distribution, the model is more capable of providing answers that incorporate knowledge and behaviors of the pretrained base model.
A very noticeable side-effect of this, that we observe immediately, is that the model is forming more coherent, longer completions. Longer, more verbose, completions are preferred by both human and machine evaluators most datasets (Dubois et al., 2023). but we find that the increased verbosity is only the most visible side-effect from the reduced overfitting to the instruction distribution; increased verbosity alone cannot explain the measured gains in performance.
5.1 OVERFITTING
In this analysis, we focus on LLaMA-2-7B models trained on the Alpaca dataset both with and without NEFTune. We examine the training loss of both models the Alpaca dataset (both

---

Published as a conference paper at ICLR 2024
LLaMA-2-7b
+NEFT
LLaMA-2-7b
+NEFT
3000
4000
3000
2000
Count
Count
2000
1000
1000
0.25
0,50
0.75 Loss (Alpaca)
1.00
125
0.6
0.8
1.0
1.2
Loss -Evol-Instruct)
Figure 4: Left: training loss on the Alpaca dataset for models with and without NEFT, computed with no added noise. Training with NEFT yields a higher training loss. Right: loss of the same model, but evaluated on the "test" Evol-Instruct dataset. NEFT yields slightly lower loss.
evaluated without noise) and the "testing" loss on the Evol-Instruct dataset. See Figure 4, which shows that the TEFTune model has significantly higher training loss but slightly lower testing loss compared to the base model trained without NEF Tune. This indicated less overfitting and better generalization when NEFTune is used.
To test our overfitting hypothesis further, we also generate responses to training prompts with these models using greedy decoding. We compare the generated responses with the ground truth responses provided in the dataset and report the results in Figure 5. We use ROUGE-L (Lin, 2004) and BLEU (up to n-gram order 4) (Papineni et al., 2002) to measure the similarity between responses. Fig- 5 shows that responses generated by the model trained with NEFTune have significantly lower ROUGE-L and BLEU scores. As ROUGE-L is based on longest common subsequence of words and BLEU is based on common n-grams between responses. higher scores on responses generated by the model trained without NEFT indicate that its responses contain a significantly larger portion of the same words in the same order from the ground truth response, as compared the outputs of the model trained without NEFTune.
Taken together, these observations imply that standard finetuning recipes, while tuned for maximal performance, significantly overfit to the instruction dataset, inducing exact reproduction of some responses. contrast, NEFTune models overfit less without reduction performance on the test set, and do not "lock-in" to the exact wording of the instruction data, as seen in the ROUGE-L metric.
5.2 LENGTH VERSUS TOKEN DIVERSITY
Due to the strong correlation between increased length and performance on the AlpacaEval task (in our experiments and for submissions to the public leaderboard), we wondered whether the in- crease in length observed with NEFTune might come at cost to the diversity of the text. To investigate this, compute the n-gram repetition rates for LLaMA-2 trained on different finetun- ing datasets with and without NEFT N-grams reoceur more frequently in longer passages, and so must control for passage length. We compute repetition and diversity scores chunk at the beginning of each sample. The fixed length cuttoffs were 50 for models trained on Alpaca, 100 for Evol-Instruct, 150 for ShareGPT, and 150 for OpenPlatypus. We choose the chunk lengths so that at least half of the generations were longer than the cuttoff, and sequences of insuf- ficient length were dropped. The diversity scores we compute are summary measure of 2-, 3-, and 4- 4-gram repetition rates called log- diversiy,,,, as described in Kirchenbauer et al. (2023); Li et al. (2022).
fixed-length
In Table 4 and Table 6, we see that NEFT models generate longer outputs than their counterparts. However, we also see that the 2-gram repetition rates as well as overall token lig-diversity for models
Note that for all models we performed generation with repetition penalty of 1.2, held constant across all experiments.
7

---

Published as a conference paper at ICLR 2024
LLaMA-2-7b
+NEFT
LLaMA-2-7b
+NEFT
100
100
Count
Count
40-
40
20
20
0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1,0
ROUGE-L
BLEU
Figure 5: Left shows the ROUGE-L of training with and without NEFT. Right shows BLEU score.
trained with and without NEFT are nearly identical, providing evidence that the longer responses do not come at the expense of repetition, and instead provide additional details.
5.3 LENGTH IS (NOT) ALL YOU NEED
To scrutinize the length-leaderboard correlation even further, we tested whether simply promoting model generate longer outputs was sufficient recover the performance gains of models trained with NEFT. See Table 5. First, we try explicitly prompting the model to give longer answers. Inter- estingly, this boosts AlpaceEval scores by 16%. We can also coerce long completions by blocking the [EOS] token until we hit 250 tokens in length, thus forcing standard model to produce answers as long as NEFT. This results in marginal improvements over standard finetuning.
Finally, we ablate the use of uniform versus Gaussian noise in the NEFT algorithm and find that Gaussian noise induces even longer outputs, but does not come with improved performance. See Table 6. While longer generations do score better, we see that no prompting strategy that we tested came close to the performance of NEFTune models.
Table 4: (Row 1) Avg. Character lengths of A AlpacaEval responses from LLaMA-2 models fine- tuned different datasets. We also report average output length for each dataset (though we trained with max sequence length of 512). NEFT increases average length. (Row 2) Whitespace-tokent
lengths of generations. (Row 3) 2-Gram repetition rates. (Row Log-Diversityy measures.
Alpaca
Evol-Instruct
ShareGPT
OpenPlatypus
(a 5)
(a 5)
(a 10)
(a 15)
Training data
270.31
1356.43
1276.76
649.39
Character Lengths
LLaMA-2 7B
375.22
864.06
1011.28
1100.98
+NEFT
1061.89
1403.59
1496.86
1694.26
Whitespace Lengths
LLaMA-2 7B +NEFT
60.5
138.99
161.04
170.41
169.36
225.56
234.99
264.12
2-Gram Repetition %
ILaMA-2 7B
1.49
3.87
4.82
2.73
+NEFT
1.72
3.79
4.58
3.21
LLaMA-2 7B
15.97
10.65
8.40
9.96
Div-Diverytty
+NEFT
16.41
10.77
8.60
9.64
5.4 HUMAN STUDY
Since our primary results are based the AlpacaEval benchmark, which is scored by large language model, we also run a small scale human study amongst the authors of this work. For a subsample of 140 instructions from AlpacaEval, present annotators with one response gener- ated by a LLaMA-2 model finetuned on Alpaca data with NEFT and another response from a model trained without NEFT, in random order.
8

---

Published as a conference paper at ICLR 2024
Table 5: We the following meta-prompts to get longer responses: "Generate long response",
"Generate a comprehensive response", and "Generate a long and comprehensive response." Longer responses score better, but do not close the gap with NEFT.
Setting (LLaMA-1)
GPT-4 Win Rate
Avg. Character Length
Alpaca-7B-NEFT
61.99
1058.46
Alpaca-7B (Long + Comp)
48.01
620.74
Alpaca-7B (Long)
44.84
614.21
Alpaca-7B (Comprehesive)
42.14
494.85
Alpaca-7B (Min New Tokens)
38.58
1110.97
Alpaca-7B
32.36
375.22
Win Rate (and Avg. Character Length) AlpacaEval evaluated by ChatGPT for different levels and types of training noise. While length does increase with noise, not always indicative of AlpacaEval Win Rate.
Table 6:
Setting
Alpaca
Evol-Instruct
OpenPlatypus
LLaMA-2-7b
48.26
(375.22)
62.55
(864.06)
57.20
(1100.98)
+Uniform Noise 5
62.55
(1061.89)
67.58
(1403.59)
60.99
(1428.31)
+Uniform Noise 10
61.18
(1009.94)
65.59
(1696.88)
60.62
(1833.85)
+Uniform Noise 15
61.86
(819.61)
66.58
(1650.65)
61.74
(1694.26)
+Gasssian Noise 5
60.93
(1371.32)
65.09
(2065.75)
59.13
(2060.92)
Human annotators preferred NEFT in 88 instances, and 22 instances were draw. This corresponds 74.6% win score for NEFT using the AlpacaEval formula (88/ (140 22)). Next, we per- formed modified run of A AlpacaEval where, instead of asking the evaluator (GPT-4) to choose between the outputs of our model or Text-Davinei-003: we present pair of responses from the standard finetuned model and a NEFT version of the same model. There, observe a win score of 92.80%.
CONCLUSIONS AND LIMITATIONS
The success of NEFTune points to the often ignored importance of algorithms and regularizers for LLM training. Unlike the computer vision community, which has studied regularization and overfitting for years, the LLM community tends to use standardized training loops that are designed for optimizer stability and not generalization. In this environment, LLM researchers have become fixated on datasets and model scaling as the primary path forward. Given the consistent gains of NEFTune, techiniques such as regularization and robust optimization may deserve to be revisted in the LLM setting.
Our study has several limitations. We adopt AlpacaEval our central measure of instruction- following ability for LLMs, which is subject to the biases of single judge (GPT-4). Additionally, due to limited compute resources, we were not able to validate the success of NEFTune on larger 30B and 70B variants across different datasets of LLaMA-2, and we had to rely on fixed hyper- parameters for most NEFTure Tne rums rather than sweeping. Finally, despite our empirical studies, do not have a conclusive understanding of why NEFTune works.

---

Published as a conference paper at ICLR 2024
7 ETHICS STATEMENT
In this work, we proposed an augmentation for instruction finetuning. Although we evaluate these models on standard benchmarks, we do not rigiously evaluate the impact of NEFTune on model safety and reliability characteristics like toxicity or refusal to provide potentially harmful responses.
REPRODUCIBILITY STATEMENT
We describe the models (in Section 3.1) and datasets (in Section 3.2) used in our experiments including all hyperparameters (in Section A.1). The compute infrastructure used was based on commodity-level CPUs and GPUs running open source software. Additionally, with submission of the review copy of the work we have included zip archive containing anonymized copy of the source code developed through the course the research. We have added a github link to the codebase well. The codebase is designed to be both useable and extensible for further research and upon publication, a link to a public copy of the source code will be added to the work.
ACKNOWLEDGMENTS
This work was made possible by the ONR MURI program, the Office of Naval Research (N0001421 12557), and the AFOSR MURI program. Commercial support was provided by Cap- ital One Bank, the Amazon Research Award program, and Open Philanthropy. Further support was provided by the National Science Foundation 2112222,,,,). and by the NSF TRAILS Institute (2229885).
Furthermore, this work was performed under the auspices of the U.S. Department of Energy by the Lawrence Livermore National Laboratory under Contract No. DE-AAC5-007NA23344 Brian Bartoldson's 's and Bhavya Kailkhura's efforts were supported by the LLNL-LDRD Program under Project No. 24-ERD-010 BLLN--CONF8855499 We are also grateful to Amar Saini who provided HPC support.
REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, al. Alpagasus: Training a better alpaca with fewer data. arXiv preprint wwXXiv2307.08700 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https: ht/lmmsssccr/ wwt./lggg/oooooooo 22021lggg2222222 03 30 - vvicuaa..
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi language mod-
Wang, Mostafa Dehghani, Siddhartha Brahma, al. Scaling els. arXiv preprint arXiv:2210.1 /416, 2022.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint vrXiv:1803.05455 2018.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314 2023.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsumori Hashimoto. Alpacafarm: simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305. 143888777. 2023.
10

---

Published as a conference paper at ICLR 2024
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends Theoretical Computer Science, (33-4):211400)) 2014.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot lan- guage model evaluation, September 2021. URL https: htt.dii.orgg////// 5281/ zenodo. 5371628.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Reprexentations, 2020.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. On the reliability of water- marks for large language models. arXiv preprint arXiv:2306. 230.23636604444 2023.
Kezhi Kong, Guohao Li, Mucong Ding, Zuxuan Wu, Chen Zhu, Bernard Ghanem, Gavin Taylor, and Tom Goldstein. Robust optimization as data augmentation for large-scale graphs. In Proceedings of the JEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 60-69, 2022.
Ariel N. Lee, Cole J. Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of llms. arXiv preprint arxiv:2308. 07317, 2023.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zeullmoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint arXiv:2210. 20022200.10077, 15097, 2022.
Chin-Yes Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL https: : / www. aclweb. org/ anthology/Woc 1013.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic hu- man falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers). pp. 3214-3252, Dublin, Ireland, May 2022, As- sociation for Computational Linguistics. doi: 10.18653/v1/2022..e 2on. URL https: //aclanthology ozg. 2xt/22022acc-1oss... acl-long. 229.
David Nukrai, Ron Mokady, and Amir Globerson. Text-only training for image caption- ing using noise-injected clip. ArXiv,
2022. URL https://api.
semanticschol wwtelaa..rrrr//ooo http///oouII::22444//
OpenAl. Introducing chatgpt, 2022. URL https::p//// /tttt::://opeaai... http://aaaaa.nmillgoooooooo chaaaa///cttbloc/cccttt.....
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Karsten Roth, Jae Myung Kim, A. Sophia Koepke, Oriol Vinyals. Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with random words and broad concepts. 2023 IEEE/CVF International Conference Computer Vision (ICCV), pp. 15700- 15711, 2023. URL https: / wew.ntseeeaeeneieeeeeei
ottp:/or///oogooeee Corpg//rrusstiia ID 2D:25937756
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, al. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2021.
ShareGPT, 2023. URL httpt:tpp///c htttps://sharegpttt com/.

---

Published as a conference paper at ICLR 2024
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An dastructionfoolloiing llama model. 2023.
https:/ /ttp//ggttbb/bt// www/to//////o/oo woom//attet ttt//tttsuualaaa wtwpaa//staffodddalaaaa
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:230..1397,, 2023a
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307. 099288, 2023b.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint 22..:22212.10000 2022.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2021.
Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. NoisyTune: A little noise can help you finetune pretrained language models better. Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 2: Short Papers), pp. 680-685, Dublin, Ireland, May 2022. As- sociation for Computational Linguistics. doi: 10.18663/1122222acll22 achsoot URL https: //aclanthology wwwphhhwlgy...o../../eo acl--sort.........
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models follow complex instructions. arXiv preprint arXiv:2304. 122444, 2023.
Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Wang Yanggang, Haiyu Li, and Zhilin Yang. Ze- roprompt: Scaling prompt-based pretraining to 1,000 tasks improves zero-shot generalization. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 4235-4252, 2022.
Hongyi Yuan, Zheng Yuan, Chuangi Tan, Fei Huang, and Songfang Huang. HyPe: Better pre- trained language model fine-tuning with hidden representation perturbation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2244-3264, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10. 116653/v1/202..alll. 182. URL htwp::ttt:/// /aclaaaaaaaa /atpaaaaaaaaooooooooooooo org/ 22220022222222222 acl-long 182.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can ma- chine really finish your sentence? In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. putational Linguistics. doi: 111855/v1//19-1000 URL https: http::////aaaathhoooooooooooo hoggg P19-1472.
Florence, Italy, July 2019. Association for Com-
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint urXiv:2205.0 200600 2022.
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for natural language understanding. In International Conference Learning Representations, 2019.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 HYPERPARAMETERS
We finetune the 7B parameter models on four A5000s and 13B parameters eight A5000s using bfloat16 precision. After doing an initial learning rate sweep on LLaMA-1 and Alpaca, we use learning rate of 5e-5 and the Adam optimizer for all 7B models after seeing 4% improvement over baseline numbers. We train all models for epochs on all datasets setting the same seed for each run with an effective batch size of 128 (4 cards, batch size 4, 8 gradient accumulation steps). When finetuning with noise we train on three levels of noise, an L2 ball of 5, 10, and 15 over the sequence lengths and report the best one AlpacaEval using ChatGPT as the evaluator. We train with sequence lengths of 512 tokens (mainly for memory and speed) like the original Alpaca setting, finding that this does not effect the outputted response length quality corroborated by the Al- paca Leaderboard numbers. In Table 10 we see that training with longer sequences does not change performance significantly. For ShareGPT, 512 tokens using the process described by Chiang et al. (2023). When training 70 billion parameter models, we use the finetuning hyperparameters found in Touvron et al. (2023b) except we use sequence length of 2048; i.c., we use weight decay of 0.1, a batch size of 64, and a learning rate of 2e-5. We finetune for a total of three epochs on Evol-Instruct 70k (Xu et al., 2023). When using NEFTune on the 70B parameter model, we use = 15 and did not explore other (potentially bet- ter) settings due to computational constraints. Additinonally, we saw an increase in average output character length from 852 to 1241 (+389).
split the multi-turn conversations into sequences of
A.2 ADDITIONAL ABLATION STUDIES
We ablated uniform and guassian noise, finding that uniform noise performs slightly better. We also ablate the decoding hyperparameters in Figure finding minimal changes in performance. Thus, we use the simpliest sampling strategy, greedy decoding, with a repetition penalty of 1.2. We also check to see if NEFT continues to yield improvments you increase the number of training epochs. From Table 12, we see that there is a plateau in performance that is reached at higher epoch counts. In Table we freeze different parts of the model understand if certain parts of the model critical for NEFT.
Table 7: AlpacaEval Win Rate with ChatGPT (GPT-4 in parentheses) evaluator under different decoding strageties from this we can see that there seems to be little variation in performance. The WizardLM and LLaMA-Chat hyperparameters were obtained from generation config files from Huggingface. All sampling techiniques had a reptition penalty of 1.2.
Hyper. Source
top-p
temp.
LLaMA2-7B (Evolve)
LLaMA2-7B-NEFT (Evolve)
Base HP O HP 1
greedy
62.55 (70.34)
67.58 (79.60)
0.1
0.1
63.11
66.83
0.35
0.5
62,48
66.71
WizardI
0.6
0.9
62.05
66.96
LLaMA-2
0.9
0.6
63.73 (70.49)
65.47
Table 8: AlpacaEval Win Rate according to ChatGPT while varying the set of trainable parame- ters when finetuning LLaMA-2-7B on the Alpaca dataset. The top two rows have all parameters set trainable.
Setting
AlpacaEval (ChatGPT Eval)
standard finetuning
48.26
NEFT NEFT+Embed frozen
62.55
61.06
NEFT+LM-heee frozen
61.12
NEFT4Attention blocks frozen
22.17
LLaMA-2 (no finetuning)
22.17
13

---

Published as a conference paper at ICLR 2024
Table 9: NEFT performs better than FreeI.B.
ChatGPT Win Rate
LLaMA-1-7B (Evolve)
62.30
+NEFT
67.45
+FreeLB (after hparam tuning)
63.48
Table 10: Using ChatGPT as the evaluator, we observe a slight performance increase when training with longer sequences the ShareGPT data compared to standard finetuning the sequence length.
LLaMA-2 (7B)
Split 512
Split 1024
ShareGPT
63.48
61.68
+NEFT
64.22
64.35
Table 11: Ablating the alpha for LLaMA-1 trained on alpaca for values with powers 10, we see that for lower values of alpha performance AlpacaEval does not change, and for very high values like 100, makes instruction finetuning very difficult.
Alpha
AlpacaEval (ChatGPT Eval)
100
0.00
50
57.52
10
61.99
5
58.32
1
49.69
0.1
47.70
0
48.50
A.3 ADDITIONAL ANALYSIS
How does the noise impact the tokens? Since our modeling involves adding random noise to embeddings during the training stage, we examined whether the added noise changes the semantics of the token sequences in the training data. For this analysis, we sample a random 5200 samples from the Alpaca dataset, embed each training point using the embedding layer of different models, and then add different levels of noise by varying the scaling factor a. then project the noised embeddings back to their closest neighbor in the embedding matrix. We compute the % of token flips in each sentence and average the flip rate across all the samples. We present the flip scores for 7 models in Fig. 7 (Left). While none of the sentences had any flips up to when a 2 25. Note that all the results presented in the paper use a < 15. Interestingly, a L.LMA-11 model finetuned Alpaca does not show any flips even at higher levels of a.
15,
some flips
Impact of noise embedding similarity We also analyzed how the similarity of embeddings changes when we perform NEFT training. We looked at the top 100 singular values of the embedding similarity matrix for all the models. We present the results in Fig.7 (Right). For given base model (LLaMA-1 I or LLaMA-2), the singular value distribution did not change across the variants, with or without NEFTune. This shows that the addition of noise during training does not impact the embedding similarity distribution very much.
How does noise impact embeddings? In the previous analysis, we evaluated whether any tokens flipped when noising sequence real training data. We also examine the embeddings of models, LLaMA-2, LLaMA-2 Alpaca, and LLaMA-2 Evol-Instruct in isolation. In this experiment, we sweep over all of the tokens in the vocabulary adding noise to each one, and count the number of noised embeddings whose nearest neighbor is different from their un-noised starting point. We present the results Fig. We vary 2 factors: base noise seale a. and the "sentence length" scaling factor I. which is used in calculating the final noise coefficient. Even high levels of noise, only
14

---

Published as a conference paper at ICLR 2024
60
60
Normal NEFTune
Normal
NEFTune
40
40
Count
Count
30
20
20
0
0.2
0.4
0.6
0.8
0.2
0.6
0.8
1.0
ROUGE-L (w/o Noise)
ROUGE-L L (w/ Noise)
Figure 6: Left shows the ROUGE-L of training with and without NEFTune. Right shows ROUGE-L of training with and without NEFTune with noise added during inference. We can see that a lot of the training data be recovered during generation when applying noise during inference time. Note the set up similar to Figure 5.
LLaMA-2 Evol-nnstrutt NEFT
Avg rate across sentences
10
LLaMA-2 Evol-Instruet
LLaMA-2 Alpaca + NEPT
LLaMA-2 Alpaca
LLaMA-2
10-3
LLaMA-1 Alpaca NEFT
LLaMA-1 Alpara
10-4
10-5
10
15
25
50
Noise scaling factor
Figure 7: Ratio of tokens per sentence flipped at different levels of noise added to embeddings.
Top 100 singular value distribution
LLaMA-2
LLaMA-2 Alpaca
500
LLaMA-2 Alpaca + NEFT
LLaMA-2 Evul-Inetret
100
LLaMA-2 Eval-Instruct + NEFT
300
L.aMA-1 Alparn
200
100
Figure 8: Top-100 eigenvalues of embedding similarity matrices across models.
15

---

Published as a conference paper at ICLR 2024
Table 12: A1 pacaEval ChatGPT Win Rate function of the number of finetuning epochs.
Epochs
LLaMA-2 (7B) Alpaca
+NEFT
1
40.50
55.09
3
48.26
62.55
5
48.94
62.24
7
48.63
60.50
9
47.45
58.14
Table 13: AlpacaEval win rates judged by ChatGPT of different regularizers. We ablated dropout over the attention probabilities over 0.05, 0.1, and 0.5, and weight decay, 0.1, 0.2, and 0.5, report- ing the best for each of the techniques. As seen below, the benefits of NEFTune are not seen by the other regularizers.
Method
NEFTune
Dropout
Weight Decay
Baseline
Win Rate
61.9
47.8
48.0
48.5
LLaMA-2
LLaMA-2 Alpaca
Fmbbed. flips
130
512
512
256
256
120
120
64
110
21
2
32
Noise sealing factor
Noise scaling factor
LLaMA-2 Evol-Instruct
4 Embedd. flips
130
aeyoster
512
120-
128
64
21
32
Noise scaling factor
Figure 9: Number of tokens flipped at different levels of noise and sentence lengths (out of 32000). Model names corresponding each plot are the title.
very small number of tokens actually flip (I 0.4%). This shows that NEFT training does not change the semantic relationships between the tokens.
Interestingly, this experiment suggests that, if considering NEFTune as a type of data augmentation applied the embeddings of training tokens, the actual effects of NEFT mostly result of perturbations in intra-token space since the noise scales used rarely amount to displacements that transform any individual token into a meaningfully different one. Nevertheless, in Figure 10 show the tokens corresponding to embeddings of LLaMA-2 Alpaca model that do actually flip even with the lowest amount of noise utilized in our study.
Table 14: a used Fig
Alpaca
Evol-uutt
OpenPlatypus
ShareGPT
OPT 6.7B
15
15
15
LLaMA-1 7B
10
10
15
10
LLaMA-2 7B
5
5
15
15
16

---

Published as a conference paper at ICLR 2024
['tt', 'elte', 'convas", 'MOH', _Introduction', "odb', "_rpyn", '_fresh', '<0x68>', '_pal', 'rie', '_RiO', 'd', "chair, 's', 'R', "_aalax,, '_klein', 'abre', "chain', 'hom', '5', 'menTam" "_acc', svo Rob'', 'JW'", 'pot', "_DES', 'ctors', "_Santos', "_xones', "ps', 'cluded', 'hit', "sch", "_we'',
'a', Legas' 'and'. "., 'x', "_veg, 'Insert', "_sure', 'd', "applicable', 'Larr', "Downloed", 'MA," '_Branch', 'uses', '_focus, "_rural', 'forences', 'categories', "_hidden', 'A,, "legt', "_Audiad', "_ungu", 'hens" '_roject', exeec', "Lentering', "_leta', "ic', _estinates', '_bct', '6m'', '_Gel', '_mal', "_hyd,, ',3,, '3)5,", "multiple', _older", "3', "_occhter,, "_Jul', "accept', 'xpath', "ändose", "oned", "tout", '55', 'dr', JSON', 'istro', _dpployment', , 'éque", "_onurmm", 'andr', 'X', 'idents', 'er', '_Arog', "_largely']
_nttoon' ' _Th, 'M", "ady', 'icheung', "_stands', "_mies", '_geoublic', 'ilt',
Figure 10: Tokens flipped at 1 and sentence length 512 in LLaMA-2
17

---

Published as a conference paper at ICLR 2024
A.4 QUALITATIVE EXAMPLES
For these qualitative examples for LLaMA-2 finetuned models, we chose one instructions that re- quires knowledge explanation and one that involves "creatativity." We pull these examples from Al AlpacaEval, particularly the Vicuna Eval subset. Note we did not activately cherry-pick these examples.
AlpacaEval (Vicuna Subset) Instruction: Can you explain the basics of quantum computing?
-LaMA-2-Alpaca-7BB Response:
Quantum computing based on the principles of quantum mechanics, which allow for phenomena such as superposition and entanglement between particles. This means that information can be processed in parallel instead of sequentially like traditional computers. Quantum bits (qubits) are used to store data instead of classical bits, allowing more complex computations to take place with fewer resources than would otherwise be necessary.
LLaMA LLaMA-2-Alpaca --a------paaaa-B----- -B-NEFT Response:
Quantum computing is based on the principles of quantum mechanics, which describe how particles such electrons and photons behave the subatomic level. Unlike traditional computers, which process information in binary digits (bits) that can be either 0 or 1, quantum bits (qubits) can exist in multiple states between 0 and simultaneously. This phenomenon is known as superposition and allows for much more complex calculations be performed than would otherwise be possible with classical computers. Quantum algorithms are designed to take advantage of this property by encoding data into qubit states so that it can be processed exponentially faster than conventional methods. Ad- ditionally, quantum entanglement-the ability for two particles to remain connected even over great distances-allows for large amounts of data to be sent across networks quickly and securely. The potential applications of quantum computing range from cryptography and artificial intelligence to drug discovery and material science. With continued research and development, quantum technology has the potential to revolutionize many industries. LLaMA-2-Sharees 2-ShareGPT-7B Response:
type of computer that quantum -metttnncc phenomena, such as superposition and entanglement, to perform operations on data. They have the potential to solve certain problems much faster than classical computers, which are limited by the laws of classical physics.
Quantum computers
One way to think about how quantum computer works is in terms of its basic build- ing blocks or "qubits." Qubits can exist in multiple states once property known as superposition), and they can also be linked together so that their state depends on the state of other qubits (a phenomenon called entanglement). These properties allow quantum computers to perform many calculations simultaneously, taking advantage of the principles of quantum mechanics to achieve exponential speedup for certain types of computations. Response:
Quantum computers are a type of computer that use quantum-mechanice phenomena, such superposition and entanglement, to perform operations on data. They have the potential solve certain problems much faster than classical computers, which are limited by the laws of classical physics.
Here is brief overview of some key concepts quantum computing:
Superposition: In classical computing. bits can only be either or at any given time. However, in quantum computing, qubits (quantum bits) can exist in multiple states simultaneously, known as superpositions. This allows for many calculations to be performed simultancously, increasing the speed of computation.
2. Entanglement: When two particles become "entangled." their properties become correlated even when they separated by large distances. For example, if two photons are created with opposite polarizations, measuring the state one photon will instantly determine the state of the other, regardless of how far apart they are. This property be used to create network of entangled qubits, allowing them to work together to solve
18

---

Published as a conference paper at ICLR 2024
complex problems.
3. Interference: Just like waves can interfere with each other to produce new patterns. qubits can also interfere with each other through process called quantum interference. By manipulating these interferences, quantum algorithms can be designed to take advantage of the unique properties of quantum systems.
4. Measurement: One of the biggest challenges in working with quantum systems is that measurements collapse the wave function, effectively freezing the system into its final state. This means that once a measurement taken, the information stored in the quantum system cannot be changed anymore. As result, quantum computations must be designed carefully to ensure that the desired outcome is achieved before the measurement is taken.
5. Error correction: Because of the inherent instability of quantum systems, errors occur during computation. To overcome this challenge, error correction codes must be implemented detect and correct mistakes made by the quantum system. These error correction techniques are critical to making practical quantum computers reality.
6. Topological Quantum Computing: Another approach to building quantum computer uses topological phases of matter, where the order parameter takes values in non-commutativv space. It has been shown that certain classes of topologically ordered states can host fault-tolerant quantum computation.
7. Degrees of freedom: In classical computing, there usually just one degree of freedom per bit whether it is set to 0 1. But in quantum computing, there be multiple degrees freedom per qubit, leading to more possibilities for encoding information and performing caleulations.
8. Classical vs Quantum gates: Gates are basic building blocks of quantum circuits, similar logic gates in classical computing. While both classical and quantum gates perform operations on data, the rules governing them differ significantly. For instance, while classical gates operate on binary digits (bits), quantum gates operate on qubits, and the principles underlying their behavior are fundamentally different.
ALaMA-2-Evolver
Sure, I'd be happy to help!
type of computation that uses quantum-mechanica phenomena, such superposition and entanglement, to perform operations data. Unlike classical computers, which use bits (which can only have one of two values, either or 1) to represent information, quantum computers use qubits (which exist in multiple states simultaneously). This allows for much faster processing times when comes to certain types of calculations.
Quantum computing
One way that quantum computers are different from classical computers is through their ability perform parallel computations. In classical computer, each bit must be processed sequentially, but in quantum computer, many qubits can be manipulated at once using something called quantum gate. These gates allow for complex algorithms be executed quickly by taking advantage of the properties of quantum mechanics.
Another key aspect of quantum computing entanglement, which occurs when two or more particles become connected in such way that the state of one particle affects the state of another, regardless of how far apart they may be. Entangled qubits can used to perform certain types of calculations much faster than non-entangled ones because the results of measurements made on them are predetermined based on their initial state.
Overall, while there is still much research being done into the potential applications of quantum computing, this new field has great promise for solving some of society's most challenging problems related eryptography, optimization, and simulation.
LLaMA-2-EaN 2-
Sure, I'd be happy help!
Quantum computing is a type of computation that uses quantum-meecaniial phenomena, such as superposition and entanglement, to perform operations on data. Unlike classical computers, which use bits (which can only have one of two values, either or 1) to represent information, quantum computers use qubits, which can exist in multiple states simultaneously. This allows for much faster processing times when it comes to certain types calculations.
The basic idea behind quantum computing is to series of quantum gates to manipulate
19

---

Published as a conference paper at ICLR 2024
the state of individual qubits. These gates are similar to the logic gates used in classical computer architecture, but they operate on a different level of abstraction. For example, instead of using AND and OR gates to build more complex circuits, quantum gates like the CNOT gate allow us create entangled pairs of qubits with specific correlations between their states.
One of the key advantages of quantum computing is its ability to solve certain problems exponentially faster than classical computers. One famous example is factorization, where we need find the prime factors of a large number. Using Shor's algorithm, a quantum computer factorize numbers much larger than what would be possible with classical methods. Another area where quantum computing has shown promise is simulating quantum systems, which could lead to insights into materials science and chemistry. However, there are also many challenges associated with building practical quantum computers. One major issue is error correction, since any disturbance to qubit can cause it to lose its quantum properties and become a classical bit. Researchers are currently working on developing error-correcting codes that can mitigate these errors and make quantum computations more reliable over time. Additionally, there are still many open questions about how best architect quantum computers, including issues related scalability and interconnectivity between components.
DAMMA--2-penPllayppae
Quantum computing is rapidly developing field that has the potential revolutionize how we process and store information. It relies on the principles of quantum mechanics, which describe the behavior of matter and energy very small scales, such atoms and subatomic particles.
In classical computers, bits represent either O or 1, but in a quantum computer, qubits can exist in multiple states simultaneously. This property, called superposition, allows quantum computers to perform many calculations in parallel, greatly increasing their processing power. Additionally, quantum computers use entanglement, where two or more particles become connected in such way that the state of one particle instantly affects the other, regardless of the distance between them. This phenomenon enables quantum algorithms to solve certain problems much faster than classical algorithms.
One example of problem that could be solved quickly by quantum computer is factoring large numbers into prime factors. While this may not seem like a significant breakthrough, it would have wide-rangee sideiianging implications for fields such cryptography, since it would enable the rapid cracking of codes used to secure sensitive data. Another area where quantum computing could have major impact is drug discovery, it could help identify new treatments for discases much faster than traditional methods.
However, building practical quantum computer with enough qubits to be useful re- mains challenge. Currently, most quantum computers are relatively small and prone errors due to environmental interference. Researchers are working on improving both the number of qubits and the accuracy of quantum computations, making progress towards realizing the full potential of this exciting technology. LLaMA-2-Openn 1aai--2--ppPPPtyp-s-eeErPP
Quantum computing is a field of computer science that deals with the use of quantum mechanics for computation. It involves developing and studying computational models based on the principles of quantum theory, which describe the behavior of matter and energy at very small scales. The goal of quantum computing is to take advantage of the unique properties of quantum systems, such superposition and entanglement, solve complex problems more efficiently than classical computers can.
In classical computing, bits represent either 0 but in quantum computing, qubits (quantum bits) can be in multiple states simultancously until they are measured. This property, called superposition, allows quantum computers to perform many calculations parallel, greatly increasing their processing power compared to classical computers. Another key feature of quantum computing is entanglement, where two or more particles become correlated such that the state of one particle instantly affects the other, regardless of the
20

---

Published as a conference paper at ICLR 2024
distance between them. Entanglement can be used to transmit and process information instantaneously, enabling faster communication and computations.
One of the main challenges in building practical quantum computers is controlling and manipulating individual quantum bits accurately enough to realize useful algorithms. Currently, most quantum computers rely on experimental platforms using trapped ions, superconducting circuits, or photons. These devices are still relatively fragile and prone to errors, known "noise," which must be corrected by sophisticated error-correction codes. However, significant progress has been made in recent years towards scaling up quantum computers and achieving higher levels of noise tolerance.
Some potential applications of quantum computing include cryptography, optimization problems, simulation of physical systems, and drug discovery. Quantum computers could also play a role solving some of society's biggest challenges, such as climate change modeling, materials science, and protein folding research. While much work remains be done before quantum computers become widely available, the field continues to attract attention from both academia and industry due its great promise for transforming various fields of study and engineering.
AlpacaEval (Vicuna Subset) Instruction: Write a symphony concert review, discussing the orchestra's 's performance and overall audience experience. LLaMA-2-Alpaca Response:
The symphony concert was truly remarkable! The orchestra played with great energy and skill, creating a beautiful soundscape for the audience to enjoy. Every piece they performed expertly executed, from the gentle melodies of Mozart to the powerful climaxes of Beethoven.
The acoustics in the hall were perfect; every note could be heard clearly and dis- tinctly. The lighting design added another layer of emotion each piece, helping draw out the emotions within the music. Overall, it was a wonderful evening full of fantastic music and unforgettable experience for all who attended. LLaMA-2-Alpaca-TI1 B-NEFT Response:
The symphony concert was truly remarkable! The orchestra played with such passion and energy that it brought tears to my eyes. Every note they struck seemed to reverberate through the hall in perfect harmony. It was clear from their performance that each musician had put in countless hours of practice and dedication create this beautiful sound.
The conductor did excellent job leading the musicians throughout the evening. His movements were full emotion as he guided them through the pieces. He would often pause for moments time allowing the audience take in every moment of the piece before moving on. This allowed us all to appreciate the intricacies of each song even more.
The crowd was filled with people of all ages; some who knew the songs well and others who were there simply enjoy the night. There wasn't one person attendance who didn't 't seem to be having great time. Even during the quieter parts of the show, you could feel the excitement the air. Whenever something particularly exciting happened everyone cheered and applauded.
Overall, can say without doubt that this symphony concert was one of the best experiences have ever had. From start to finish, everything about it was absolutely amazing, If given the chance, I will definitely go back again next year. LLLMA-2-Share-.
Response:
On Saturday evening at Carnegie Hall, I had the privilege attending a symphony concert featuring the New York Philharmonic Orchestra under the direction of Maestro Lorin Maazel.
The program for the evening consisted of three works by some of classical music's most renowned composers Beethoven's "Egmont" Overture, Tchaikovsky's Piano Con- certo No. I with soloist Lang Lang, and Dvořák's Symphony No. 9 in E minor, Opus 95 ("From the New World").
21

---

Published as a conference paper at ICLR 2024
Maestro Maazel led the orchestra through each piece with precision and passion, drawing out rich and vibrant sounds from every section of the ensemble. The strings played with great depth and emotion, while the woodwinds and brass added colorful accents throughout the evening.
Lang Lang was a particularly impressive performer as well, displaying technical mas- tery and artistic sensitivity in his interpretation of Tchaikovsky's concerto. His playing was full of energy and expression, and he seemed completely immersed the music.
As for the audience, it was clear that everyone present was there to appreciate the beauty and power of live orchestral music. There were frequent bursts of applause and cheers throughout the night, and many people around me were wiping tears of joy from their eyes during the final movement of Dvořák's 's symphony. It was truly moving experience be surrounded by so many people who shared love for this timeless art form.
Overall, the concert was true triumph for both the musicians stage and the audi- ence in attendance. It was a reminder of why we continue to support and celebrate live music, and left the hall feeling inspired and uplifted. LLaMA-2-Sarr-7B-NEa.
Response:
On Saturday night Carnegie Hall, the New York Philharmonic put breathtaking performance of Symphony No. by Ludwig van Beethoven. Under the direction Music Director Jaap van Zweden, the orchestra delivered powerful and emotional rendition of this classic work that left the packed house in rapturous applause.
The evening began with captivating performance of Mozart's Overture Die Zauberflöte, which set the stage for the grandeur of what was to come. As the lights dimmed and the orchestra launched into the opening notes of the Fifth Symphony,
clear that this would be a special evening. Van Zweden expertly guided the orchestra through each movement, drawing out every nuance of the music while maintaining sense of momentum and energy. One of the standout features of the performance was the orchestra's 's dynamic range. From the softest pianissimo passages to the loudest fortissimo climaxes, every member of the ensemble played with precision and depth. The brass section, in particular, shone throughout the evening, delivering rich and full-bodied sound that filled the hall.
In addition the superlative playing of the musicians, the acoustics of Carnegie Hall were also a major factor in the success of the evening. This iconic venue has a warm and intimate atmosphere that allows listeners seemed to hang in the air, enveloping the audience in a sonic embrace that made it easy to forget the outside world and lose oneself entirely in the beauty of the music.
fully immerse themselves in the music. Every note
As the final chords of the symphony faded away, the audience erupted into thunderous applause, showering the musicians with praise and admiration. It was a truly magical moment, one that will stay with me long after the evening came to close. Overall, would say that this was a once-in-a l-lfeiime experience that any lover of classical music should not miss.
LLaMA-2-BE 2-Evolle70k Response:
recently attended symphony concert at the renowned Carnegie Hall in New York City. The evening's program featured works by Beethoven, Mozart, and Tchaikovsky, performed by the prestigious New York Philharmonic Orchestra under the baton of maestro Alan Gilbert.
The concert began with Beethoven's lively "Leonore" Overture No. 3, which the tone for the evening's 's festivities. The orchestra played with great energy and precision, showcasing their mastery of the classical repertoire. Maestro Gilbert conducted with grace and fluidity, drawing out each note and phrase to maximum effect.
Next up was Mozart's Symphony No. 29, a beautiful work that demonstrated the composer's deftness with melody and harmony. The orchestra played with even greater nuance than before, creating a rich tapestry of sound that transported me to another world. particularly enjoyed the woodwind section, whose delicate phrases added depth and dimension to the piece.
Finally, were treated to Tchaikovsky's iconic Symphony No. 5. This was undoubtedly
22

---

Published as a conference paper at ICLR 2024
the highlight of the evening, as the orchestra brought all of their skill and passion to bear on this towering masterwork. From the haunting opening theme to the triumphant finale, every moment of the performance was filled with emotion and intensity. It was truly thrilling experience witness such accomplished musicians bring these timeless compositions life.
Overall, the evening was a resounding success. The New York Philharmonic Orchestra proved themselves to be among the finest ensembles in the world, playing with technical prowess and artistic sensitivity. Maestro Gilbert showed himself to be a skilled conductor, guiding the orchestra through each work with care and attention to detail. And the acoustics Carnegie Hall itself were simply breathtaking, providing a perfect backdrop for the music. If you have the opportunity to attend symphony concert, highly recommend it you won't be disappointed!
Response:
As took my seat in the concert hall, anticipation was building as the lights dimmed and the orchestra began to play. The first piece on the program was Beethoven's 's Symphony No. 1, which immediately captured my attention with its powerful melodies and intricate rhythms. The conductor led the orchestra with precision and enthusiasm, and the musicians responded with a dynamic performance that showcased their exceptional skill and passion for music-making.
The second movement, marked "Andante Favori," was particularly beautiful, with del- icately played strings and woodwinds creating serene and introspective mood. The third movement, "Allegro," was full of energy and drive, with the brass section adding a bold and assertive presence the music. The final movement, "Alla Da Prestissimo." was thrilling conclusion to the symphony, with lightning fast tempos and dazzling virtuosity from all members of the orchestra.
Throughout the performance, was impressed by the cohesion and unity of the en- semble, as well as the individual excellence of each musician. The sound they produced was rich and full, yet also crystal clear and precise, and there were many moments where was left breathless by the sheer beauty and power of the music.
In addition to the excellent playing, another highlight of the evening the warm and welcoming atmosphere created by the orchestra and conductor. They interacted with the audience throughout the performance, sharing insights into the music and encouraging us to enjoy the journey along with them. This made for truly engaging and immersive ex- perience, that left me feeling inspired and uplifted long after the last note had faded away.
Overall, this symphony concert was a remarkable event that I will remember for years to come. The orchestra's superb performance combined with the audience's 's enthusiastic support to create a magical night of music-making that would recommend to anyone looking for a truly unforgettable musical experience.
LLaMA-2 22- Oeennppeepppur
Title: A Night of Enchantment with the Boston Symphony Orchestra
Date: March 1 15th, 2023
Venue: Symphony Hall, Boston MA
Review by John Doe for The Boston Globe
The Boston Symphony Orchestra (BSO) returned to its home stage at Symphony Hall last night after a two-weck hiatus, and they did not disappoint they presented their first program under the baton of Music Director Andris Nelsons since his recent appointment extension was announced. With captivating lineup featuring works from three different eras and composers, this evening proved to be nothing short of enchanting for both musicians and audiences alike.
Opening the show was Jean Sibelius' iconic "Finlandia," which set the tone for what would prove to be exquisite journey through various musical styles throughout history.
23

---

Published as a conference paper at ICLR 2024
Performed without intermission, the BSO immediately captured our attention with their impeccable precision and dynamic range particularly during the stirring brass passages that brought into the heart of Fimnish nationalism. It was clear from the outset that Maestro Nelsons had prepared his ensemble meticulously; every note seemed carefully considered yet effortlessly executed, creating a seamless blend between soloists and full orchestra.
Next up was Pyotr Ilyich Tchaikovsky's beloved Violin Concerto in D Major, per- formed by one of today's most acclaimed violinists, Leonidas Kavakos. From the very first notes, it became apparent why he has earned such widespread praise; his technical prowess was on full display while also exhibiting immense depth of feeling throughout each movement. His partnership with the BSO was truly special, resulting in moments of pure magic when all eyes were fixed upon him as he soared above the orchestral accompaniment.
After intermission came another treat: Gustav Mahler's Symphony No. 4 in G Major ("Tragic"). This work often receives less attention compared to some other Mahler sym- phonies due to its relative brevity (approximately half an hour), but make no mistake there much beauty and emotion packed within these few minutes. Under Mr. Nelsons' guidance, we experienced every twist and turn along the way, culminating in thrilling climax that left us breathless before suddenly fading away into silence.
As if that weren't enough, the encore offered something entirely unexpected: Samuel Barber's Adagio for Strings. Originally composed part of string quartet arrangement, this haunting melody has since become synonymous with mourning and loss thanks largely its inclusion in films like Platoon and The Elephant Man. Hearing it live provided goosebumps galore as it filled Symphony Hall from top to bottom; even those who may have been unfamiliar with the piece prior could not help but feel moved by its poignancy.
In conclusion, tonight's performance demonstrated exactly why the Boston Symphony Orchestra remains one of the world's preeminent ensembles nearly two centuries after its founding. Each musician appeared fully invested in their craft, delivering stunning interpretations of timeless classics alongside more contemporary pieces. As someone lucky enough to witness this magical event firsthand, can confidently say that will remain etched in my memory forever just like many others who attended last night's sold-out show. 2aa-------Pppppupssseeeee Title: A Night of Enchantment with the Boston Symphony Orchestra
Response:
The Boston Symphony Orchestra (BSO) presented their latest concert at historic Sym- phony Hall in Boston on February 1st, 20XX. The evening was filled with captivating music from some of the most renowned composers throughout history. With conductor Andris Nelsons leading the way, this night proved to be one of enchantment for both performers and audiences alike.
As we took our seats inside Symphony Hall, anticipation filled the air as people chat- ted excitedly about what they were expecting from tonight's show. The hall itself is quite impressive; its ornate design featuring intricate details like marble columns and stained glass windows creating atmosphere of grandeur perfect for such event. Once everyone had found their seat, there was complete silence except for the occasional cough or shuffling of feet until finally, the lights dimmed signaling that it time for the show to begin.
The first piece performed for the rest of the evening grandiose yet serene. As expected, Andris Nelsons brought out each note vibrantly while still allowing space for individual instruments to shine through. It was clear from his direction that he had prepared extensively for this performance, ensuring every aspect of the music came alive under his baton.
Jean Sibelius" "Finlandia," Opus 26, which set the tone
Next up was Pyotr Ilyich Tchaikovsky's iconic Violin Concerto in D Major, Opus 35, featuring soloist Gil Shaham. From the very first notes played by Shaham, it became
24

---

Published as a conference paper at ICLR 2024
apparent why he has earned critical acclaim worldwide - his technical prowess combined with emotional depth made for an unforgettable performance. He seamlessly transitioned between different styles and tempos, keeping us enthralled throughout the entirety of the concerto. Whenever he stepped back into the ensemble after his solos, you could hear a collective sigh of relief as if we had all been holding our breath waiting for him to return. Andris Nelsons conducted flawlessly alongside him, adding even more depth and beauty to this already beautiful work.
After intermission, the BSO returned to the stage for Johannes Brahms' Symphony No. in E Minor, Opus 98. This piece marked another highlight of the evening, demon- strating once again how well these musicians section had its moment to shine without ever overpowering the others, resulting in a harmo- nious blend of sound that left us spellbound. Andris Nelsons showed great versatility here too, expertly navigating through various sections while maintaining momentum throughout the whole movement.
come together one cohesive unit. Each
To end the night on high note, the BSO closed with Gustav Mahler's Symphony No. in D Major ("Titan"). Right from the opening bars, it was evident that this would be no ordinary performance. There was an energy within the orchestra that seemed to infect those watching, causing many people around me to clap along enthusiastically despite not having reached the final movement just yet! Andris Nelsons continued to lead them effortlessly, guiding through this epic journey full of emotion and passion. By the time we reached the climactic finale, cheers erupted from the crowd as if we had all been released from some of magical trance. It truly felt like we had witnessed something special, something that will stay with us long after the last note had died away.
In conclusion, the Boston Symphony Orchestra put an incredible show last night, displaying impeccable musicianship and artistry throughout their performances. Their dedication to bringing new life to classic works while also exploring lesser-known gems was greatly appreciated by all who attended. If you have the opportunity to catch one of their future shows, do not hesitate allow yourself to become enchanted by their magic!
Here couple more examples from the NEFTune models that performed the best AlpacaEval (LLaMA-2 trained on Evolve-Instruct).
AlpacaEval (Vicuna Eval) Instruction: What if Vincent van Gogh had been a successful artist during his lifetime?
LLaaMA-2EEvolver 2-oovee0077
As an AI language model, I don't have personal beliefs or opinions but based on historical facts and researches, it is believed that Van Gogh would not have experienced mental illness as he was recognized for his artistic talent in his early years before developing psychiatric problems later in life. He may have continued to create more masterpieces and become of the most famous artists of all time. His legacy might also be different from what we know today since he did not achieve recognition until after his death when his brother sold some of his paintings at auction.
LLaMA-2-Evann 2-TE
As an AI language model, I don't 't have personal beliefs or opinions but here are some possible scenarios based on historical facts and speculation:
1) Van Gogh would likely have continued to create art throughout his life, as he was pro- lific in his output even before his breakthrough success. However, with financial stability and recognition, he may have experimented more with different styles and themes, pushing himself creatively in new directions.
2) His relationships with others might also have changed. With money and fame comes social status, which could have opened up opportunities for him to mingle with other artists, writers, and intellectuals of his time. He may have formed deeper friendships and collaborations with fellow creatives, potentially leading to further innovation within the arts community.
25

---

Published as a conference paper at ICLR 2024
3) Unfortunately, mental illness can be difficult to treat, especially in the late 19th century when medical knowledge was limited. Despite his success, it's unlikely that Van Gogh's struggles with depression and anxiety would have disappeared. In fact, they may have inten- sified due the added pressure and expectations placed upon him by society and his peers. 4) Ultimately, whether not Van Gogh became a successful artist during his lifetime wouldn't t necessarily change the impact made on modern art history. His work has en- dured through countless exhibitions, books, films, and reproductions, cementing his legacy as one of the most influential painters all time.
AlpacaEval (1st) Instruction: What are the names of some famous actors that started their careers on Broadway?
LLaMA-2-Evolve700s
1. Meryl Streep
2. Robert De Niro
3. Al Pacino
4. Kevin Spacey
5. Nathan Lane
6. Audra McDonald
7. James Cagney
8. Bette Midler
9. Richard Gere
10. John Lithgow
M-MMA-22EXYOve70O-NEETTT
1. Al Pacino He made his stage debut in 1967's "Does a Tiger Wear Necktie?"" and went star in several other Broadway productions, including "The Indian Wants the Bronx"" " (1968), "The Resistible Rise of Arturo Ui" (1969) and "Serpico" (1973).
2. Meryl Streep She began her career Broadway in 1975 with the play "Julia," followed by roles in "Lost in Yonkers" (1991), "A Little Night Music" (2009) and "Three Tall Women" (2018).
3. James Earl Jones The actor first appeared on Broadway in 1957's "The Long Dream," going on to star in numerous plays and musicals throughout his career, including "Of Mice and Men" (1974), "Fences" (1987) and "Driving Miss Daisy" (1987).
4. Nathan Lane He made his Broadway debut 1985's "A Funny Thing Happened on the Way to the Forum," winning Tony Award for his performance. Other notable shows include "The Producers""" (2001), "The Nance" (2013) and "Angels in America" (2018).
5. Bette Midler The singer-actress got her start Broadway in 1965's "Fiddler on the Roof." later earning critical acclaim for her performances in "Forbidden Planet" (1989), "Golden Age" (2005) and "I'll Eat You Last: A Chat With Sue Mengers" (2013).
Here is a case where both models from LLaMA-2 trained on Evolve-Instruct cannot output the correct answer:
AlpacaEval (Vicuna Eval) Instruction: Given that f(x) 5x 2x + 3, find the value f(2).
LLaMA-2-Evolve70k: f(2)=5(2} -
=5*8.447
=19-6+3
22
To find the value of f(2), we need to substitute x=2 into the function and simplify it. We have:
f(2) = =52)3 2(2)+
= 40 3
= 45
Therefore, f(2) = 45.
26


Reviews:
[{"review_id": "g2qFtDDx8N", "reviewer": "Reviewer_h2qY", "summary": "This paper introduces NEFTune, a simple yet effective augmentation technique that improves the finetuning process of language models by adding noise to the embedding vectors during training. This is an empirical paper. The authors demonstrate that this method can substantially improve model performance, showcasing a huge increase from 29.79% to 64.69% on the AlpacaEval task when applied to the LLaMA-2-7B model fine tuned with Alpaca. Furthermore, the paper highlights NEFTune's ability to outperform strong baselines on more instruction datasets, reporting a 10% improvement for models trained with Evol-Instruct, an 8% improvement with ShareGPT, and an 8% improvement with OpenPlatypus. The authors also emphasize that even powerful models that are trained with RLHF, such as LLaMA-2-Chat, can still reap benefits from additional training with NEFTune. Overall, the paper establishes NEFTune as a valuable tool for enhancing the performance of language models across various tasks and datasets.", "questions": "1) “While longer generations do score better, we see that no generation-time strategy comes close to the performance of NEFTune models.” The authors mention this in the paper, however it would be great if they can clarify further about this experiment and display those results in the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The research highlights a critical need to shift focus in the field of Large Language Models (LLMs) from predominantly concentrating on expanding datasets and model sizes, to also giving due attention to optimizing algorithms and regularization techniques. This is essential for enhancing model generalization and addressing overfitting, particularly when working with smaller instruction datasets. I completely agree with the authors on this. More emphasis must be laid on the regularization based methods for LLMs especially while finetuning on small datasets.\n2) The authors introduce a simple yet effective regularization method of introducing noise into the embedding vector. Their comprehensive experimentation across a variety of instruct finetuning datasets and LLMs comparisons substantiates the effectiveness of their approach.\n3) The authors deduce that an LLM trained with NEFTune yields longer and more comprehensive text generations, supported by qualitative examples where the NEFTune-generated text appears more detailed than its counterpart. However, they raise concerns about potential repetitiveness in these extended outputs. To address this, they employ n-gram and log diversity metrics, ultimately concluding that the text encompasses more information than mere repetition. They also conduct experiments to ascertain whether the generation of longer, comprehensive responses inherently leads to improved model performance. Their findings confirm that NEFTune outperforms this baseline, demonstrating its efficacy.\n4) The authors attribute the substantial performance of NEFTune, surpassing that of baseline methods, to its ability to mitigate overfitting and enhance generalization. They support this claim with experimental results presented in Figures 4 and 5, although I do have some questions regarding this aspect.", "weaknesses": "I hope my comments will further strengthen the work.\n\n1) The authors assert that their method outperforms baseline approaches by reducing overfitting. In this context, I would appreciate the inclusion of standard deviation values for the results, calculated over five random seeds. A small standard deviation with NEFTune would indeed validate its efficacy. Conversely, a large standard deviation might indicate that the stochastic nature of the noise added to the embedding layer doesn’t genuinely contribute to a regularization effect. Inclusion of standard deviation is insightful when finetuning an LLM on small datasets with a goal to mitigate overfitting [1, 2].\n\n2) I noticed that the BLEU and Rouge-L scores on the training data for models with NEFTune applied appear to be relatively low. This observation leads me to wonder if the model is able to effectively learn from the training dataset. To gain a more comprehensive understanding of the training process, I would kindly request the inclusion of the training and validation loss curve plotted as a function of iterations for the +NEFT and without NEFT method. This additional information would be greatly appreciated and beneficial for a more thorough evaluation of the model's learning dynamics. \n\n3) I've noticed that in the tables, such as Table 3, terms like \"+NEFT 5\" and similar are mentioned. It would be helpful if these terms could be explained or described directly in the captions of the tables to provide immediate context and clarity for readers. \n\n4) Though adversarial ML literature (Zhu et al., 2019; Kong et al., 2022) has been cited for the choice of the noise α/√Ld. However, since it is the backbone for the entire work, I would suggest the authors to have a detailed discussion about it in the work explaining the reason behind this choice. \n\n[1] Zhang, Haojie, et al. \"Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptively.\" Advances in Neural Information Processing Systems 35 (2022): 21442-21454.\n\n[2] Somayajula, Sai Ashish, et al. \"Bi-level Finetuning with Task-dependent Similarity Structure for Low-resource Training.\" Findings of the Association for Computational Linguistics: ACL 2023. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qUe8l2ybSJ", "reviewer": "Reviewer_Agi9", "summary": "This paper proposes NEFTune, an augmentation strategy for instruction fine-tuning language models.  NEFTune augments samples by adding uniform random noise to the embeddings.  Fine-tuning language models with NEFTune helps models generate longer responses leading to better performance on OOD conversational and instruction eval tasks without harming knowledge evaluated through Q/A tasks. The authors further suggest that NEFTune performs well by reducing overfitting on fine-tuning datasets.", "questions": "Q1: Given that models have access to the same information, but tend to generate more text, there are some concerns around hallucination the model might be doing.  Have the authors checked this?\n\nQ2: Does NEFTune improve smaller models such as GPT-2? \n\nQ3: Following on Figure 4, is it possible to measure Alpaca training loss and Alpaca validation loss difference? If the model is overfitting and NEFTune reduces overfitting, we should see this discrepancy decrease.  It appears not to be an overfitting argument but smoothing the embedding space where data from the OOD dataset like Evol-Instruct may be.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The main strength of this paper are its strong results on improving conversational ability.  Authors show that across 5 different eval datasets, NEFTune improves performance by 8-35%, while retaining performance on knowledge and reasoning Q/A tasks. The proposed approach is simple to implement requiring only a simple change to add noise to the embeddings of the LLM.  This makes it so that the authors can easily test the method on large models (7B) and varied datasets.\n\n* The authors have also made efforts to demonstrate the effects of adding noise to the embeddings in particular leading to the model generating more text, and leading to higher loss (reduced overfitting).  \n\n* Augmentations in NLP for training LLMs are relatively unexplored.  The proposed method is similar to prior work in vision, but new to LLM training which typically only trains for a small number of epochs to avoid overfitting.", "weaknesses": "* While performance looks strong with NEFTune, the authors do not evaluate with other augmentation strategies such as those in https://arxiv.org/abs/2106.07499.  These methods can help with limited datasets and overfitting.  NEFTune can appear stronger with comparison to other augmentation and regularization strategies.\n\n* Experiments are done only with 7B parameter models - primarily LLAMA-2 7B.  While results appear strong, and are applicable to large models, these models are trained on large amounts of data and have learned good embedding spaces.  It will be interesting to know if results with NEFTune scale with model size.  Does NEFTune work with small models and larger models? Particularly for smaller models, where the embedding space may not be as strong, or the model is not as susceptible to overfitting.\n\n* Figure 4 shows that Alpaca loss increases when training with NEFT as this adds noise to the embedding space.  Does this mean that performance will be worse on Alpaca eval?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vL1VXL25QU", "reviewer": "Reviewer_nFQe", "summary": "The authors demonstrate that adding noise to the embedding vectors during finetuning leads to significant performance increase in large language models. The results were demonstrated using two evaluation benchmarks, AlpacaEval and OpenLLM Leaderboard tasks. Only the former demonstrated significant increase where the NEFT models increased the output length while preserving diversity.", "questions": "- I am curious to know why are there no performance gains on the OpenLLM Leaderboard tasks?\n- Are there any other forms of regularization at the embedding level that replicates this result?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The authors demonstrated empirically that a simple regularization of adding uniform noise to the embedding features lead to an average improvement of 15% across 5 datasets.\n- The performance gains were demonstrated using 3 different LLMS, which suggests the generalizability of the method to other architectures.", "weaknesses": "- Overgeneralizing performance gains: While the authors demonstrated significant performance gains using the AlpacaEval dataset with 805 instructions, the NEFTune model did not show significant improvements on the OpenLLM Leaderboard tasks (Fig 3), which is more critical since it tests for reasoning and truthfulness. The authors' claim should consider this lack of improvement in their abstract and title so as to not increase the hype of this method without strong foundations. \n\n\n- Contribution of uniform noise is unclear: The authors clarified that longer outputs caused AlpaceEval scores to increase by 16%. To demonstrate that their method did not cause spurious outputs, they evaluated the model output in terms of 2-gram, log-diversity and whitespace lengths (Table 4). I am curious to know how the NEFT method increases output length without increasing 2-gram repetition and maintaining diversity. Is the model output riddled with sentence fillers, and results for 4-gram analysis were not included in the main text? The authors provided a qualitative example of the NEFT output. I am curious to know if this was a specifically chosen sample. The authors conducted human studies of which human annotators preferred the non NEFT in 30 instances (140-(80+22)). It will be helpful to show the output where the human annotators preferred the non-NEFT output over the NEFT ones. Additionally, I do not think Gaussian Noise 5 should be directly compared against Uniform Noise 5? Perhaps other forms of noise distributions should be considered to understand what the uniform noise is doing. \n\n- Need more clarity on computation:  As the authors mentioned in the Conclusion & Limitations, it is unclear why NEFT works on the AlpacaEval and not on the OpenLLM Leaderboard. More evaluation is needed on why there is a huge disparity in performance gains between metrics. Furthermore, a framework or theory on why the uniform noise improves AlpacaEval metric should be explored. Plotting the training and test loss distribution does not add much to the understanding of the computation. A more thorough analysis of why adding uniform noise seemingly improves performance needs to be explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "491duBvQEA", "reviewer": "Reviewer_kT9q", "summary": "This paper introduces a method for fine-tuning large language models. The authors propose an extremely simple modification to the standard procedure, adding noise to the embedding vectors during fine-tuning. Their method is called NEFTune. The authors experiment with several language models and benchmarks, showing strong results across the board.", "questions": "1. Do the authors think this method would also work for pre-training? \n2. In Table 3, do the numbers next to the method name correspond to the value of alpha? If so, I'm surprised to see such a big variance, and also surprised by the non-convex behavior in many cases. Perhaps this also ties with weakness #2.\n3. This is not a question, but I just wanted to thank the authors for starting their bar plots at zero. It is refreshing not to be visually misled.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper has many strengths.\n\n1. For starters, the simplicity of the method is hard to understate. Researchers and practitioners can try this with only a few lines of code. This is very valuable and a great strength of this paper.\n2. The experiments show strong results across the board. The authors study many models and datasets, and ubiquitously see large gains.\n3. Fine-tuning large language models is a vibrant research direction, and as a paper that advances our understanding and capabilities in such enterprise, I believe this would be of interest to many in the community.\n4. There are many ablations in this work that are informative to the readers.\n5. The paper is clear and well written", "weaknesses": "1. All experiments in this paper are done with autoregressive models. Studying other kinds of models (e.g. BERT, T5) would be a valuable addition to this paper, since these models are still used for many downstream applications today.\n2. The authors don't present error bars in the experiments, which . Fine-tuning language models can be notoriously noisy (e.g. [1]), and precisly understanding the magnitude of the noise in the presented experiments would be very valuable.\n3. There is little analysis on scaling trends. Despite some experiments with QLORA, almost all experiments in the paper are conducted with 7B parameter models. Showing that the gains from the proposed method do not diminish vanish with scale would greatly strengthen the paper, as it would demonstrate it's potential for larger models.\n4. There is little discussion on hyper-parameter tuning. It would be great if authors would show that the comparisons are fair but ensuring the computational budget spent for tuning hyper-parameters for the baseline and their method is equal. One concern is that the baseline is more poorly tuned that their method.\n\n[1] Dodge, Jesse, et al. \"Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping.\" arXiv preprint arXiv:2002.06305 (2020).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)