Fine-Tuning on Noisy Instructions: Effects on Generalization and
  Performance
Authors: name: Ahmed Alajrami, name: Xingwei Tan, name: Nikolaos Aletras

Instruction-tuning plays a vital role in enhancing the task-solving abilities
of large language models (LLMs), improving their usability in generating
helpful responses on various tasks. However, previous work has demonstrated
that they are sensitive to minor variations in instruction phrasing. In this
paper, we explore whether introducing perturbations in instruction-tuning data
can enhance LLMs' resistance against noisy instructions. We focus on how
instruction-tuning with perturbations, such as removing stop words or shuffling
words, affects LLMs' performance on the original and perturbed versions of
widely-used benchmarks (MMLU, BBH, GSM8K). We further assess learning dynamics
and potential shifts in model behavior. Surprisingly, our results suggest that
instruction-tuning on perturbed instructions can, in some cases, improve
downstream performance. These findings highlight the importance of including
perturbed instructions in instruction-tuning, which can make LLMs more
resilient to noisy user inputs.