A gentle push funziona benissimo: making instructed models in Italian via contrastive activation steering
Authors: Daniel Scalena, Elisabetta Fersini, Malvina Nissim (Università di Milano Bicocca, University of Groningen, 2024)
Topic: Language Models

Abstract
Adapting models to a language that was only partially present in the pre-training data requires fine-tuning, which is expensive in terms of both data and computational resources. As an alternative to fine-tuning, we explore the potential of activation steering-based techniques to enhance model performance on Italian tasks. Through our experiments we show that Italian steering (i) can be successfully applied to different models, (ii) achieves performances comparable to, or even better than, fine-tuned models for Italian, and (iii) yields higher quality and consistency in Italian generations. We also discuss the utility of steering and fine-tuning in the contemporary LLM landscape where models are anyway getting high Italian performances even if not explicitly trained in this language.

Introduction
The strong rise in capabilities of the latest large language models (LLMs) has brought significant improvements in a wide variety of downstream tasks. These abilities mainly derive from the instruction-tuning procedure (IT), i.e., model fine-tuning on instruction datasets, and enable the models to follow user-prompted instructions. Most LLMs, however, are mainly pre-trained and fine-tuned in English, and while other high-resource languages are included in the training data, they are not present to the extent needed to achieve out-of-the-box performances comparable to English. A strategy to address this has been, in the past few years, to fine-tune models with language-specific instructions, such as the Stanford Alpaca dataset AUTHOR, which has been automatically translated in multiple languages -- the Italian version of it has been used to train the Llama~2-based Camoscio model AUTHOR. A combination of \sim 240K training instances from three automatically translated instruction datasets was used to train the latest Llamantino AUTHOR, the most recent Llama~3-based instruction-tuned model for Italian. This approach has proven effective, but using large amounts of machine-translated texts is far from optimal: although the translation is generally good for high-resource languages, the language's unique linguistic and cultural aspects are often not represented by the training data. In addition, one must consider the usual substantial (computational) costs associated with large datasets. With recent developments in interpretability research, new approaches are arising to localize and steer different language model aspects. These techniques mainly work with an inference-time injection, allowing for targeted interventions during the generation phase without incurring the high costs associated with any additional training. Such techniques, relying on the assumption that models are already capable of performing specific tasks, aim at enhancing some of the internal activations leading to specific solutions, thereby also increasing overall performance. They have proved successful towards specific tasks, such as model detoxification, but also toward more generalist and wide-ranging tasks AUTHOR. We explore the potential of steering for Italian-instructing a pre-trained LLM as an alternative to fine-tuning, adopting a steering technique based on contrastive examples. We observe that this approach, with much less data~( \ll 100 instances instead of 240K) and no additional training required, enables performances comparable to standard fine-tuning approaches and yields high-quality Italian generations.

Conclusion
To instruct in a specific language a pre-trained LLM, steering is computationally much less expensive than fine-tuning with hundreds of thousands of (automatically translated) examples. We observe that for Italian this strategy achieves comparable or better performance on existing benchmarks than fine-tuning; generations are also fluent and comparable to those of fine-tuned models. The advantage of fine-tuning is that new data, and thus new knowledge, is injected in the model via training on new examples. At the same time, this might also trigger so-called catastrophic forgetting, yielding degradation in the output. We suggest that in the context of creating a new language-specific instructed LLM, this advantage makes sense only insofar culturally relevant and native data is used in the fine-tuning phase, so that the model can truly be enriched with language-specific knowledge, both grammatically and pragmatically. If translated data must be used, then it is incredibly more effective to use steering which requires much fewer examples (less than 0.5%) and a simple inference-time injection, making this an accessible method for virtually any language. Using native examples for the steering procedure, and possibly style-specific examples, might also yield interesting results.