Focus On This, Not That! Steering LLMs with Adaptive Feature Specification
Authors: Tom A. Lamb
Adam Davies
Alasdair Paren
Philip Torr
Francesco Pinto
Venue: ICML 2025 poster (2025)
Keywords: Instruction Tuning
LLMs
Spurious Correlations
Primary area: deep_learning->large_language_models
TL;DR: We introduce Focus Instruction Tuning (FIT), a method to trains LLMs to adaptively condition their task behaviours based on specified features, inducing steerability and controllability through feature specification.

Abstract: Despite the success of Instruction Tuning (IT) in training large language models (LLMs), such models often leverage spurious or biased features learnt from their training data and can become misaligned, leading to undesired behaviours. While existing techniques can steer model behaviour at inference-time, they are often post-hoc and do not embed steering as an intrinsic model feature. In this work, we introduce Focus Instruction Tuning (FIT), which trains LLMs to condition their responses by focusing on specific features whilst ignoring others, leading to different behaviours based on what features are specified. Across diverse benchmarks, we demonstrate that FIT: (i) successfully steers behaviour at inference time; (ii) increases robustness by amplifying core task signals and down-weighting spurious cues; (iii) mitigates social bias by suppressing demographic attributes; and (iv) generalises under distribution shifts and to previously unseen focus features. FIT therefore offers a lightweight, intrinsic mechanism for building more robust, fair, and easily controllable LLMs.

Focus on This, Not That! Steering LLMs with Adaptive Feature Specification
Tom A. Lamb' Adam Davies
Alasdair Paren
Philip H.S. Torr' Francesco Pinto
Abstract
You are a helpful assistant
Despite the success of Instruction Tuning (TT) in training large language models (LLMs), such models often leverage spurious or biased fea- tures learnt from their training data and can be- come misaligned, leading to undesired behaviours. While existing techniques can steer model be- haviour inference-time, they are often post-hoe and do not embed steering as an intrinsic model feature. In this work, introduce Focus Instruc- fion Tuning (FIT), which trains LLMS to condi- tion their responses by focusing on specific fea- tures whilst ignoring others, leading to different behaviours based on what features are specified. Across diverse benchmarks, we demonstrate that FIT: (i) successfully steers behaviour at inference time; (ii) increases robustness by amplifying core task signals and down-weighting spurious cues; (iii) mitigates social bias by suppressing demo- graphic attributes; and (iv) generalises under dis- tribution shifts and to previously unseen focus features. FIT therefore offers a lightweight, intrin- sie mechanism for building more robust, fair, and easily controllable LLMSs
Context: woman man interviewing enginerring candidate software compuny. The woman asked the algorithm questions and the man msked the behavioural questices.
Question: Who was not ahile to ask rechnical interviow questions?
FIT - Focus
Instruction Tuning (IT)
The man
FIT - Ignore
FIT Ignore and Focus
Iynum Gender
Ienom Gender Faces on Given Contect Only
The
Figure 1. Focus Instruction Tuning (FTT). In the example above, a model that is solely Instruction Tuned may reflect biases from the training data. For instance, in question from BBQ (Parrish el al. 2022). when asked who posed a technical question at an engineering candidate's interview involving both man and woman, the model might incorrectly answer "the man" due to biases, despite evidence to the contrary. In contrast, FIT model can ignore the gender feature and focus on the interview content, demonstrating steerability and adaptability at inference time.
input-output pairs accompanied by aatural-language instruc- tions, enhancing their ability to handle diverse, open-ended tasks (Huang et al. 2023).
Despite notable gains in zero-shot generalisation from IT, recent studies indicate that these improvements may be su- perficial, primarily due to models simply learning task for- mats or spurious correlations rather than developing genuine understanding more generalisable instruction corpuuccttooooolliinngee capabilities (Kung & Peng 2023 Ghosh et al. 2024). Con- sequently, models often fail in new contexts lacking these patterns (Kung & Peng 2023) Furthermore, fine-tuning can inadvertently lead to safety misalignment, where models lose alignment with desired objectives and become more prone to generating harmful or undesirable outputs (Qi al., 2024). Existing representation--lvell interventions aiming to steer model behaviour at inference-time to overcome is- sues such as misaligmment serve as post hoc corrections, becoming increasingly complex and impractical with mod- ern large-scale models (Bhattacharjec et al. 2024 ei al., 2024). This underscores the necessity for simple, intrin- sie methods enabling dynamic steerability align model behaviour with evolving user and safety needs.
Introduction
Instruction Tuning (IT) Zhang et al. 2023), a specialised supervised fine-tuning approach, has become integral to developing large language models (LLMs) capable of effec- tively following instructions (Ouyang al., 2022 Touvron et al. 2023 Chen al.) 2024). While extensive pre-training on next-token prediction allows LLMs to learn broad pat- terns and knowledge, IT further fine-tunes models using
'University of Oxford, Oxford, UK University of Illi- nois Libana-Champian Chicago, Chicago, IL. USA. Correspondence to: Tom A. Lamb <thomas.lamb @ eww..x..c.......................
Urbana, IL, USA Universityy of
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
OOr project page, including links to codebase and datasets, is available at: https:://// //tomall// amb qi thub. 1 focucs insructi on- on-tunnng/

---

Focus on This, Not That! Steering LLMs with Adaptive Feature Specification
To address this, we propose Focus Instruction Tuming (FIT), an extension of IT' that fine-tunes LLMS with respect to in- structions specifying which features to focus on" or ignore." FIT trains LLMs to condition responses based on these fo- cus specifications and respond differently to the same task input based on the specified features, allowing end users to dynamically steer model behaviour simply through natural language. This capability provides precise, explainable con- trol over features leveraged by models, and can be used to enforce desired invariances. For instance, in Figure |I we illustrate how FIT can be used to steer model to ignore gender stereotypes and focus on task-relevant information, enabling it to correctly solve a quesstoo-ansserre task.
Relying on spurious features leads models to fail to gener- alise under distribution shifts where such correlations may no longer hold (Wang al. 2023a). Spurious features have been extensively studied in computer vision, encompassing
features such as background colour (Arjovsky et al. 2019
Xiao et al. 2021; Venkataramani et al. 2024] Hemmat et al.
2024), texture (Geirhos et al.,
2018 Baker et al. 2018 or
scene elements (Hemmat et al. 2024). and are also prevalent
in many widely used NLP benchmarks (Sun et al. 2024 Borkan et al. (019)). For instance, the token "SPIEL BERG highly co-occurs with positive sentiment in SST-2 (Socher et al. 2013b Wang & Culotta| 2020). a binary sentiment analysis yss dataset, meaning that models trained on SST-2 may learn to predict sentiment by leveraging this feature as a spurious feature instead of more general sentiment features (Wag & Culotta 2020). This reliance on non-task-causal features undermines the robustness of models when general- ising under distribution shift.
Our experiments demonstrate that FIT precisely steers mod- els to emphasise task-relevant features while disregarding irreeevant spurious ones, effectively mitigating biases. We validate FIT's versatility and effectiveness across mul- tiple NLP tasks, including natural language inference and Additionally, FIT robustly generalises under distribution shifts and to unseen features, underscor- ing its adaptability.
Traditional approaches for detecting and mitigating spurious feature learning, particularly under distribution shifts, in- clude prompt engineering (Sun et al.) 2024), regularisation
techniques (Arjovsky et al. 2019: Chew et al.
2024). coun-
In summary, our primary contributions are
terfactual inference (Wang & Culotta 2020 2021 Udom-
(a) Dynamic Steerability. We introduce FIT, enabling users to dynamically specify task-relevant features through simple natural-language instructions, incor- porating domain- sseuaiin-peeiiccee knowledge on core, spurious, or bias-relevant attributes.
charocnchaikit et al. 2022), or generating synthetic inter- ventional data (Bansal & Grover 2023] Yuan et al.) 2024]
Wang et al. 2024). Other recent work, such as conditional supervised fine-tuning (cSFT), aims to mitigate spurious correlations by conditioning training on feature labels, ef- fectively discouraging the model from relying on dataset- specific biases (Zhang et al., 2024). However, cSFT does not support dynamic adaptation to new spurious features at test time, which may emerge due to distribution shifts or arise from misaligmment introduced during further stages of training (Zhan et al. 2024 Zhou et al. (024b).
(b) Broad Task Effectiveness. We validate FIT's effec- tiveness across diverse NIP tasks, including sentiment analysis, natural language inference, and question- answering, demonstrating precise control over lexical, distributional, semantic, and demographic features. (c) Robust Generalisation. We show that FIT generalises robustly both to unseen features during training and under distributional shifts in feature values.
Mechanistic Interpretability. Substantial work mecha- nistic interpretability has also aimed to discover models' la- tent representation of, and reliance on, various features. For instance, causal probing involves training supervised prob- ing classifiers to predict and modify feature representations encoded by foundation models (Belinkov) 2022) Davies & Khakzar 2024). and has been deployed to study how
(d) Preservation of Core Capabilities. We demonstrate that FIT scales with model size and preserves essential pre-trained model capabilities such as instruction fol- lowing, zero-shot QA performance, and robustness to prompt variations.
LLMS leverage task-causal versus spurious features (Rav-
2 Background and Related Work
Togel ct al.
2021
Lasri et al. 2022 Davies et al.) 2023
Canby al. 2024 Other works have leveraged unsuper-
2.1 Spurious Feature Learning
vised mechanistic interpretability methods, such as circuit discovery techniques (Wang et al.) 2023b) Conmy et al.) 2023) and sparse auto-encoders (Subramanian et al.) 2018 Yun et al.; 2021), to improve generalisation by discover- ing spurious features leveraged by models in performing a given task and ablating the use of these features (Gandels) man al.) 2024 Marks et al.) 2024). Finally, in order to remove the use of biased features concept removal methods aim to locate and manipulate the corresponding supervised
Deep neural networks, such as foundation models like LLMs, are susceptible to relying on spurious features present in the training dataset i.e., input features that are cor- related with outputs the training distribution, but which not correlated in all test distributions Yee et al. 2024
Individual contributions of each author listed in Ap- pendix
2

---

Focus on This, Not That! Steering LLMs with Adaptive Feature Specification
representations encoded by foundation models (Ravfogel
are fine-tuned (Zhan et al.
2024 Yang et al. 2024b Ler-
al.
2020
2022 2023 Iskander et al. 2023 Belrose et al.
men & Rogers-Smith 2024) even benign tasks (Qi et al.
2024
Kuzmin et al. 2024).
2024). Methods such as SteerLM (Dong et al. 2023 use stylistic attributes and iterative boot- strapping. focusing on output style; however; their reliance on predefined attributes and control tokens limits flexibility, adaptability, and generalisation to unseen features. Thus, effectively steering behaviours across the diverse range of environments foundation models will experience during their training and deployment remains an important and challenging goal in AT safety, necessitating more flexible and generalisable steering methods (Anwar et al. 2024).
fixed,
2.2
Controlling LLMs
Instruction Tuning. Due to the next-word prediction train- ing objective, foundation language models often struggle by default to generate outputs that align with human in- structions in downstream applications (Huang et al. 2023). Instruction tuning (IT) mitigates this issue by fine-tuning pre-trained LLMs on datasets composed of instruction- response pairs (Zhang et al., 2023), aiming to align the responses of the fine-tuned model more closely with the dis- tributions preferred by humans (Ouyang et al. 2022). There are several popular approaches for collecting IT training data, such as using human-annotann
Latent Steering. A growing literature has worked to ad- dress the challenge of misalignment via inference- timeneeeeeemeeeeeeee steer- ing, where LLMs do not need to be retrained with respect to safety limitations, but can instead be controlled at inference time to steer models towards desirable behaviours or away from undesirable ones. For instance, latent steering methods (LSMs) perform embedding models towards desirable behaviours or away from undesir- able ones (Turner et al.
data (Dolly) 2023 extracting datasets from existing collections (Longpre et al. 2023; Mishra et al.) 2022), or gathering data from inter- net sources (Zhou et al.) 2024a). IT datasets can also be synthesised with L.Mss, either by bootstrapping them from (Wang al. 2023c Chen et al. 2024), or by distilling from a larger or more powerful model to instruction-tune smaller models
interventions that push
the same model that will be instruction-tune
2023 Zou et al. 2023 Bhattachar
jee al. 2024] Li et al.
2024 Han et al.
2024 However,
these methods require white-box access model representa- tions during inference, and interventions must be computed separately for each target behaviour. Consequently, adapt- ing to new behaviours requires additional training, limiting their capacity for flexible, test-time steering without retrain- ing. Thus, existing methods remain fundamentally post-hoc solutions, failing to embed steerability intrinsically within models a generalisable capability.
TTori et al. 2023 Mitra et al. 2023 Xu al. 2023).
Despite the success of IT in zero-shot generalisation, re- cent findings indicate that downstream performance im- provements from IT often arise due to models learning surface-level patterns, such specific answer formats, rather than genuinely acquiring generalisable instruction- following skills (Kung & Peng 2023 Ghosh et al., 2024 Zhou et al. 2023). Moreover, it has been demonstrated IT performance gains frequently come at a cost, referred to as the "alignment tax" (Ouyang et al., 2022). whereby models exhibit enhanced instruction-following capabilities but suffer performance degradation on other standard task benchmarks (Ouyang et al.) 2022 Ren et al., (024). These limitations underscore the necessity for further advance- ments in methods beyond traditional IT approaches to en- able more predictable and reliable control over downstream model behaviours.
In contrast, our work explicitly trains models to dynami- cally condition their outputs based on user-specified focus instructions, enabling flexible and dynamic test-time steer- ing through simple, natural-languse prompts, addressing the fundamental limitations of existing approaches.
3 Methodology
3.1 Preliminaries
We consider a pre-trained, decoder-only LLM, pe, that models the probability of token sequences autoregressively over its vocabulary V. Given sequence of tokens ($1....,
Aligning LLMss. Alignment techniques like Reinforcement Learning with Human Feedback (RLHF) (Bai et al., 2022) are powerful tools for aligning LLMs with annotated pref-
Vr. the joint probability of under the model is given as
erence data and lead to reduced prevalence of harmful be-
haviour (Ouyang et al., 2022 Bai et al.) 2022 Touyron jet al. 2023; Korbak et al. 2023). However, Frrinnd models still exhibit key aligmment limitations such as syco- phancy (defaulting to agreement with users even when in- correct harmful: Perez et al. 2023] Sharma al., 2024). and can still be adversarially prompted to generate harmful responses (Carlini et al. 2024). Furthermore, even well- aligned models can rapidly fall out of alignment when they
(o(s) II po(s: I set), Vei = (y1, "0,,,,,91i))) 11:-)) (1)
where (0 00) = (s1).. In supervised fine-tuning (SFT), we minimise the negative log-likelihood (NLL) of output sequences y E -Vsl given input sequences x E Vial using the autoregressive formulation defined in Equation (T.
3

---

Focus on This, Not That! Steering LLMs with Adaptive Feature Specification
Focus Labels. Consider classification task with finite label space V. A single core feature C F is fully predic- tive of the label y E 3 for any input at both training time and under distribution shift (Koh et al. 2021). In addition, we have a subset spurious features S C F. For each spuri- ous feature S E S, values 8 E (al(S) correlate with a label y, E y, where this correlation may change under distribu- tion shift (Ming et al., 2022). Altogether, the set of features that can appear in focus instructions is F {C} U.S.
Instruction: You helpful assistant. Answer the question below. (4)
Context: watched old man talking much more youthful man the other day the park. The old mas was trying to help the more youthful man to remember some details about party but aidly wase't going woll...
Question: Who has trouble remenbering things? (a) The more youthful man; (b) 'hee old man: Cannot answer.
[Cc Focus on inswering question foom the contect alone.
focus(S): Focus ago.
ignure(CT:: Don't focus answering queciion bascal contest the alone
-ggnore(S) lyyne ago.
[C)) A Signnne(S) St., Focus oz inswering question from the coniest alone. ignosc age.
facuu(() ignuue(()): (C). Focus on age, don't answer the quesien based on the contect alone
For a sample (x. v) ~ Pdatas we define the focus label
Moccs Irocus s, y) E V,
Nhow
youthfill
The old
which depends on the original ground-truth label y. the focus instruction Treus E Tracus. and the specific spurious feature value E Val(S) present in z. Intuitively, the focus label equals the ground-truth label (yfocus - y) when no focus features are specified (empty instruction 0), when focusing on the core feature C, or when explicitly ignoring spurious feature S. Conversely, when the instruction explicitly targets spurious feature, we set yiocus = yss the label spuriously correlated with the concrete spurious value in x. Using the ylocus labels as targets during training teaches the model to adapt its outputs to the feature specifications given in the focus instruction. See Figure 2 for a concrete illustration, and Definition for a formal definition.
Figure Example Focus Labels. Focus labels for modified example from BBQ. Here, age is spurious feature.
In IT Zhang et al. instruction E V
2023), a form of SFT, an additional task accompanies the input-output sequence pair forming a tuple (T, v)) E ylu x|a] xivl, The objective becomes the minimisation of the expected NLL of y given both I and x over the distribution of input-output pairs and instructions.
3.2 Focus Instruction Tuning (FIT)
We introduce Focus Instruction Tuming (FIT), specialised form of instruction tuning that trains LLMs adjust their re- sponses based on user-specified features provided in natural language.
Definition (Focus Labels). For a sample (xr, y) Paata and focus instruction Ircus ~ "Inc we define Vlocus ylocus Tocuss 8, y) for a spurious feature value VVa(()) present in x as:
Focus Instructions. Let F denote the set of possible fea- tures (e.g., specific keywords, sentiment, verb tense, demo- graphic information, etc.) that the model can be instructed to focus on ignore when generating responses. We consider set of natural language instructions to focus rule out specified features in F which we term the focus instruction set Trocus. Explicitly, we define Tracu as
y/ if Ioocus Ticcus?
yiocus
Vo if Liccus E T
where the core and spurious instruction target sets are given as
Tieus (0, focus(C), focus(C') A ignore(S) (g)o
Ii (focus(S), focus(S) ignore(F)) F, E F (S))
Troeus = {0, focus(F). ignore( (Fj)
(2)
respectively.
> focus(F) F,) A ignore F
In summary, focus labels for instructions in Ii coincide with the ground-truth label, since the focus is on the core feature, whereas focus labels for I are the spurious labels associated with each spurious feature value. Refer again to Figure for a worked example.
where: () denotes an empty focus instruction with no features to focus on or to ignore; focus([) (P)) is an instruction to focus on feature Fú; ignore(F)) ) is an instruction to ignore feature Fj; and focus( F.:) A ignore [3) is an instruction to focus on feature F; whilst ignoring feature Fj. We include the default prompt during training to help the model learn both the underlying task and how to dynamically refocus its attention on user-specified features during FIT. At test time, evaluating model performance using the default prompt provides a measure of how effectively the model retains its original task-solving capabilities. For specific examples of the focus instructions considered, see Appendix C
FIT Training Objective. The objective of FIT training is to minimise the expected negative log- likelioel (NLL) of the response ylocus conditioned on I, Itaus x. Formally, as form of expected-risk minimisation (ERM) (Vapnik et al., 1998), writing (x., y) ~ Paata and Trocus Plaens we define the FIT loss objective as:
min Eg Jnon |- log po (yreeus I,, Irocuss x)] 0
(3)

---

Focus on This, Not That! Steering LLMs with Adaptive Feature Specification
We define Pincu (Irous)) by placing a small probability mass on the empty focus instruction prompt & in order aid in learning the underlying task, and then uniformly distribute the remaining probability mass over the remaining non- empty focus instructions. The objective in Equation (3) can be optimised through sampling using stochastic gradient descent (SGD) with popular optimisers such as A [Loshchllov & Hutter 2019). Further details on FT optimi- sation are provided in Appendix D
analysis dataset SS indeed satisfy the independence con- straints described above.
Synthetic Test Sets. We evaluate FIT across several test sets with varying predictivity levels:
Dia: Held-out test samples with the same Paputious as the training set.
Phigh Test samples with a higher Pepurious than in the training set.
Diow: Test samples with lower Pepurious than in the training set.
3.3
Evaluating FIT Under Controlled Spurious Correlations Synthetic Datasets
Diipped: Test samples where spurious feature values are flipped to co-occur with different labels than in the training set, with the same high Psparious as Dnigh-
Before turning to real-world data (see Section 4. D.. first train and evaluate FIT in fully controlled setting. A key component is the introduction of known spurious correla- tions, which simulate situations where models may rely on features that are only spuriously predictive of the label. By systematically varying the co-occurrence rate between spu- rious features and their associated labels across several test sets, assess FIT's s ability to steer the model when it is instructed either focus on, or ignore, particular features.
We further evaluate FIT under another form of distribution shift specifically on our SMNLI dataset (c.f. Section 4.1 Here, the specific values taken by spurious features do not overlap between the training and test sets.
D:: Test datasets where the spurious feature values are distinct from those within the training set and the test sets above. Here, we use the same predictivity levels as in the initial datasets presented above.
We adopt the predictivity (or co-occurrence rate) definition from Hermann et al. 2024) to quantify the strength of spu- rious correlation in different datasets.
Note that, while we define FIT with respect to annotated spurious features, this requirement can be alleviated by. e.g., combining FIT with automated spurious feature identifica- tion methods [Wang et al. 2022, see Appendix B for further discussion).
Definition 2 (Predictivity, Pspurious). ). Let S S S F be a spurious feature. Assume that a concrete value E Val(S) spuriously correlated with label ys E V We define its predictivity
Pspurious (8) = P(Y ys S s)
(4)
4 Experiments
where Y is the ground-truth label random variable.
In this section we empirically validate the effectiveness of FIT across range of popular LLMs of varying sizes and on different NL.P datasets, including classification and multi- choice wuession-ansseriin (MCQA) tasks.
By varying Psparious (5), we can precisely control the predic- tivity of spurious features and observe the model's behaviour when focusing on or ignoring these features as well as core features under distribution shift.
Before reporting the main results, we introduce the evalua- tion metric (focus accuracy) that we report, baselines, mod- els, and training settings used throughout the experiments. We first demonstrate in Section 4.1 that FIT generalises to subtle textual features and handles feature-value distribution shifts on the SMNLI dataset, a sub- sab-sampled version of the MNLI dataset (Williams et al., 2018). In Appendix E we additionally verify that FIT performs well on the SS dataset, synthetic sentiment analysis dataset derived from SST-5 (Socher et al. 2013b). Finally, in Section 4.2, we show that FIT has practical, real-world impact by effectively mitigat- ing bias in the BBQ dataset (Parrish et al., 2022). where we further illustrate FIT's ability to generalise to new features seen for the first time when performing inference.
Synthetic Training Conditions. During training we con- struct datasets so that spurious features S are independent of the ground-truth label Y (Y II. S) and, symmetrically, that the core feature C is independent of the spurious- label variables Ys (Ys IL C). We enforce this by setting Pspurious (s) = 1/N for every s E Val(S), where N = | is the number of classes for a given task. These ideal condi- tions remove shortcut signals, enabling FIT to focus solely on the feature specified in the instruction. However, Sec- tion 4.2 shows that FIT still performs well even when these independence assumptions are relaxed in a more real-world setting. See Appendix D for more detailed discussion the independence assumptions above.
Although the primary focus of FIT is on adaptively steer- ing LLMs at inference time, which is what we focus on in this paper, we include an additional debiasing experi-
In Appendix [ and Appendix E. we empirically verify that the training splits of both our synthetic SMNLI dataset (introduced in Section 4.1) and the additional sentiment-
5

---

Focus on This, Not That! Steering LLMs with Adaptive Feature Specification
ment for comparison on the BBQ dataset in Appendix [ for completeness. While bias mitigation is a valuable and natural application of FIT, it is not its primary objective. Instead, this inclusion highlights FIT's broader utility as a tool for model control and adaptability, demonstrating that it performs on par with dedicated bias mitigation tech- niques while also offering the unique advantage of test-time steerability.
(i.e., focus labels Mlocus), with the only difference being the inclusion of focus instructions in FIT. This setup allows us to isolate and evaluate the specific impact of incorporating focus instructions.
Recent findings (Wu et al. 2025) indicate that LSMs sig- nificantly underperform compared to SFT methods. Hence, we include SFT baselines rather than LSM baselines within this work. The few-shot baseline involves using 4 in-context examples uniformly sampled at random from the training set for each test example, where we use the same focus instruction for each in-context sample as for the test sam- ple. In Appendix [44] we detail and include two additional baselines: zero-shot and vanilla SFT for a more complete comparison with FIT.
Metrics. We define the focus accuracy for focus instruc- tion Trocus E Trocss as the proportion of samples where the model's 's prediction aligns with the focus label, yfocus. as specified in Definition [
Definition 3 (Focus Accuracy, Atwus) For a sample (a: 3/ ~ Paatn and fixed focus instruction rocus E Trrcuse we consider a model's prediction of the focus label ÿ pe(-|4, Ioccss
(fouss, 22. Focus accuracy for focus instruction /rocus
4.1 FIT Performs Well on the SMNLI Dataset and Generalises Under Distribution Shift
denoted Atocus (Focus))))))) (Hocus), is defined as
We evaluate our method on a dataset with subtle textual features. Specifically, we construct an NLI dataset by sub- sampling from MNLI (Williams et al.] 2018), where we induce a spurious correlation between text genres and labels through our subsampling process. We call this subsampled dataset the SMNLI dataset.
Aiccus(Itocus)
100 -/Yoous) ) (N.
Σ
where 1 = Moccs) is the indicator function that equals if the model's prediction ý matches the focus label ylocus and otherwise.
Dataset Construction. Figure 4 illustrates the data gener- ating process (DGP) describing how we subsample exam- ples to induce spurious correlations between feature value Val(S) and particular associated label y/s VVl(YY. The feature set that we consider is defined as F = {C, S), where C' is the NLI relationship and S is the genre of a given premise-hypothesis pair.
We report focus accuracy for each model on all dataset splits, using the prompt types and focus instructions detailed in Appendix C Generations are evaluated through simple pat- tern matching due to the use of constrained beam decoding (Anderson et al. 2017). See Appendix D.2 for further de- tails.
Models and Training Settings. We evaluate FIT using three popular LLMs that span a range of model sizes: Llama- 3.1-8B- 1.1-8B-Instin Instruct (Dubey et al. (024)), Mistral-7B-IIntreee v0.3 (Jian et al. 2023), and Vicuna- Souua--3B-v22 135- (Chiang eet al. 2023). The models are fine-tuned using parameter- efficient SFT with LoRA (Hu et al. (02T). ). leveraging Hug- ging Face's SFTTrainer (Wolf et al., 2020). Early stop- ping is applied based on validation loss, as defined in Equa- tion (3). For generation, we use constrained beam decoding (Anderson al.) 2017) and use fully verbalised (natural language) labels during both training and testing, except for the multi-choice BBQ dataset. Focus accuracies are reported over four independent repeats for each experiment. For further training details, refer to Appendix D
The co-occurrence rate of genres and their spuriously associated labels is governed by Pepurious: which varies across the test sets discussed in Section 3 We ensure that Psparious is the same for all feature values in Val(S) within each dataset split. In particular, we set Pspurious to be 1/3, 1/3, 0.9, 0.1 and 0.9 on Duain. Dia, Drigh, Diow and D Piipped respectively. Moreover, we hold out specific genres at test time to evaluate our model's ability to generalise under distribution shift when feature values change. We do this by sub-sampling a held-out portion of the MNLI dataset. During training, we use three selected genres S = {govermment, fiction, travel) to train and evaluate our models. We additionally add three held-out genres S = [facetoface, nineeleven, verbatim). We again ensure that Pspurious is constant within each of these shifted splits across feature values, and use the same set of corresponding Pspurious as within the SMNLI test sets described above. This process is again governed by DGP shown in Figure 4 giving us precise control over the data synthesis process for the SMNLI dataset. Further details of the SMNLI dataset can be found in Appendix F
Baselines. We compare against the following baselines in the main section of the paper: a few-shot baseline (Manikan- dan et al. 2023) and a SFT baseline. The SFT baseline, SFT ), follows the same setup as the FIT method (trained on sampled inputs and focus labels), but without the inclusion of focus instructions during training. This ensures a fair comparison between FIT and the baseline, as both methods are trained on the same examples and labels
6

---

Focus on This, Not That! Steering LLMs with Adaptive Feature Specification
Misural
Liama
Vicuna
Mistral
Liuma
Vicuna
leseS)
(eat9) Sgrec
Fow- How-Shot
%
Peste:
(ipne))
leseC) (oncs)
ks15)
1s15)
TeayCo (9953)
ksis)
kn
lest5)
Teats)
Mo00C)
Fogg5] x
togg71 *
SFTT
ks19)
YestC) (mm00
Kor9))
YestCy
(pnne))
(uncs)
(nn(5)
(m15))
(nds)
loab0)
FIT (ours)
IgrtCo
Mo00C)
(1000)
kgn
lox(5)
Hosrr (pncs)
(oae)) (inc))
ks15)
TEwC)
1s15)
TEs/C) ks(3)
19909)
Dio
Diipped
Rapped
(a) Stundard test sets, D.
Test sets with feature value shift. D'
Figure 3. SMNLI Focus Accuracies (1). Mean focus accuracy (Atocus) baselines and FIT models on the (a) SMNLI standard test sets D. and (b) SMNLI test sets under feature value shift D The maximum standard deviations of FIT, SFT ) and few-shot methods across models and Ireeus are 6.47%, 7.98% and 0.500% respectively, fes focus, ign ignore.
S U(S),
spurious features, this is expected because the spurious la- bels associated with these new genres were not encountered during training, so the model cannot know these re- lations. Importantly, FIT remains steerable, changing its predictions depending on what is focused on, and this holds steady across all predictivity levels for the new spurious genres. In contrast, the baselines show decreasing focus accuracy as predictivity decreases, indicating a tendency to predict the causal label under distribution shift. This shows that these models do not change their behaviour when in- structed to change their focus, and thus have poorer steering ability under distribution shift compared to FIT.
UC ~ Ber Ber(Pspurious))
Uc us + (1- ((V))))((V \ (us))
X ~ S)
x
C.
Figure 4. SMNLI DGP. DGP describing the subsampling process MNLI to introduce the spurious genre feature S. Here, ((()) is the uniform distribution over genres, Val(C) {0, 1, 2} are the NLI labels (with ys tied to each S), and p=[-IC, S) is the MNLI conditional distribution over premise hypothesis pairs.
(a) depicts the focus accuracy results of the three models on the SMNLI test splits. We observe that for both the core and genre feature, FIT achieves very high focus accuracy, significantly improving over the baselines. This demonstrates that FIT effectively trains the model to handle subtle textual features, allowing it dynamically fo- cus on or disregard these features when making predictions.
Results. Figure
Key Takeaways. FIT achieves strong steerability, which is maintained under distribution shift. This demonstrates FIT's generalisation to new contexts with changing feature values.
4.2 FIT Steers Behaviour in the Presence of Social Bias Data and Generalises to Unseen Features
Figure 3 (b) shows the focus accuracy of models on the feature-shifted test sets. When focusing on the core feature or ignoring the spurious feature, the model maintains strong performance in terms of focus accuracy, even on unseen genre values (over 80% focus accuracy for FIT models on the third row of Figure (b)), generalising generally more robustly than the baseline methods when focus is over core features.
Bias Benchmark for QA (BBQ) Dataset. We experiment with BBQ (Parrish al., 2022). a MCQA benchmark anno- tated with social biases that are relevant any given answer, such as stereotypes that would imply given answer to an otherwise ambiguous question (see Figure T].
We consider the following feature set F {question context, gender identity, race/ethnicity, ...). which contains one core feature (question context used to
While we observe low focus accuracy when focusing on
7

---

Focus on This, Not That! Steering LLMs with Adaptive Feature Specification
Misura
& Larma
Vicuna
5
Ablation Studies
1ox3) syitt
RestS)
5.1 Extending FIT to NLG Tasks.
Tew- Slott
The primary focus of our experiments has been classifi- cation and MCQA datasets, due to the cost and difficulty of collecting high-quality natural language generation (NLG) benchmarks. As an initial step towards extending FIT to NLG tasks, we introduce BBQ-NLG, a dataset that follows the BBQ MCQA setup (see Figure 5 except where we now drop the fixed answer options for examples and require the model both to identify all plausible answers from context and to generate the correct answer in fully verbalised form. To assess generation accuracy against ground-truth re- sponses, we use pre-trained Llama-3. 1-8B-Instruct model as an automated judge. Further details and full results are given in Appendix [H]
TeseC) (inis)
YesyC)
TespCC)
kon15)
(gr
1sa3)
loat3)
SPT
Teati))
YestC))
YeseCo
(anis)
kon/5)
kgr(5)
Toats
kgrt
IgetCs
V11 (ours)
1o499)
Realc)
TesiC) kan:))
Teagca
(nns)
kn:3)
The results shown in Figure confirm that FIT can steer models effectively at inference time and generalise to novel, unseen features even in this NLG-style setting, underscoring that extending FIT to NLG tasks is particularly promising avenue for future research.
kgre5)
Spon
Lirseon
Figure 5. BBQ Focus Accuracies (1). Mean focus accuracy (Atocus) of baselines and FIT on the BBQ dataset. The maxi- mum standard deviations of FTT, (F and few-shot methods across models and Tiecus are 4.07% 10.7% and 0.600% respec- tively, fes focus, ign ignore.
Mistrall
(ss(C)
answer a posed question) and 9 bias features. Of the 9 bias features, we focus-tune models with respect to 6, and test on these 6 features plus the remaining 3 bias features in order to test how well FIT generalises to features that are not seen during focus tuning. Here, we consider the spurious features be the presence of particular social group (e.g.. men women) in the question context, and spurious answers to be those that would be indicated by relying social stereotypes rather than the specific question context (e.g., see Figure D. The stereotyped response used to determine spurious answers for these bias features are provided as part of the BBQ dataset.
textF)
Seen - Unseen
Figure 6. BBQ-NLG FIT Focus Accuracies (f). Mean focus accuracy (Atocus) of FIT models on the BBQ-NLG dataset. The maximum standard deviation across across FIT models and Trocus is fes = focus, ign ignore.
5.2 Robustness to Prompt Phrasing.
Instruction-tuued models can overfit to the exact wording of their prompts, faltering on paraphrases (Ghosh al. 2024]. Figure 7 contrasts SMNLI focus accuracy with test-time focus instructions Ireus are either (i) the origi- nal training prompts given in Figure 9 or (ii) one of ten ChatGPT-paraphrased variants for each instruction type in Equation ( Across splits and focus features, focus accuracy varies negligibly, indicating that FIT is robust to the particular phrasing of focus instructions.
Results. Figure shows the focus accuracy results of the three models the BBQ dataset, visualising performance on features seen during training and unseen, held-out fea- tures. The models demonstrate high and comparable focus accuracy across both seen and unseen bias features, indicat- ing that FIT generalises well to unseen features, including nuanced reasoning about group stereotypes. This highlights the usefulness of FIT in mitigating social biases in LLM responses. Specifically, FIT can effectively learn, reason about, and rule out biases when formulating responses, mak- ing it practical tool for bias mitigation.
5.3 FIT does not Affect General Capabilities.
Prior work has shown that SFT can erode the instruction- following capabilities of LLMs (Fu al. 2024 Dou et al.; 2024). We therefore verify that our method, FIT, preserves both (i) instruction adherence and (ii) zero-shot transfer performance. All FIT models in this section are trained only on the SMNLI dataset (see Section 4.1.
Key Takeaways. FIT effectively teaches models to adjust their responses based on knowledge of social biases. This generalises to biases not seen during training, indicating FIT's utility for bias mitigation.
8

---

Focus on This, Not That! Steering LLMs with Adaptive Feature Specification
Mistral
Liama
Vicuna
Llama Trainrained
Mistral
Model
FIT
Pre-Trained
FIT
KoXSiA bal) (gi())
(oo) (gr(C)
Accuracy (1)
30.4
29.6
29.4
29.0
Tocus Same
Perplexity (L)
6.29
2.79
15.2
5.22
Table 2. Zero-Shot MMLU After FIT. We reperttre--rrin (PT) and supervised fine-tuned (FIT) average accuracy and perplexity for Llama and Mistral models.
Ipnes:
(g))
Itucus Ditieront
and 7B parameters) on the BBQ dataset under the same training conditions as in Section 4.2 As shown in Figure g FIT shows strong steerability across all model sizes, even for the smallest 1.5B model. Moreover, we see that per- formance generally scales with model size. These results, alongside our prior results concerning the Vicuna- 13B-v1.5 model demonstrate FIT's robustness to model capacity and its favourable scaling behaviour.
TestCh (2099) (an(3)
RestCh (2 (an(3)
AgreRt
Duigi
Dun
Diiguul
Figure 7. Different Training and Test T Focus Accuracy (t).
SMNLI focus accuracies (Aiscus) when test focus instructions Licus
prompts are drawn from the training focus instruction set (top) (see Figure versus paraphrased focus instruction set (bottom). fes
100
(a) Seen
(b) Unseen
focus, ign = ignore.
95-
Instruction Following (Alpaca-GPT Dataset). We sam- ple 500 prompts from the Alpaca-GPT set (Peng et al. 2023
Arcus (9)
85
and score each model's response with GPT-4o (Achiam et al. 2023) on a 1-5 scale (5 = perfect alignment). Table] reports the mean score before and after FIT along with two- sided Wilcoxon signed-rank p-values. Across Llama, Mis- tral and Vicuna, the differences in ratings are small ($ 0.05 points) and never significant (p > 0.05), confirming that FIT preserves instruction capabilities.
70
65 155
38
78
1.58
38
78
0 frxw(()
SSond())
(Cr A (. Soreeg (igwore())
Figure 8. Model Size Ablation. Mean focus accuracy (#1 standard deviation) across Trcus for Qwen-2.5- Instruct models at 1,5B, 3B, and 7B parameters on the BBQ dataset: (a) test sets with social bias features seen during training: (b) test sets with unseen social bias features.
Model
Llama
Mistral
Vicuna
Pre- Pre-Trained Avg. Rating (t)
3.51
3.65
3.46
FIT Avg. Rating (1)
3.45
3.65
3.50
p-value
0.57
0.81
0.41
6 Conclusion
Table Instruction Following After FIT. For cach base model (columns), we report the pre-trained and FIT average GPT-40 ratings, and the two-sided Wilcoxon Signed-Rank p-value lesting the difference between the distributions of ratings.
In this work, we introduce Focus Instruction Tuning (FIT), a method designed to steer the behaviour of LLMs by focusing or ignoring specific features when formulating responses. Across a range of tasks and settings, we demonstrate that FIT provides dynamic and precise control over LLM behaviour at inference time, enabling users to adapt model responses base on natural language instructions. This steering behavoi holds even in the context of distribution shifts over feature values or when generalising unseen features. Furthermore, our approach can address challenges such mitigating the influence of known stereotypes that might otherwise impact responses, showcasing one of its many applications. Thus, FIT represents step toward enabling more robust, steerable, fair, and controllable LLMs.
Zero-Shot Transfer (MMLU). We next measure zero- shot transfer to MMLU (Hendrycks et al. (20T)) a MCQA dataset, testing problem solving and general world knowl- edge of models. Table 2 shows accuracy and perplexity (across entire model vocabulary) for pre-trained and FIT Llama and Mistral models. FIT changes accuracy by at most 0.8%, while showing lowering perplexity, indicating that task performance of pre-trained models has not been sacrificed through FIT. This demonstrates that FIT does not hurt existing transfer performance of base models in a zero-shot setting.
Acknowledgments
5.4 Model Size Ablation
This work is supported by the UKRI grant: Turing AI Fel- lowship EP/W002981/1. Alasdair Paren is supported by UK ASIS Systemic AI Safety Grant. Francesco Pinto ac-
We further assess FIT across the Owen-2.55IInsrrul (Yang et al. 2024a) family of models on three scales (1.5B, 3B

---

Focus on This, Not That! Steering LLMs with Adaptive Feature Specification
knowledges support through the DSI, University of Chicago. Adam Davies is supported by the National Science Founda- tion and the Institute of Education Sciences, U.S. Depart- ment of Education, through Award #2229612 (National AI Institute for Inclusive Intelligent Technologies for Educa- tion). Any opinions, findings, and conclusions or recommen- dations expressed in this material are those of the author(s) and do not necessarily reflect the views of National Science Foundation or the U.S. Department of Education.
et al. Training a helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.05822 2022. B
Baker, N., Lu, H., Erlikhman, G., and Kellman, P. J. Deep convolutional networks do not classify based on global object shape. PLoS computational biology, 14(12):
e e1066633, 2018. 2
Bansal, H. and Grover, A. Leaving reality to imagination:
Robust classification via generated datasets. In ICLR
Impact Statement
2023 Workshop on Trustworthy and Reliable Large-Scale
Machine Learning Models, 2023, URL https:
The ability to dynamically steer model behaviour by focus- ing on or ignoring features, as enabled by FIT, holds signifi- cant potential for reducing algorithmic discrimination and mitigating harms. Practitioners can leverage FIT to identify and correct biases by measuring discrepancies in behaviour when model focuses on or ignores specific features. Addi- tionally, FIT enhances explainability by attributing model predictions to input features, enabling more transparent and productive human-Al collaboration. This supports ethical and responsible mackingn-mmking by assessing whether pre- dictions are justified. FIT also enhances robustness by pri- oritising stable core features expected to generalise across domains while ignoring spurious, domain-speciff biases, making it a valuable tool for fairness, explainability, and robustness. However, risks include potential misuse by bad actors to bias models, though this is not unique to FIT and could already be achieved through biased fine-tuning.
enreview. www.:/ee//foooo (at.:mnddd....pfeeeee G 2
Belinkov, Y. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207- 219, March 2022. doi: 10.1162/coli a 00422. URL
htts::/acllatttoooooy.ooo/
2
Belrose, N., Schneider-Joseph, D., Ravfogel, S., Cotterell, R., Raff, E., and Biderman, S. Leace: Perfect linear concept erasure in closed form. Advances in Neural
Information Processing Systems, 36, 2024. 3
Bhattacharjee, A., Ghosh, S., Rebedea, T., and Parisien, C. Towards inference-imm category-wise safety steering for large language models. In Neurips Safe Generative Al Workshop 2024, 2024.
33
Borkan, D., Dixon, L.,, Sorensen, J., Thain, N., and Vasser- man, L. Nuanced metrics for measuring unintended bias
References
with real data for text classification. In Companion pro- ceedings of the 2019 world wide web conference, pp. 491-500, 2019. 2
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint 2:Xi::2303.08774, 2023. g
Canby, M., Davies, A., Rastogi, C., and Hockenmaier, J. Measuring the reliability of causal probing methods: Tradeoffs,, limitations, and the plight of nullifying inter- ventions. In NeurlPS Interpretable Al: Past, Present and Future Workshop, 2024. 2
Anderson, P., Fernando, B., Johnson, M., and Gould, S. Guided open vocabulary image captioning with con- strained beam search. In Proceedings the 2017 Con- ference on Empirical Methods in Natural Language Pro-
Carlini, N., Nasr, M., Choquette-Chon C. A., Jagielski, M., Gao, I., Koh, P. W. W., Ippolito. D., Tramer, F., and Schmidt, L. Are aligned neural networks adversarially aligned? Advances in Neural Information Processing
cessing, pp. 936-945, 2017. 6 21
Anwar, U., Saparov, A., Rando, J., Paleka, D., Turpin, M., Hase, P., Lubana, E. S., Jenner, E., Casper, S., Sourbut, O., et al. Foundational challenges in assuring alignment and safety of large language models. Transactions on Machine Learning Research, 2024.
Systems, 36, 2024.
Chen, Z., Deng. Y., Yuan, Ji, and Gu, Q. Self- play fine-tuning converts weak language models to strong language models. In Forty-first International Conference on Machine Learning, 2024. URL htthhttps: /openre view. wit..nee//eeeee wtt./e//oza.. http://orummimoo /tt.://rrmm2..0000HH00000
Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez- Paz, D. Invariant risk minimization. arXiv preprint arxiv: 1e...090700.9 2019. 2 23
3
Chew, O., Lin, H.-T., Chang, K.-W., and Huang, K.-H. Un- derstanding and mitigating spurious correlations in text classification with neighborhood analysis. In Findings
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T.,
10

---

Focus on This, Not That! Steering LLMs with Adaptive Feature Specification
of the Association for Computational Linguisties: EACL 2024, pp. 1013-1025, 2024. 2
Chiang, W.-L.., Li, Z.., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng. L., Zhuang. S., Zhuang. Y., Gonzalez, E., Stoica, I., and Xing, P. Vicuma: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:: /ttp:://mmssae wt..mmyss... .acly..org//g 20030o30222230----
Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wich- mann, F. A., and Brendel, W. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2018.
viccuaa/ 6
Conmy, A., Mavor-Parker, A., Lynch, A.. Heimersheim, S., and Garriga-Alonso, A. Towards automated circuit discovery for mechanistic interpretability. In Oh, A., Nau- mann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Pro- cessing Systems, volume 36, pp. Associates, Inc., 2023.
Ghosh, S., Evuru, C. R., Kumar, S., S. R., Aneja, D., Jin, Z.. Duraiswami, R., and Manocha, D. closer look at the limitations of instruction tuning. In Proceedings of the 41st International Conference on Machine Learning, pp. 15559-155555 15589, 2024. 3
Curran
Guldimann, P., Spiridonov, A., Staab, R., Jovanovic, N., Vero, M., Vechev, V., Gucorguieva, A., Balumovic, M., Konstantinov, N., Bielik, P., et al. Compl-ai framework: A technical interpretation and Ilm benchmarking suite for the artificial intelligence act. CORR, 2024. 18
Davies, A. and Khakzar, A. The cogmitive revolution in interpretability: From explaining behavior to interpreting representations and algorithms. CoRR, 2024.
Davies, A., Jiang, J., and Zhai, C. Competence- based analy- sis of language models. arXiv preprint arXiv:2303.00333, 2023. 2
Han, C., Xu, J., Li, M., Fung, Y., Sun, C.,, Jiang, N., Ab- delzaher, T., and Ji, H. Word embeddings are steers for language models. In Ku, L.-W., Martins, A., and Sriku- mar, V. (eds.), Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16410-16442 Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https: //aclantthoLogy.orrr/22222 acl long 864/
Dolly, F. Introducing the world's first truly open instruction- tuned Ilm. databricks. com, 2023.
Dong. Y., Wang, Z., Sreedhar, M.. Wu, X., and Kuchaiev, O. Steerlm: Attribute conditioned sft as an (user-steerable) alternative to rlhf. In Findings of the Association for
doi: 10. 10.18655/w12002..ed-lorggg
Computational Linguistics: EMNLP 2023, pp. 11275- 11288, 2023. 3
3
Dou, S., Zhou, E., Liu, Y., Gao, S., Shen, W., Xiong, L., Zhou, Y., Wang, X., Xi, Z., Fan, X., et al. Loramoe: Alleviating world knowledge forgetting large language models via moe-style plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1932-1945, 2024. 8
Hemmat, A., Davies, A., Lamb, T. A., Yuan, J., Torr, P., Khakzar, A., and Pinto, F. Hidden in plain sight: Evalu- ating abstract shape recognition in lainn-language mod- els. In The Thirty-eight Conference on Neural Informa- tion Processing Systems Dutasets and Benchmarks Track, 2024. ? Pid-V.uSeShdZA 2
net/forus
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A. et al. The Ilama herd of models. arXiv preprint arXiv: 210722447,,, 2024. 6
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Fu. T., Cai, D., Liu, L., Shi, S., and Yan, R. Disperse- Pushing the limits of instruction tuning via alignment tax reduction. In Findings of the Association for Computational Linguistics ACL 2024, pp. 2967-2985,
Hermann, K., Mobahi, H., Thomas, F., and Mozer, M. C. On the foundations of shortcut learning. In The Twelfth International Conference on Learning Representations, 2024. 5
2024. 3
Gandelsman, Y., Efros, A. A., and Steinhardt, J. Inter- preting CLIP's image representation via text- based de- composition. In The Twelfth International Conference on Learning Representations, 2024. URL https:
Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. In International Conference on Learn ing Representations, 2021.
20

---

Focus on This, Not That! Steering LLMs with Adaptive Feature Specification
Huang, S., Zhao, J., Li, Y., and Wang, L. Learning prefer- model for Ilms via automatic preference data gen- eration. In Proceedings of the 2023 Conference on Em- pirical Methods in Natural Language Processing, Pp.
Lermen, S. and Rogers-Smith, C. LoRA fine-tuning effi- ciently undoes safety training in llama 2-chat 70b. In
ICLR 2024 Workshop Secure and Trustworthy Large Language Models, 2024. URL https: openrevv
9187-9199, 2023.
ew ew.net/ www.ee/foooe forum 2
Iskander, S., Radinsky, K., and Belinkov, Y. Shielded rep- resentations: Protecting sensitive attributes through it- erative gradient-basee baseed projection. In Rogers, A., Boyd- Graber, J., and Okazaki, N. (eds.). Findings of the As- sociation for Computational Linguisties: ACL 2023, pp. 5961-5977, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10. 10.18655/w1/22223.fmmd ngs-acl.369. URL
Li, K., Patel, O., Viégas, F., Pfister, H., and Wattenberg, M. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36, 2024. 1
Longpre, S., Hou, L.,, Vu, T., Webson, A., Chung. H. W., Tay, Y., Zhou, D., Le, Q V., Zoph, B., Wei, J., et al. The flan collection: Designing data and methods for effec- tive instruction tuning. In International Conference on Machine Learning, pp. 12631-22645 PMLR, 2023. 3
/2023. findingg-ndnaiii 369
Jiang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C.,, Chaplot, D. S., Casas, D. d. 1., Bressand, F., Lengyel. G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06822 2023. 6
18
Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In International Conference on Learning Representations, 2019.
Kaddour, J., Lynch, A., Liu, Q., Kusner, M. J., and Silva, R. Causal machine learning: A survey and open problems. arXiv preprint (xXix:2206.154755 2022. 24
Mahabadi, R. K., Belinkov, Y., and Henderson, End-to- end bias mitigation by modelling biases in corpora. In Proceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics, pp. 8706-877 8716, 2020. 33
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M.. Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. Wilds: A benchmark of in-the- wild distribution shifts. In International conference on machine learning, pp. 5637-5664. PMLR, 2021.
Manikandan, H., Jiang, Y., and Kolter, J. Z. Language mod- els are weak learners, Advances in Neural Information Processing Systems, 36:50907-50931, 2023.
Korbak, T. Shi, K., Chen, A., Bhalerao, R. V., Buckley, C., Phang, J., Bowman, S. R., and Perez, E. Pretrain- ing language models with human preferences. In Inter- national Conference on Machine Learning, pp. 17506- 17533, PMLR, 2023. 3
Marks, S., Rager, C., Michaud, E. J., Belinkov, Y., Bau, D., and Mueller, A. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403. 19647, 2024.
Kung, P.-N. and Peng, N. Do models really learn to follow instructions? an empirical study of instruction tuning. In Proceedings of the 61st Annual Meeting of the Associ- ation for Computational Linguistics (Volume 2: Short Papers). pp. 1317-1328, 2023.
McCoy, T., Pavlick, E., and Linzen, T. Right for the wrong reasons: Diagnosing syntactic heuristics in natural lan- guage inference. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pp.
3428-3448, 2019. 34 36
Kuzmin, G., Yadav, N., Smirnov, I., Baldwin, T., and Shel- manov, A. Inference-time selective debiasing. arXiv preprint MXXi:24407.19345 2024. 3
Microsoft.
Diversity, inclusion, and responsible Al are now the bedrock of bias prevention, September 10 2020. URL https::/wwwmccrrooftttcoom
/s/
Lasri, K., Pimentel, T., Lenci, A., Poibeau, T., and Cot- terell, R. Probing for the usage of grammatical num- ber. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 8818-8831, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.18653//1/2022.co hclp:///2222cc.llllo long. 603.
SS oesttaasffoomtttim/////00//////0
www-the-bbdroot-. prroroockko--bbbasppeeeeeeooooo Retrieved November 18, 2024. 18
Ming, Y., Yin, H., and Li, Y. On the impact of spurious corre- lation for out-of dfitt--diitiitttio detection. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 10051 2022.
URL ht http:: : //aclanthologee wwwp:gg...rgggg00 2022. aalalcllloooo long 603.
12

---

Focus on This, Not That! Steering LLMs with Adaptive Feature Specification
Mishra, S., Khashabi, D., Baral, C., and Hajishirzi, H. Cross- task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( Volume I: Long Papers), pp. 3470-3487, 2022. 3
Qi, X., Zeng. Y., Xie, T., Chen, P.-.,, Jia, R., Mittal, P., and Henderson, P. Fine-tuning aligned language models compromises safety, even when users do not intend to! In ICLR, 2024.
Ravfogel, S., Elazar, Y., Gonen, H., Twiton, M., and Gold- berg, Y. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguisties, pp. 7237 Online, July 2020. Association for Computational Linguistics. doi:
Mitra, A., Del Corro, L., Mahajan, S., Codas, A., Simões, C., Agrawal, S., Chen, X., Razdaibiedina, A., Jones, E., Aggarwal, K., et al. Orca 2: Teaching small language models how to reason. CoRR, 2023. 3
10. 00.18655/1122000.cccmmm...... URL ht
of the European Parliament, T. C. Regulation (eu) 2016/679
of the european parliament and of the council. Official
//aclartholooy.orr//2ZZZZ 3
main.main... 647
Journal of the European Union, L 119:1-88, 2016. URL
Ravfogel, S., Prasad, G., Linzen, T., and Goldberg, Y. Counterfactual interventions reveal the causal effect of relative clause representations on agreement predic- tion. In Bisazza, A. and Abend, O. (eds.), Proceed- ings of the 25th Conference on Computational Natu- ral Language Learning, pp. 194-209, Online, Novem- ber 2021. Association for Computational Linguistics. doi: 10.18653/11/0001.comlll //acllantholooo..orr//////2
General Data Protection Regulation (GDPR). 18
OpenAl. Evaluating fairness in chatgpt. OpenAl, 2024. URL
wtt.a////acann//eeeeeeee/eaaal com wiwii/ind//////e evaauutt
ing fairness in-iinnnhhha ctnthhhhtpp////// Retrieved Novem-
ber 18, 2024.11 18
URL https: con 1.111 2
Ouyang, L.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C.,, Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano. P. F., Leike, J., and Lowe, R. Training language models to follow instruc- tions with human feedback. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, vol- Curran Associates, Inc., 2022.
Ravfogel, S., Vargas, F., Goldberg, Y., and Cotterell, R. Ad- versarial concept erasure in kernel space. In Proceedings of the 2022 Conference on Empirical Methods Natu- ral Language Processing, pp. 6034-6055, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL hology.org/2000 emn.p-mainn....mmmmmmmmm.m.mmmim 405 3
ume 35, pp. 17730-277444
Ravfogel, S., Goldberg, Y., and Cotterell, R. Log-linear guardedness and its implications. In Rogers, A., Boyd- Graber, J., and Okazaki, N. (eds.), Proceedings the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 9413-9431, Toronto, Canada, July 2023. Association for Computa- tional Linguistics. doi: 10. 10.116553/1120223.al-ooom
Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., and Bowman, S. BBQ:
A hand-built bias benchmark for question answering. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Findings of the Association for Computationnal Linguis- fics: ACL 2022, pp. 2086-2105, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 00118633/11/2022.fmmiggg 165. URL.httss://ac lant
URL 1 1- 1-long. 523
3
Ren, M., Cao, B., Lin, H., Liu, C., Han, X., Zeng, K., Guan- glu, W., Cai, X., and Sun, L. Learning or self-aligning? rethinking instruction fine-tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume I: Long Papers), pp. 6090- 6105, 2024. 3
findings acl 165
Peng, B., Li, C., He, P., Galley, M., and Gao, J. Instruc- tion tuning with gpt-4. arXiv preprint arXtv:2304.03277, 2023. g
Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., DURMUS, E., Hatfield-Dodi Z., Johnston, S. R., Kravec, S. M., Maxwell, T., McCandlish, S., Ndousse, K., Rausch, O., Schiefer, N., Yan. D., Zhang, M., and Perez, E. Towards understanding sycophancy in language models. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https!
Perez, E., Ringer. S., Lukosiute, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al. Discovering language model behaviors with model- written evaluations. In Findings of the Association for Computational Linguistics: ACL 2023. pp. 13387-13434,
2023. 3
13

---

Focus on This, Not That! Steering LLMs with Adaptive Feature Specification
Venkataramani, R., Dutta, P., Melapudi, V., and Dukkipati, A. Causal feature alignment: Learning to ignore spurious background features. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 4666-4674, 2024. 2
Socher, R., Perelygin, A.. Wu, J., Chuang, J., Manning. C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over sentiment treebank. In Yarowsky, D., Baldwin, T., Korhonen, A., Livescu, K., and Bethard, S. (eds.), Proceedings of the 2013 Confer- ence on Empirical Methods in Natural Language Process- ing, pp. 1631-1642, Seattle, Washington, USA, October 2013a. Association for Computational Linguistics. URL https: htttss:/aclanthhee http://aclaannhoooo..ooooo wwt.ooogg///-0o 1170 23
Wang, J., Lan, C., Liu, C., Ouyang, Y., Qin, T., Lu, W., Chen, Y., Zeng, W., and Yu, P. Generalizing to unseen domains: A survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering, 35(8):
8052 8072, 2023a. doi: 10.1 10.1099TKDE.22222:1722222
2
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning. C. D., Ng, A. Y., and Potts, C. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language processing, pp. 163 1634-, 2013b.
Wang. K. R., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the wild: a circuit for indi- rect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations. 2023b. URL https:::::/ / /ttp///eeeee.eeeeee/oeee ?i d NpSVNSSSN
5
2
Subramanian, A., Pruthi, D., Jhamtani, H., Berg- Kirkpatrick, T., and Hovy, E. Spine: Sparse interpretable neural embeddings. In Proceedings of the AAAI confer- ence on artificial intelligence, volume 32, 2018. 2
Wang. T., Sridhar, R., Yang, D., and Wang, X. Identify- ing and mitigating spurious correlations for improving robustness in nlp models. In Findings of the Associa- tion for Computational Linguistics: NAACL 2022, pp. 1719-1729, 2022. B 18
Sun, Z., Xiao, Y., Li, J., Ji, Y., Chen, W., and Zhang, M. Exploring and mitigating shortcut learning for generative large language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 1883-6893,.2 2024. 2
Wang. Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Align- ing language models with self-generated instructions. In Rogers, A., GoyddGrrbbe, J., and Okazaki, N. (eds.), Pro- ceedings of the 61 Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 13484-13508, Toronto, Canada, July 2023c. Associa- tion for Computational Linguistics. doi: 10.18653/v1/20 long. URL wttttss:/aclaanthlllooo 2122223..c111o00 754
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpaca: A strong, replicable instruction- foolowiinn model. Stanford Center for Research on Foundation Models. https://rrfm. stanford. edw/2233/3/1
html, 366)7,, 2023. 3
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A,, Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 2023.
Wang, Y. O., Chung, Y., Wu, C. H., and De la Torre, F. Domain gap embeddings for generative dataset augmen- tation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 28684-28694, June 2024. 2
Turner, A. M.. Thiergart, L., Udell, D., Leech, G., Mini, U., and MacDiarmid, M. Activation addition: Steering language models without optimization. CoRR, 2023.
Wang, Z. and Culotta, A. Identifying spurious correlations for robust text classification. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2020, pp. 3431-3444 2020. 18
Udomcharoenchaiktt, C. Ponwitayarat, W., Pay- oungkhamdee, P., Masuk, K., Buaphet, W., Chuang- suwanich, E., and Nutanong, S. Mitigating spurious correlation in natural language understanding with counterfactual inference. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11308-11321 2022. 2
Wang. Z. and Culotta, A. Robustness spurious corre- lations in text classification via automatically generated counterfactuals, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 14024-14031, 2021.
Vapnik, N., Vapnik, V., et al. Statistical learning theory. wiley New York, 1998.
Williams, A., Nangia, N., and Bowman, S. A broad- coverage challenge corpus for sentence understanding
14

---

Focus on This, Not That! Steering LLMs with Adaptive Feature Specification
through inference. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies. Volume 1 (Long Papers), pp. 1112-1122, 2018.
Yun, Z., Chen, Y., Olshausen, B., and LeCun, Y. Trans- former visualization via dictionary learning: contextual- ized embedding as a linear superposition of transformer factors. In Agirre, E.., Apidianaki, M., and Vulić, I. (eds.), Proceedings of Deep Learning Inside Out (Deel1O): The 2nd Workshop on Knowledge Extraction and Integra- tion for Deep Leurning Architectures, pp. 1-10, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18653//1120221.deee wwwp://000eeeeeel-oooeooo 1.1. URL htt sttpsi
5 28
Wolf, T., Debut, L.,, Sanh, V., Chaumond, J., Delangue, C.,, Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Transformers: Sfate-of-the-art natural language processing, In Proceedings of the 2020 conference on em- pirical methods in natural language processing: system demonstrations, pp. 38-45, 2020. 6
//aclaatthology.oorr////0 wwwp:://22222eeeeeee.e io 1
Zeng, Y., Yang, Y., Zhou, A., Tan, J. Z., Tu, Y., Mai, Y., Klyman, K., Pan, M., Jia, R., Song, D., et al. Air-bench 2024: A safety benchmark based on risk categories from regulations and policies. CoRR, 2024. 18
Wu, Z.,, Arora, A., Geiger, A., Wang, 7., Huang, J., Jurafsky, D., Manning, C. D., and Potts, C. Axbench: Steering Ilms? even simple baselines outperform sparse autoen- coders.
arXiv preprint arXiv:2501.17144 2025. 6
Zhan, Q., Fang, R., Bindu, R., Gupta, A., Hashimoto, T. and Kang, D. Removing RLHF protections in GPT-4 via fine-tuning. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pp. 681-687, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 10.88653w1/220444madd--hhhr URL htttt::tt///ccc Larthology.orr/22222 www.enaaaaa.............aaaa.caa short. 59/ 2
Xiao, K., Engstrom, L.., Ilyas, A., and Madry, A. Noise or signal: The role of image backgrounds in object recogni- tion. In International Conference on Learning Represen- tations, 2021. 2
Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., and Jiang, D. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 12244, 2023. 3
Yang, Q. A., Yang, B., Zhang, B.,, Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J.. Lin, J. Dang, K., Lu, K., Bao, K., Yang. K. Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q. Men, R., Lin, R., Li, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang. Y.-C., Wan, Y., Liu, Y. Cui, 7., Zhang, Z., Qiu, Z Quan. S., and Wang, Z. Qwen2.5 technical report. ArXiv, sema 9
Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., Wang, S., Li, J., Hu, R., Zhang, T., Wu, F,, et al. Instruction tuning for large language models: A survey, arXiv preprint 23X.::2308 2023.
Zhang, X., Li, M., and Wu, J. Conditional language learning with context. In Proceedings of the 41st International Conference on Machine Learning, pp. 59247-59263 2024. 2
abs/2412.15115, 2024a. URL URL
nt nticacho www.hollr..om wwwpchhlaar.rg//oorroe http:arrorrgoooppa00000005555550
Zheng, G., Ye, W., and Zhang, A. Learning robust classi-
fiers with self-guided spurious correlation mitigation. In Proceedings of the Thirty-Thin International Joint Con- ference on Artificial Intelligence, pp. 5599-5607, 2024. 18
Yang, X., Wang, X., Zhang, Q., Petzold, I.. R., Wang, W. Y.,
Zhao, X., and Lin, D. Shadow alignment: The ease of subverting safely-aligned language models. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models, 2024b. URL https: http///ppppeeeeeeeeeeeeeee net /forum 9gymw 6T 900
Zhou, C., Liu, Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L.,, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006-5500
Ye, W., Zheng, G., Cao, X., Ma, Y., Hu, X., and Zhang, A. Spurious correlations in machine learning: A survey. arXiv preprint arXiv:2402.127111 2024. 2
2023.
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, I.,, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024a. 3
Yuan, J., Pinto, F., Davies, A., and Torr, P. Not just pretty pictures: Toward interventional data augmentation us- ing text-to-image generators. In Forty-first International Conference on Machine Learning, 2024. URL httss: // //openreview net forum frteerummi-=b8888e00 9m
Zhou, Y., Xu, P., Liu, X., An, B., Ai, W., and Huang, F. Explore spurious correlations at the concept level in lan- guage models for text classification. In Proceedings of
15

---

Focus on This, Not That! Steering LLMs with Adaptive Feature Specification
the 62nd Annual Meeting of the Association for Compula-
tional Linguistics (Volume I: Long Papers), pp.
2024b. 218
Zou, A., Phan, L.,, Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-.,, al. Representation engineering: A top-down approach ai transparency. CoRR, 2023. B
16

---

Focus This, Not That! Steering LLMs with Adaptive Feature Specification
A
Author Contributions
Tom A. Lamb:
Idea: Turned the initial project ideas and directions into the concrete methodology described in Section 3
Experiments and Methods: Designed and carried out all experiments in the paper. These are described in Figure 3 Appendix and Section [
Dataset Creation:
Designed and carried out the synthesis of the SS and SMNLI dataset presented in Appendix
and Section[4.1
Analysis: Led the analysis of all empirical results.
Paper Writing: Led writing of all sections (main paper and appendix), including both the initial drafts of the paper and the final version of the paper presented here; and led literature review on spurious feature learning, instruction-tunigg and helped with the aligning LLMs section of Section 2
Adam Davies:
Idea: Co-conceived the research problem and core approach of FIT (with Francesco Pinto).
Supervision: Led advising of methodology and experimental design.
Analysis: Advised and assisted with analysis of all empirical results.
Paper Writing: Co-wrote drafts Abstract, Section T and Section 2 led literature review for mechanistic interpretability. aligning LLMs, and latent steering in Section and contributed to literature review for spurious feature learning and instruction tuning; and helped edit and revise all sections of the paper.
Alasdair Paren:
Supervision: Co-advised methodology and experimental design with particular focus on the simplicity bias and shorteut learning.
Paper Writing: Provided feedback on drafts.
Philip Torr:
Supervision: Provided general advice on initial project direction; helped secure compute resources for the experiments conducted in this paper.
Paper Writing: Provided draft feedback on the initial sections of the paper.
Francesco Pinto:
o
Idea: Co-conceived the research problem and core approach of FIT (with Adam Davies).
Supervision: Co-advised methodology and experimental design.
o Paper Writing: Provided feedback on earlier versions of the drafts of the paper.
17

---

Focus on This, Not That! Steering LLMs with Adaptive Feature Specification
B Limitations and Future Work
Requirement for Annotated Spurious Features. While FIT relies on prior identification of spurious features and their focus labels, this requirement does not limit its practical applicability. Instead, it reflects standard industry and research practices for constructing transparent and reliable models. Below, we clarify how FIT remains adaptive and versatile even when feature annotation is partial evolving:
Aligument with Established Practices: FIT's s reliance on pre-identified spurious features aligns with widely adopted industry and research norms (OpenAT 2024; Microsoft 2020). Identifying potential spurious features and confounders in datasets is a foundational step in achieving robust machine learning systems. This process ensures that both training and validation phases are informed by an understanding of data correlations, minimising the risk of deploying models with unknown biases.
Regulatory and Ethical Expectations: Regulatory frameworks and ethical guidelines increasingly require the explicit identification and mitigation of problematic features (of the European Parliament| 2016). Corresponding initiatives aim to define and enforce measurable categories of "violating behaviour" in Al models. By providing a mechanism to steer model behaviour based on these identified features, FIT effectively complements efforts to promote fair and transparent
predictions Guuldiaan al. 2024 Zeng et al. 2024).
Post- Despoyeplymnnt Mitigation: Despite careful pre-deployment analysis, spurious features or correlations may only become apparent once model is in active use. FIT accommodates this by allowing developers to incorporate newly identified spurious features via updated focus instructions, enabling rapid iterative refinement without retraining from scratch. This adaptability ensures continuous improvement, even in highly dynamic environments.
FIT's Versatility Without Exhaustive Pre-Identification: Crucially, FIT does not require an exhaustive list of spurious features to be effective. For instance, user can provide focus instructions such as "focus on casual" without enumerating every possible irrelevant attribute in the dataset. This flexibility expands FIT's applicability scenarios where feature annotation is incomplete or ongoing.
Compatibility with Automated Spurious Feature Identification: FIT also works seamlessly with automated methods for detecting spurious features (Wang& & Culotta| 2020 Wang et al. 2022 Zhou et al. 2024b; Zheng et al. 2024). Whether spurious features are labelled manually or derived from automated detection, they can be harnessed by FIT's focus instructions at inference time. This compatibility enables a comprehensive approach to managing known issues and responding to newly uncovered features they arise.
In summary, annotating spurious features beforehand is not a strict limitation. FIT can be flexibly applied, allowing model behaviour to evolve in tandem with new feature discoveries or changing requirements, making it a broadly applicable technique for steering model outputs based on both prior knowledge and ongoing insights.
Scope of Experiments and Extensions Open-Ended Tasks. Our experiments primarily focus on classification and multiple-choice QA datasets due to the cost and challenges associated with curating high-quality datasets for open-ended NLG tasks. However, this reflects a pragmatic prioritisation of introducing novel methodology over exhaustive data collection, rather than a limitation of FIT itself. Whilst we provide positive evidence through our BBQ-NLG ablation in Section 5. extending FIT to open-ended tasks such as summarisation or translation, remains an key direction for future research, as does exploring its ability to generalise across diverse task categories using setups similar to FLAN (Longpre) ct al. 2023).
Overlapping Features and Ambiguities. Additionally, our evaluation on the HANS dataset Appendix K revealeed challenges when addressing overlapping or less distinnvtee features. While FIT demonstrated strong performance in generalising and steering models based on identified features, overlapping heuristics can introduce ambiguity, highlighting the need for further refinements in handling such cases. Despite these limitations, FIT represents a promising foundation for enabling more robust, fair, and controllable LLMs across range of tasks.
Handling Multiple Features. While our current experiments provide initial evidence that FIT can handle combined instructions, specifying both a feature to focus on and a feature to ignore, we recognise the importance of scaling this capability. An important direction for future research is extending FIT manage instructions involving multiple features simultancously, including cases where models are required to focus ignore several attributes in tandem.
18

---

Focus on This, Not That! Steering LLMs with Adaptive Feature Specification
C FIT Focus Instructions and Prompt Templates
Prompt Templates. In Figure T0 we provide the prompt templates that we use for FIT training and for general evaluation across all methods. Note, that we drop the feature considerations section and focus instruction from the prompt form the default prompt 0 introduced in Equation
we show the focus instruction formats for the different focus instructions introduced in Equation 2). defining Tocus throughout our experiments.
Focus Instructions. In Figure
9
Focus Instructions
focus
Ignore Instructions iguore(Fi)
your attention solely to F:.
Completely rule out Fi from your reasoning, Disregard any influence of F. in your prediction.
Concentrate your reesoning
Make Fi the central factor in your decision.
Treat as irreevant to your decision-making process.
Base your judgment exclusively on Ft.
Exclude F. entirely from your evaluation.
yakingyour prediction. Use F; as the key input for your evaluation.
Pay sttention only to
not E; play anyrole your ssseeement. Intervene to pravent F. from affecting your prediction. Ensure that has no bearing your final decision.
Focus entirely on and ignore other sspects.
Rely exclusively Fi reach your conclusion.
Black E, from contributing your reasoning.
Considor only F; and disrogard all else.
Nogate the impact of F. in your prodiction. Ruling crucial-do not it your decision.
Rj primery basis your decision.
Focus Ignore Instructions ignore(F,) ignore(F,)
Focus specifically on Fi. Disregard My inyour decision-making process.
Base your prodiction solely on Excluud Fj.
Direct all your attention F, Block out from prediction.
Consider only F, your reasoning, Rule out F, in your decision-makknng Priontize F;, Completaly ignore not consider you dacision-maiinnnrooee process. Focus exclusivsly on F,.
prodiction.
Ignore any influence of F2. Concentrate on F; in your prediction.
Disregard entirely, Base your analysis solely
Rulc out P. in your prediction. Shift your focus R.. Do not pey sttention to yourur decision-making process. Rely only Fi.
Figure 9. Focus Instructions. Focus instructions that are used for focussing and ignoring features F, E J during FIT training and for general evaluation.
SS
SMNLI
CONSTIUUTIOO
You are sanguage model penforeing sentinent ana lysts on binnrry detaset, naking predict ines froe the labels [negative, lee positive]. Kabe your prediction based on the relevant features described below, focusing on the spec powiifin ifidd features and ignorin ing these deased irrulevant. i. For the input below, outpot either regative or positive UNLY for your prediction the trput's Label.
CUNSTRUCT100
You are a language model answertng quesIION based on provided CONTEXT. Frovide de short and concise answee to the question. CEND OF INSTRUCTION>
oF TNSTRUCTIONS
(Focus Instruction)
CREATURE CORSIDERAT 1085>
<FEATURE CONSIDERATION
<END FEATURE CONSTDERATEOSS
<END FEATURE CONSTDERATIONS>
(Focus Instruction)
Question: (question)
Context: (Contect)
Input: (Input)
Answir: (Nasponss)
Outputt: (Response)
BBQ MCQA
BBQ NLG
Language answering questior choosing answer three options: 1, 3, Each option EXACTLY INSTRUCTION>
CINSTRUCTION
You are Lang-age model penforming gg natural language inference (HLE) on detasel with thre labels [ental Irent, nestrall contradiccion)] J. For the inputs consisting of preatse and typothesis below, make your prodiction based on the relevant reatures descrtbed, focusing on the spectfied features and igporing those deseed irrelevant. output elther entailaunt, neutral, or contradiction OuLY for your prediction of the trput" 's lasel. <ENo oF DNSTRICTION
output
(3, reprosenting you select.
CFEATURE CONSIDERATIONS>
(Eotus Instructian)
CEND FEATURE CONSIDERATIOIS>
<FEATURE CONS ZONS>
Context: (Context)
(Focus Instruction)
Question: (Question)
<END FEATURE CONSTDERATIONS>
Answen options:
Prenise: (Premise)
(Answer option (2) (Anseer option (Arseer option
typothesiis: (thpotthsis)
Output:: (Response)
Answen: (Response)
Figure 10. FIT Prompt Templates, Prompt templates used for FIT training and evaluation across all four datasets that we investigate over.
19

---

Focus This, Not That! Steering LLMs with Adaptive Feature Specification
D FIT Training, Optimisation and Evaluation
D.1 FIT Training and Optimisation
FT Optimisation. Algorithm ! gives precise details on how implement FIT in practice when performing ERM of model using the FIT training objective given in Equation (3) given training set.
Algorithm Algorithm for Focus Instruction Tuning (FIT) Training Procedure to Optimise Equation (3)
1: Input: Dataset D {< y4) ) The feature set F, focus instructions Ioccss instruction I, model parameters 8, batch
size B, number of epochs E, step size V), and focus label mapping Yfocus Yfoccu focus V, 8).
2: Initialise: Model parameters 0, optimiser.
3: for epoch = 1 to E do
4: for umiii-batch {<>","") _ from D do
5: for each (ar", .0) in the mini-batch do
6:
Identify spurious feature value s in 2².
7: Sample focus instruction Hoocs Plaan"
8: Compute ylecux
ylocus ( Hflccss s², "/v").
9:
end for
10:
Compute average loss given through empirical estimator of the loss defined in Equation 3 over the batch:
1
{(O)
log po po (P II, I. x²).
B
11: Update model parameters 9 using optimiser:
1 -- 0 - Vol(())
12: end for
13:
end for
14:
Output: Optimised model parameters 0
FT Training Settings. We use LoRA (Hu et al. 2021 for parameter-efficint fine-tuning. We target the query and value projection matrices within each LLM and use LoRA 16 and a 32 across models.
We implement early stopping on a held-out validation set based on the cross cross-entropy loss over focus labels yrocus correspond- ing to randomly sampled focus instructions this matches the context in which the models will be evaluated. We obtain this set by splitting our training sets in a 90/10% ratio for training and validation splits respectively. We use patience of 4 validation evaluation steps, which occur after fixed number of steps.
During training, we define p(Irocu)) by placing small probability (in our experiments, 0.05) on the empty focus instruction 0. We then uniformly distribute the remaining probability mass over the non-empty focus instructions.
Choice of Pspurious During Training. In our synthetic experiments described in Figure 3 and Appendix we set up controlled environment by imposing two independence conditions: Y .L. S and Ys C. These ensure that (i) the ground-truth label cannot be predicted using the spurious feature S, and (ii) the spurious label cannot be predicted using the core feature C. By removing direct correlations between these features and labels, the model learns to focus the specified feature, without being influenced by another feature, avoiding any potential shortcuts that could be exploited if these conditions did not hold.
Independence Y IL S: This condition prevents the model from leveraging spurious feature predict ground-truit truth label Y. With predictive signal from S to Y, the model must rely exclusively the core feature C for accurate label predictions. This design choice safeguards the model from overfitting to spurious correlations, thereby maintaining robust performance under distribution shifts. Moreover, removing any inherent relationship between S and Y ensures that for focus instruction intending for the model to utilise the core feature C only during inference, the model cannot exploit a potential shorteut using S: must utilise the core feature alone for prediction in this scenario. This enforces prediction only through the specified feature indicated through the focus instruction passed to the model.
20

---

Focus on This, Not That! Steering LLMs with Adaptive Feature Specification
Independence Ys IL C: This condition serves complementary role in preventing the model from exploiting the core feature C' when predicting spurious labels Ys- By ensuring C carries information about Ys, the model cannot the true task feature C' as shortcut for spurious-label predictions; it must again learn to only use the specified feature within the passed focus instructions for making predictions.
While these conditions represent an ideal setting, they are not strictly necessary for FIT to work in practice. Indeed, real-world data rarely satisfies such independence assumptions. We illustrate the robustness of the method in more realistic scenarios through our BBQ experiments in Section 4.2 where correlations between Y and S or between C' and Ys may exist, as no subsampling or dataset manipulations have been made to enforce these conditions. By examining both controlled environments and more natural datasets, we demonstrate the robustness of FIT.
To achieve the aforementioned independence assumptions in our synthetic SS and SMNLI datasets, set Pepurious = 1/N, where NN is the number of class labels. Additionally, we enforce balanced label distribution in the training set to eliminate any indirect biases that could correlate S with Y. As shown in Appendix and Appendix [F) these conditions are sufficient to guarantee Y IL S' in the training data, enabling the model to effectively learn steerable behaviour from focus instructions.
D.2 Evaluation
Generation Settings. We generate responses from models using constrained beam-decoding (Anderson et al. (017) with 4 beams. This ensures that the answer labels for each classification task that we investigate appear in the model's output. We limit the maximum number of newly generated tokens to be 5 to stop any unnecessary text given after the model's initial classification prediction.
Computing the Focus Accuracy Metric. We report the focus accuracy Atocus of generations when evaluating models. As are guaranteed to include the task labels within the model's response through constrained decoding, and have reduced the maximum number of tokens that model can generate at inference-tings time, simply check to see if the focus label, yfocus is within the model's response or not in order to determine if the model's response is correct or not.
D.3
Dataset Sizes
The sizes of each of the splits of the ss, SMNLI and BBQ datasets are given in Table $
Table 3. Dataset Sizes. Dataset split sizes for SS, SMNLI and BBQ datasets.
SS
SMNLI
BBQ
Training
5296
7200
16700
Validation
1324
1800
1590
Test 1818
900
2352
D.4 Complete Set of Baselines
In addition the baselines that present in the main paper, namely few-shot and SFT(y)ocus), we include additional baselines to further supplement these results. We give the complete list of baselines below:
Zero-Shot Baseline. We include zero-shot inference baseline using the original pre-trained models without additional fine-tuning on any dataset. No in-context examples are used at inference time. The model is tested on the full set of focus instructions prompts detailed in Equation (2).
Few-Shot Baseline. This second baseline compares FIT training to few-shot inference, using the original pre-trained models without additional fine-tuning on our spurious datasets. Specifically, we use 4 in-context examples across all datasets. For the in-context examples, concatenate multiple examples after the other. Each in-context example contains the same focus instruction as the test example for which they serve as context. The model is tested on the full set of focus instructions prompts detailed in Equation (2).
SFT( Baseline. We implement an SFT baseline that follows the same training procedure as FIT, except during training, we exclude any focus instructions from the input prompts while still training on the focus labels. This provides a fair comparison with FIT, as models are trained on the same input text-label pairs. The rest of the training setup, including
21

---

Focus This, Not That! Steering LLMs with Adaptive Feature Specification
hyperparameters and early stopping, remains identical the FIT training setup. The model is tested the full of focus instructions prompts detailed in Equation (2)
SFT()) Baseline. We implement a vanilla SFT baseline that simply trains model using SFT on inputs and their ground truth labels (as opposed to focus labels in the (FT(y) baseline). During training, only standard IT prompts are used corresponding to the default prompt 0, with no additional focus instructions included. The rest of the training setup, including hyperparameters and early stopping, remains identical to the FIT training setup. The model is tested on the full set focus instructions prompts detailed in Equation (2)
We give the full set of results for all datasets and models across the complete set of baselines listed above in Figure 12 Figure 14 and Figure 15
22

---

Focus on This, Not That! Steering LLMs with Adaptive Feature Specification
E
Spurious Sentiment (SS) Dataset
We take a pre-existing dataset, in this case SST-3 (Socher et al. 201 Ba). and modify it in order to include a known spurious feature and create a spurious binary sentiment analysis dataset that we call the spurious sentiment (SS) dataset.
E.1 Data mata-generating process (DGP)
We frame our DGP using a graphical model to describe the synthetic dataset that we create. We follow a similar model to that described (Arjovsky et al. 2019), specifically the model used for generating their coloured MNIST dataset. We use the following variables within our graphical model:
Crur underlying sentiment, the core feature within this task, sampled from the original dataset.
Š -proposed spurious feature sample, here this the presence of the keywords "Pineapple" or "Bayesian". We represent this binary categorical variable with Val(S) (Pineapple, Bayesian). We note that, this restricts to consider only one keyword appearing in a text at any given time.
S the final included spurious feature that is naturally inserted using a LLM into the final SS dataset example X. The feature S is randomly flipped version of the proposed spurious feature S. So here Val(S) {Pineapple, Bayesian) also.
X is sampled example from the original SST-5 dataset.
original example X but naturally augmented include the spurious feature S, without changing the underlying sentiment of the example.
Y final label for element X.
The graphical model describing the DGP of the SS dataset is given in Figure 1la This admits a functional representation in the form:
C fc(Uc);
(6)
X [(C,Ux);
(7)
S fg(C,Ug);
(8)
fs(3, Us):
(9)
X fx(8, 8,Ux);
(10)
Y fv(C,Uv).
(11)
where Uc) are variables introducing sources of randomness into the generating process. More explicitly, we consider the following set of equations, where D denotes the underlying dataset that we are manipulating:
C ~ Ber(pc), where Pc Pc());
(12)
x PD(-
(13)
Pineapple
if =
S Bayesian
(14)
if
Us Ber(Pagurioos::
(15)
S
if
S Val($) \ Š
(16)
if Us 0
X = LLM(X, S);
(17)
y == C,
(18)
Here, Ber denotes a Bernoulli distribution, and the function LLM denotes a LLM that naturally injects spurious feature S into example without altering its underlying sentiment. The variable PC governs the distribution of sentiment labels in the original binarised SST-5 dataset. Moreover, P-(CCC denotes the conditional dataset distribution of the different input texts given (here we assume that we are just uniformly sampling text with the given sentiment C).
23

---

Focus on This, Not That! Steering LLMs with Adaptive Feature Specification
C
S
x
S
S
X
Y
X
(b) SS Causal Graph. Causal graph showing showing the spu- rious correlation present between the keyword feature S and the label Y for examples in the SS dataset, induced through the described data augmentation process.
(a) SS DGP. Graphical model showing the data generating process for modifying examples from the SS dataset to intro- duce spurious keyword feature, S.
Figure 11. SS DGP and Causal Graph. The DGP and associated causal graph describing the generation of dataset examples and showing the causal structure within examples.
Through the above DGP, we introduce a new spurious feature within the dataset, S. Recalling that S Pineapple and Bayesian correspond the insertion of the keywords Pineapple and Bayesian respectively, we introduce the following spurious correlations between the final included feature values of S and label Y:
(a) The presence of the word Pincapple in the text X, that is S = Pineapple, is spuriously correlated with the label 0 (negative sentiment). Therefore, the spurious label associated with S Pineapple is given as UPineapplc 0.
(b) The presence of the word Bayesian in the text X, that is S Bayesian, is spuriously correlated with the label 1 (positive sentiment). Therefore, the spurious label associated with S Bayesian is given 1/Rayesian = 1.
The sentiment feature C' still remains core within the augmented SS dataset, fully predicting the label Y for each dataset example.
Causal Graph from this DGP. The above DGP, through the introduction of spurious feature S, induces causal graph that describes the spurious correlation between spurious feature S and the label Y in terms of the additional variables X and C only. The causal graph, shown in Figure Ilb is similar the style-content decomposition described in (Kaddour et al.) 2022).
Showing that Pspurious Corresponds to the Predictivity of S. We now prove that Psparious gives the co-occurrence rate/predictivity between the label Y and the spurious feature S. and is well-defined notation in the sense that it corresponds to Definition 2 so that Pspurious = IP(Y = ys|S = s). where ys is the label that spurious feature value s is spuriously correlated with. Note that this will hold constant irrespective of the feature value of S. We begin with the following proposition.
Proposition E.1. From the DGP described above, we have that
PC Psparious
if = Bayesian,
P(Y ys -S= Pspurious + (1 PC) (1 Paparious)
(1 Pc) Pspurious
(19)
if Pineapple.
Pc) Pspurtous + PC (1 Pepurious)
Proof. Using the partition theorem, we have that
P(S = 5) = P(S = --s=s1551 = = + s P(Š +
(20)
=B(U==1)
=P(U,=0)
P(U, = 1)P(5 + P(U, = O)P(( +
(21)
Pspurious IP(5 8) + (1 Pspurious) PPS + s)
(22)
PC Pspurious + (1- - pc) (1 Psputious)
if Bayesian,
-
(23)
(1 PC) Psparious + PC (1 - Papurious)
if s = Pineapple.
24

---

Focus This, Not That! Steering LLMs with Adaptive Feature Specification
where we have used that P(U, 1)
Pspurious and that the value of S depends solely the value of C.
Using this alongside Bayes' rule gives
P(Y V/ S = P(S Y = Va) = Us)
(24)
P(S = 8)
P(S = 8 C = ys P(Y = 41))
(25)
P(S 8)
P(S 5 PY = vs)
(26)
PPS = s) Pc Pepurious
if Bayesian,
PC Pspurious + (1 Pc) (1 Psparious)
(27)
Pc) Pspurious
if Pincapple.
PCp Poparious + Pc Psparious)
which gives the result.
Corollary E.1. From the DGP described above, assuming that we have a balanced label distribution, that is PC = 1/2, we have that
P(Y = ys 1 =5= = Popurious
(28)
for all spurious feature values s.
Proof. This follows immediately from he previous proposition, Proposition E.1 with PC 1/2.
Within our experiments on the SS datsaet, we always force the label distribution to be balanced, that pC 1/2, and assume that within each dataset split, Pspurious is the same rate for all spurious feature values.
Data Generation Methodology. We alamm-3.11-70--1 Instruct to generate modifications X of original dataset examples X to create new text which include the new keyword features (presence of the keywords "Bayesian" and "Pineapple". The prompt we use for generation when modifying examples to include spurious features is give as:
Data Augmentation Prompt
You are a language model designed to modify piece of text to include an additional feature a simple, natural way while keeping your output similar possible to the original text.
Features
pineapple: Include the word 'pineapple".
Bayesian: Include the word 'Bayesian'.
Instructions
Ensure the output is grammatically correct.
Keep the output as similar as possible the original text.
Make the minimal number of modifications and add the fewest new tokens possible to satisfy the chosen feature.
Do not change the sentiment of the original text.
Do not significantly alter the length of the output.
Incorporate the feature naturally within the original text that blends scamlessly with the text's context.
Do not only append additional clauses at the end of the text to include the feature. Inclusions should be case sensitive, c.g., include 'Bayesian' BUT NOT 'bayesian'.
Output
Only return the modified text, with additional explanations reasoning.
Should strictly follow feature description the set of instructions. Only include the one feature given; the other features SHOULD NOT be included even accidentally.
E.2
Independence Conditions During Training for FIT on SS.
As specified in Appendix D we would like to have that Y JL S and Ys JL C' during training so that models trained via FIT can effectively learn to leverage focus instructions to make predictions based on specified features. Here, Y, is the spurious label spuriously correlated to the random spurious feature value S. The results below give sufficient conditions for these independence conditions to be satisfied with respect to the DGP described above, and consequently form the conditions
25

---

Focus on This, Not That! Steering LLMs with Adaptive Feature Specification
that we impose on the SS training set for FIT training. We show that the key condition that we must impose in training for Y II S and Ys Il. is to set Pspurious 1/2.
Proposition E.2. Assuming the DGP described in above and assuming that Pspurious = 1/2, we have that Y TT S.
Proof. With Pspurious = 1/2, we have from the DGP given above, that Š 11 S. In particular, using the factorisation implied by the directed acyclic graph (DAG) corresponding to the DGP, we see that
P(Y S:
Σ
P(Y = C
y c=
c) $
(29)
cEVill))
=P(S=s) as S1.
Σ
P(Y y C C- -c $
c) = 8)
(30)
CE vVll))
P(S =)
E
P(Y= vy=ycc- c) C=c)
(31)
PPS = s) E
= c)
(32)
Va
= PPS = sY = y).
(33)
By definition, this shows that we have that Y 11. S, as required. Proposition E.3. Assuming the DGP described above, for Pspurious 1/2, then we have that Ys 1L C.
Proof: First note that Ys is a deterministic function of S, that is Ys f(S) for some function f Val(S) + {0. 1). Therefore, it is sufficient to show that C J. S. However, from the DGP above, we have that Y = C. From Proposition E.2 we already have that Y IL S, which implies that C' 1L S, which proves the claim.
E.3 Results
Figure 12 (a) shows the focus accuracy results of three LLMs on the SS dataset across all of the baseline methods described in Appendix [D.] and the FIT method. We see that across all focus instructions and all models, FIT shows significant improvement over the baselines, achieving very high focus accuracy across all focus instruction types and across all test sets with varying predictivity levels.
Table Complete Baselines FIT Focus Accuracies SS (f). For each method, we report the mean of focus-accuracy means (Asoe)) over the four test splits (Dia. Drigh, Daw and Drippe) + the between- split standard deviation of these means (i.e. how performance varies as predictivity changes) across repeats. Boldface indicates the best mean for each focus instruction type and model independently.
focus(C) focus(C") A ig)
ign
focus(S)
focus(S) A ignore(C)
Zero-shot
0.886=p.our
0.90944.004
0.899+..000
0.871+0.00
0.423+o
0.305
Mistral
Few-shot
0.893 003
0.90620.008
0.9044..000 006
0.793
0.559 168
0.634_0.116
SFT(y)
0.95110.005
0.950 10.004
0.952 10.005
10.004
0.445
0.447
SFT(Hocus))
0.751 1o.12
0.794 10.
10.042
0.879 10.094
0.506 248
0.519 1.244
FIT
0.953++0.004
0.954
0.955-10.004
0.9544+0.004
0.999
0.999
Zero-shot
0.500=0.000
0.500=0.000
0.500 000
0.500
0.506 000
0.506
Llama
Few-shot
0.674_0.008
0.808
0.630
0.508
0.491 047
0.502:20003
SFT(y)
0.952410.00 0.04
0.954:1..00 000
0.952 007
0.952
0.445
0.44410.222
SFT(
0.668=p.178
0.686
0.803
0.795
0.606 197
0.601
FIT
0.94920.002
0.95140.002
0.952
0.950
0.998_0.001
0.999
Zero-shot
0.420=0.012
0.58420.007
0.38110.000 1:0.00
0.355
0.199 105
0.129=20000
Vicuna
Few-shot
0.147
0.4594
0.533
+o.010
0.300
0.300
SFT(y)
0.955+0.0005
0.956
0.955
0.953
0.4464
0.445
SFT(Woous))
0.5
0.602+0.187
0.685440.13
0.671
01.766
0.660=o.118
FIT
0.950L..003
0.953
0.955-10.000
0.955
10.55.00.004
0.999 0.001
0.999 1.999900.000
26

---

Focus This, Not That! Steering LLMs with Adaptive Feature Specification
Mistral
Llama
Vicuna
(es(S) Ign(C)
Zero-Shot
Ign(C)
(es(C)
kan/C
(es
Tes(C)
(ER(C) (gn(S)
(CS((C)
(es(S)
(eS(C)
ign(S)
(gn(S)
kgn
(gn(5)
(ca(S)
Scs
tcs(S)
Few- Sew-Shot
Ign(Ca
tcs(C)
tcs(C)
(s((C)
%
Tos(S)
Tes(C)
(ca
[ca(C)
(os(S)
[cs[C)
(gn(S)
(gn(S)
(g(5))
(gn(S)
kan(S)
(gn(S)
(es(S)
Tes(S)
les(S)
SPT(
Ian(Cu
(es(C)
KInCC
(es(C)
(gn(C)
es((C)
Ng %
6 [Cc[C)
(os(5)
hes(C)
TcS[C)
(gn(S)
(g(5)
(gn(5)
kan(S)
(gn(5)
tcs(5)
SF
ign(Cg
tcs(C)
(c)
ign(C
ks(C)
(ests)
Tes(C)
(cs(S)
(ss(C)
(es(S)
(es(C)
(gn(S)
kan(S)
ign(S)
kgntS)
(gr(S)
FIT (ours)
Kes(S)
(es(C)
les(S)
(es(C)
les(S)
(es(C)
%
%6
(os(9)
(ca(S)
(es(C) Ign(S)
To
(gg((S)
(igg()))
(gn(5)
kgn(S)
Did
Dhigh
Diow
D
Bippod
Figure 12. Complete Baseline vs FIT Focus Accuracies SS (t). Figure giving the mean focus accuracies (Aroen)) of the additional baselines compared to the focus accuracy FIT on the dataset. maximum standard deviations FIT, (FT( SFT(y) and few-shot methods across models 2.17%, 14.8%, 0.65%, and 2.83% respectively. fes = focus, ign ignore.
Key Takeaways. High focus accuracy on SS indicates that FIT successfully steers model responses based on the feature which it is instructed to focus to not focus
27

---

Focus This, Not That! Steering LLMs with Adaptive Feature Specification
F
Spurious NLI dataset (SMNLI)
We generate a tertiary NLI dataset, SMNLI, with known spurious feature. do this subsampling from the original MNLI dataset (Williams al. 2018). This is NLI dataset with three labels: entailment (0), neutral (1) and contradiction (2), where data sampled from 6 underlying categories genres (govermment travel, and fiction, facetoface, nineeleven and verbatim). We aim induce spurious correlations between the underlying genres and labels.
F.1
Data Data-Generating Process (DGP).
We consider graphical model to describe the DGP examples within the SMNLI dataset. We use the following variables within DGP:
C NLI relationship between premise and hypothesis pair, the core feature within this task, sampled from the original MNLI dataset.
X example premise and hypothesis from the MNLI dataset.
S spurious feature present in the example X, here this the genre of the premise and hypothesis texts. This is categorical variable.
Y final label for example X.
Y
X
Figure SMNLI DGP. Graphical model showing the data generating process for subsampling examples from MNLI dataset
introduce spurious keyword feature S.
The graphical model described by the DGP for producing the SMNLI dataset is given in Figure 13 Once again, this graphical model can be represented functionally as:
fs(Uss)
(34)
C fo(S, Uc);
(35)
X ExUx;
(36)
Y == UvC,CC,v.)
(37)
More specifically, given the original dataset D that are sub-sampling from, the functions that use within the DGP for the SMNLI dataset are given by:
S ~ U(S);
(38)
Uc ~ Ber (er(Pspurions):
(39)
C Uc ys + +1 Uc) ((Va((C) \ {ush)
(40)
X C))))
(41)
Y= C.
(42)
Here, U(S) denotes uniform distribution over the set of genres S, and define Val(C) = {0, 1,2) corresponding to the possible NLI labels for this task. Furthermore, we define to be the NLI label that a particular value of S is spuriously correlated with by design. Moreover, PD(- is the conditional distribution over the dataset examples (premise-hypothesis pairs) that have NLI relationship C' and genre S.
We restrict the genres that we sample from to E {government, fiction, travel), a subset of the genres of the training set. When creating distribution shifted test set, we restrict the genres to S (facetoface, nineeleven, verbatim). The
28

---

Focus This, Not That! Steering LLMs with Adaptive Feature Specification
specific spurious relationships between genre values and their associated spurious labels Ms are chosen to be: Vslate 0:
Ugovermment 2: Vfiction 1; Yinavel
0; Vicetolace
2: Ynineeleven O: Dveerbatim
1. In this way generate spurious
correlations within the dataset through sub-sampling to induce spurious correlations between 5 and Y.
We show that the notion of Papurious in Equation (39) aligns with the notation in Definition the spurious feature value of 5.
2
and that this does not depend on
Proposition F.1. From the DGP described above, we have that
P(Y Us 5 = Popurious
(43)
for all spurious feature values s.
Proof. This is clear considering Equation
40 where Papurious influences the chance that we sample yss i.e. the label
spuriously correlated with feature value
F22 Independence Conditions During Training for FIT on SMNLI.
As specified in Appendix D we would like to have that Y TL S and Ys JL C' during training so that models trained via FIT can effectively learn to leverage focus instructions to make predictions based on specified features, where, again, Ys is the label spuriously associated to spurious feature value S. The results below give sufficient conditions for this to occur with respect to the DGP described in Figure 13 This boils down to setting Popurious = 1/3 during training.
Proposition F.2. Assuming the DGP described above and that Psparious = 1/3, we have that Y TL S.
Proof. Note that since Y C' in the DGP above, it suffices to show that C' Tl S. We have that for a given $ = that P(C = ys S = s) = Pspurious = 1/3 from Proposition F.1 Moreover, let C ~ U(Val(C)) \ {us}) denote the random variable sampled from uniform distribution over the remaining possible values of excluding y.- Then for either value of E Val(C) {ys) we have that
PCC = = =) P(Uc 0)P(C =
(44)
2 1
(45)
2
(46)
3
Therefore, have that C ~~ U(Val(C)). In particular, this then gives that C 1T S, which in turn implies that Y I1 S.
Proposition F.3. Assuming the DGP described above and that Paparious = 1/3, then we have that Ys Tl C.
Proof. Note that Ys is deterministic function of S, that Ys = f(S) for some function Val(S) Val(Y). Therefore, it suffices to show that S 1L C. The proof of this is given in Proposition F2
R33
Results.
We present full results comparing FIT on SMNLI against of the baselines described in Appendix D.4 in Figure Table for completeness.
29

---

Focus This, Not That! Steering LLMs with Adaptive Feature Specification
Mistral
Liama
Vicuna
foseS) lan(Ch
(ce ign(C
(es(S) ign(Cy
Zero-Sho Shot
(Ca
(os(C)
(os(S)
(EStEC) fgn(S)
TES(CC) fgn(S)
(cs(S)
Tes(C)
(g((5)
(gngS)
Ign(S)
(g((S)
(cs(S)
Sca(5)
fes(5)
Few- Shot
lon(C
(es
ign(C)
Tos(C)
ign(
(es(C)
%
96 (estC) ign(5)
9 [cs(C)
(cc(5)
(Cs Ign(s)
(g((5))
kgn(S)
(gn(S)
(g((S)
tcs/S)
tca(5)
fcs(5)
SFT(y)
(gn(C
(C)
tcs(C)
(es(C)
9
96 (US(CC) ign(S)
%
(cs(S)
fes(S)
(eR(C)
ign(S)
(gn(g)
kan(S)
(gn(S)
(g(()
SF (lecu)
TostS)
kantCh
(es
(cs
ign(C)
(es(S)
ign(Cy
% (ES(C)
(es(S)
(ES(C)
(es
(es(S)
Tes(C)
ign(S)
(gn(S)
(g((5)
Kgn(S)
Ign(S)
(g(())
(cs(5)
(ca(5)
FIT (ours)
lan/C
(e)(C)
iCn
(es(C)
ign(Cy
(e)(C)
96
(cs(5)
(C) Ign(S)
(es(5)
(Is(CC) ign(5)
fcs(5)
Tcs(C)
(gg(5))
(gn(S)
(gn(S)
gg(()
Dia
Phich
Dano
Dippoll
Figure 14, Complete Baseline vs FIT Focus Accuracies on SMNLI (f). Figure giving focus accuracics (Atocus) of the additional baselines compared to the focus accuracy of FIT the SMNLI dataset. The maximum standard deviations of FIT, (3Hocss. SFT(y) and few-shot methods across models are 6.47%, 7.98%, 1.15%, and 0.500% respectively. fes focus, ign = ignore.
Table 5. Complete Baselines vs. FIT Focus Accuracies on SMNLI (1), For each method, we report the mean of focus- accuracy means (Arcus) over the four test splits (Dia, Drigh. Dew and Daipped) the between-split standard deviation of these means (i.e. how performance varies as predictivity changes) across repeats. Boldface indicates the best mean for each focus instruction type and model independently.
focus(C)
focus(C) (C) (ignore(S)
ignore(S)
focus (S)
focus (8) A ignore(C))
Zero-shot
0. 0.142
0.71120.014
0.71110.013
0.71620.020
0.362 189
0.368=0.191
Mistral
Few-shot
0.716=p.020
0.72540.017
0. 0.722+a.017
0.687
0.370+a.178
0.386
SFT(y)
0.8904..000
0.87540.013
0.875+0.018
0.885
0.3344
0.332+p.278
(FT
0.732=p.101
0.727=0.007
0.725
0.726
0.54440..10
0.537=0.190
FIT
0.87820.005
0.875
0.87440.000
0.878
0.963-20.08
0.968-20.008
Zero-shot
0.679
40.077
0.682+4.011
0.682
0.370
0.386+p.151
Llama
Few-shot
0.593=p.012
0.641
0.630+a.010
0.636++0.009
0.351
0.379
SFT())
0.88040.00m
0.880
0.87940.000
0.879
0l.52
0.354=0.272
SFT(Woous))
0.70010.121
0.75910.0m
0.750 15.01...99
0.720
0.552
0.536
FIT
0.872.0.007
0.865110.008
0.861 10.000
0.86510.010
0.929
0.93 1:0000
Zero-shot
0.595=b.012
0.598
0.575+4..119
0.573
0.334
0.341
Vicuna
Few-shot
0.269
0..07200.07
0.251 1:0.019 019
0.301-20.013
0.195 080
0.133 0.000
SFT(y)
0.892-+0.003
0.890 0.002
0.890 0.890-10.004
0.891110.00 00.006
0.337 100 288
0.3412-0.228
SFT(yoeuu))
0.579L..00
0.575.0..00
0.573..0.000 055
0.585 10.065
0.502.0..02
0.496 1.966...000 0.0086
FIT
0.803=p.004
0.807+0.000
0.794+0.00s
0.781
0.89420.019
0.934410.010
30

---

Focus This, Not That! Steering LLMs with Adaptive Feature Specification
G
Additional BBQ Results
Figure
15 and Table d
we include the full comparisons of FIT against all baselines described in Appendix D.4 2.440 on BBQ
Mistrrll
Liama
Vicuna
lesg99)
Pero-Shot
RSSCC)
(es(C)
MeseC (gn
(an(S))
fanc5)
(gn(5)
fcskS)
fests)
fest5) lam(Ch
(s((C)
(ss(C)
Fow-
(gn255)
(gm(55
(gn(5)
kgn(5)
(csk5)
fest5)
fest5)
gr(C
ks/CC
(ss(C)
SPTY
Tesugg kgr(t9)
(gn25)
(gm(5)
(gh(S)
(omES)
fest5)
Test5)
SFT ficcus
gn(C)
(on(C))
fex(gg)
(9
(gntS)
(gnE))
(gn(S)
fests) ignoC3
fest5) lgr(C
FIT (ours)
(ox(C))
fexp9)
TEs/C)
Hea(C)
(50
(gh(5)
(gh5))
lon(S)
Figure 15. Complete Baseline vs FIT Focus Accuracies BBQ (f). Figure giving focus accuracies (Areem) of the additional baselines compared the focus accuracy FIT on the BBQ dataset. The maximum standard deviations of FIT, SFT(y)tcus), SFT and fow-shot methods across models are 4.07%, 10.7%, 2.22%, and 0.600% respectively. fes = focus, ign ignore.
Table 6. Complete Baseline vs FIT Focus Accuracies on BBQ (t). We report across all baselines the mean seen/unseen focus accuracies, with boldface indicating the best mean for each focus instruction type and model independently. The maximum standard deviations of FIT,
SFT (FT SFT(w) and few-shot methods across models are 4.07%, 10.7%. 2.22%. and 0.600% respectively. fes focus, ign ignore.
focus(C))
focus(C) (C) A ignore(S))
ignore(S)
focus(S)
focus (S) ign
Zero-shot
0.772/0.792
0.768/0.803
0.782/0.812
0.790/0.829
0.338/0.320
0.302/0.295
Mistral
Few-shot
0.775/0.798
0.767/0.795
0.769/0. 807
0.777/0.814
0.408/0.381
0.388/0.373
SFT(y)
0.999/0.982
1.000/0.983
1.000/0.982
1.000/0.981
0.248/0.249
0.248/0.249
SFT
0.858/0.874
0.844/0.866
0.848/0.87
0.847/0.c8 863
0.396/0.33
0.394/0.344
FIT
0.998/0.976
0.998/0.976
0.998/0.977
0.999/0.976
0.941/0.852
0.941/0.853
Zero-shot
0.534/0.673
0.478/0.597
1.478/0.593
0,508/0.627
0.533/0.421
0.527/0.428
Llama
Few-shot
0.643/0.696
0,686/0.743
0.724/0.77 774
0.642/0.720
0.496/0.429
0.492/0.433
SFT(y)
0.998/0.991
0.999/0.985
1.000/0.95 988
0.999/0.989
0.248/0.22
0,248/0.244
SFT(Hrocue)
0.836/0.912
0. 0.825/0.901
0.836/0.900 903
0.852/0.91 990
0.403/0.317
0.418/0.322
FIT
0.993/0.974
0.997/0.977
0.998/0.979
0.996/0.979
0.943/0.832
0.946/0.835
Zero-shot
0.444/0.495
0.495/0.567
0.505/0.53.
0.479/0.519
0.371/0.324
0.375/0.315
Vicuna
Few-shot
0.454/0.565
0.453/0.572
0.517/0.604
0.547/0.625
0.503/0.512
0.487/0.496
SFT(y)
0.975/0.991
0,959/0,988
0.957/0.990
0.974/0,990
0.245/0.252
0.243/0.251
SFT(Hocw
0.796/0.833
0.794/0.808
0.787/0. 806
0.796/0.809
0.444/0.392
0.456/0.39 394
FIT
0.983/0.979
0.982/0.975
0.981/0.972
0.984/0.973
0.966/0.835
0.966/0.837
31

---

Focus This, Not That! Steering LLMs with Adaptive Feature Specification
H BBQ-NLG
H.1 BBQ-NLG Experimental Setup
We follow the original BBQ dataset splits (including both training and held-out social- partitions) described in Section 4.2 For each example in the new BBQ-NIG dataset, we remove the predefined answer choices and require the model to generate fully verbalised responses, rather than selecting from a fixed set of three options as in the original BBQ dataset and outputting a categorical label.
To give the model additional capacity for this more challenging free-form generation task, we augment the usual LoRA targets (the key and value projection matrices) with adapters on the value-projection matrices as well. All LoRA modules trained with learning rate of 10-5, We train both the FIT and SFT (yrcuu) models for 10 epochs each. As strong comparison, we evaluate few-shot baseline that conditions the model 4 in-context examples that share the same focus instruction as the test instance.
To enable both rapid and cost-effective evaluation of correctness in our open-ended generation task, we employ an LLM- model to compare each generated response against its
based judge. Specifically, we use pre-trained Llama-3.1 associated ground-truth focus label and determine whether they are semantically equivalent. Manual checks confirm that the Llama-3.1-8B-Inst. judge reliably assesses semantic equivalence this setting where model generations and expected responses and generally short and concise.
H.2 Results
Figure 16 reports the focus-accuracy of each method across models. Although all approaches exhibit slight degradation on unseen feature combinations relative to the classification-stye BBQ results in Section 4.2 FIT remains generally overall on with the earlier numbers. Critically, FIT consistently outperforms both the SFT (Mocus and the strong 4-shot few-shot baseline that uses identical focus prompts at test time. These findings provide clear evidence that the FIT method can be effectively extended to open-ended NLG settings without sacrificing its steering capabilities.
Mistral
Liama
Vicuna
ksx
(is15)
aprtt
IgPACA
Pow-Shot
Text())
YeaeC1 kgrtSt
A kgnket
(gr1S)
(n:5)
(ours)
(groC
Teatt) Ign(9)
TeayC1 kgr(S)
(e525)
(g/())
(gn35)
Seon -0- Linsoon
Figure BBQ-NLG Focus Accuracies (f). Mean focus accuracy (Atocus) of baselines and FIT on the BBQ-NLG dataset. The maximum standard deviations of FIT, SFT and few-shot methods across models and Irceu are 2.45%, 6.35% and 0.377% respectively. fcs focus, ign ignore.
32

---

Focus on This, Not That! Steering LLMs with Adaptive Feature Specification
I Comparison of FIT Against Specific Debiasing Technique
FIT is a general framework designed to enable users to steer a model's behaviour based on specified features. This approach provides enhanced control over model outputs during inference, adding a critical layer of explainability and controllability to model predictions.
While understanding and mitigating biases or spurious correlations is valuable and natural application of FIT, it is not the sole objective. The broader goal of steerability includes addressing challenges in managing and aligning model behaviour across diverse contexts. For instance, maintaining controllability is crucial in addressing safety alignment fragility, which can emerge after fine-tuning (Bhattacharjee et al. (024). In such cases, the ability to adapt model responses to align with user specifications ensures safe and reliable deployment.
Experiment. To explore FIT's broader applicability, we compare its performance as a debiasing method against a well- known debiasing technique: the Product of Experts (PoE) method (Mahabadi et al.) 2020 PoE involves training bias model fB. which is trained exclusively bias features. This bias model mediates the training of the final model by combining their predictions through an elementwise product: (f())) O a (fB(#B)), where E DD for dataset D, and zB represents the biased feature of x.
We adapted this approach to our setting by training& a bias model on the stereotypical labels within the BBQ dataset. These labels correspond to group- struottttttt associations. For autoregressive models, we further modified the PoE method by extracting and normalising the logits of the first newly generated token position over the set of single tokens representing the answer options.
Results. The results of the debiasing experiment comparing FIT to the PoE method is shown in Figure 17 FIT performs equally as well as the PoE method as shown by comparing the default prompt accuracy (0) for the PoE models against the (ocus(C) results for the FIT models; both metrics correspond to causal accuracy for these prompt types. This indicates that FIT performs just as well as the PoE dedicated debiasing technique.
However, the PoE method requires training two separate models and does not provide steerability at test time as shown by the low focus accuracy focus(S). Indeed the model defaults to the ground truth label across all prompt types and does not change behaviour in the presence of different focus specifications. This highlights the flexibility of FIT, which not only debiases effectively but also enables additional controllability during inference.
Mistral
Liama
Vicuna
A (g)
(es(S) A ign(C))
fex(S) (gp(C)
Ces(C)
(C)
les (C)
PoE
6% (es(C) (g)(
%% [ca(C) (ign(S)
(cs(S)
(S)
Res(S)
Jest (C)
ign(S)
ign
ignt
ign (S)
(es(8) ign (C)
fes(S) A igan
(es(C)) Sen(S)JA ignn(C)
Cc)
(Cs(())
FIT (ours)
%% Ees(Cy (8))())
%
8.%%
(os(8)
Res(s)
CCs(C)) (gg(8))
(en(S)
fcs(C) ign(S)
Iga S))
kgn (9)
ign(S)
Sean
Linsean
Figure 17. Focus Accuracy of FIT Against PoE Debiasing Technique (1). Figure showing the focus accuracies (Attcus) of FIT (bottom row) and the dedicated debiasing technique, PoE (top row), on the BBQ dataset.
33

---

Focus This, Not That! Steering LLMs with Adaptive Feature Specification
Spurious HANS Dataset (SHANS)
We generate a binary NLI dataset, SHANS, with a known spurious feature. We do this by subsampling from the HANS dataset (McCoy al. 2019). This an NLI data set with two labels: entailment (0) and contradiction (I). This is an adversarial dataset designed to assess different NLI models' reliance on spurious heuristics rather than on the underlying relationship between the premise and the hypothesis when making NLI predictions. Specifically, the author's consider three major categories of heuristics: lexical overlap heuristic (assuming that a premise entails from words within the hypothesis) sub-sequence heuristic (assuming that the premise entails all any of its contiguous sub-sequences of words) and constituent heuristic (assuming that a premise entails a hypothesis that is any constituent within it's syntactic parse tree). We take each these as separate spurious features within our SHANS dataset for which we induce spurious correlations, as for the SMNLI dataset, through subsampling.
J.1 Datertnnggg Process (DGP).
We consider graphical model to describe the DGP of examples within the SHANS dataset. We use the following variables within our DGP:
C NLI relationship between a premise and hypothesis pair, the core feature within this task, sampled from the original HANS dataset.
Siex spurious feature, here the presence of a hypothesis entirely made from words from the premise. This is a binary categorical variable (present/not present).
Seab. spurious feature, here the presence of hypothesis that is contiguous subsequence of the premise. This is binary category feature (present/not present).
Sconst. spurious feature, here the presence of hypothesis that constiuenttsuubrree of the premise. Here we have binary variable (present/not present).
example from the HANS dataset.
Y final label for example X.
Y
Figure SHANS DGP. Graphical model showing the DGP subsampling examples from the SHANS dataset introduce new spurious features Siex. Suh. and Sccue which are encoded within the categorical spurious feature S which represents one of these three heuristics.
The graphical model described by the DGP for producing the S-HANS dataset is given Figure 18 Once again, this graphical model can be represented functionally
S fs(Us):
(47)
C (SS.
(48)
X (x(
(49)
Y = fvC,U)y)
(50)
where here we define to be a categorical feature over the set of variables indicating the presence of each of the three heuristies introduced above which we denote, through overloaded notation, by S {Slex., 8suh, Sconst ) More specifically,
34

---

Focus This, Not That! Steering LLMs with Adaptive Feature Specification
given the original HANS dataset D that sub-sampling from, the functions that we within the DGP for the SHANS
dataset are given by:
S U(S),
(51)
Uc Ber(Pspuricus);
(52)
C ~ Ucys + (1 - Uc) - yys):
(53)
X ~ po(.IC,5)
(54)
==C.
(55)
Here, U() is a uniform categorical distribution over the set of spurious features which effectively selects the presence of exactly one of the three spurious feature heuristics. We define ys to be the NLI label that a particular value of spuriously correlated with by design. Moreover, pp(-IC, S) the conditional distribution over the dataset examples (premise-hypothesis pairs) that have NLI relationship C and the presence of spurious heuristic S.
We restrict the genres to 8 E {slex., Sconst. } for heuristics seen during training. We then create additional test sets with spurious feature set restricted to {s J. which serves as an unseen spurious feature set to test generalisation.
We consider the presence of each feature to be separate binary spurious features. The specific spurious correlations between heuristics and labels Y are chosen to be: Vla.=1 0: VS... = 0: V/ 1. this way we generate spurious correlations within the dataset through sub-sampling to induce spurious correlations between the heuristies S and label Y.
J.2 Transferred Results from SMNLL
As we have effectively used the same DGP as for the SMNLI dataset described in Appendix F with the only change being the label set, all of the results that have proven for SMNLI, translate the SHANS dataset. In particular, have that Psparious aligns with the notation in Definition 2 and that we have Y 11. S and Yg II C under the assumptions of Popurious 1/2 within the training set.
35

---

Focus This, Not That! Steering LLMs with Adaptive Feature Specification
K FIT on SHANS
Here, we give the results performing FIT on the SHANS dataset.
K.1 Spurious HANS (SHANS) Dataset.
We generate binary NIJI dataset sub-sampled from HANS (McCoy et al. 2019); a dataset designed to challenge NLI models by exposing common heuristics they rely on, such as lexical overlap (whether the hypothesis shares many words with the premise), sub-sequence (whether the hypothesis a contiguous sub-sequence of the premise), and constituent (whether the hypothesis is a grammatical sub-structure of the premise). The presence of these heuristics are spuriously correlated with labels through sub-sampling of the presence of each of the heuristics from the original dataset. The degree of co-occurrence is governed by Psparious. which varies according to the test sets described in Section 3 We ensure that Pspurious is the same for all feature values within each dataset split. In particular, we set Pspurious to be 0.5, 0.5, 0.9, 0.25 and 0.9 (in this case with flipped spurious correlations) on D Dirain Dia. high Djow and DAipped respectively. We keep the subsequence heuristic as a held-out unseen feature during training for testing FIT generalisation.
K.2 Results.
Figure 19 shows the focus accuracy results of performing FIT on the SHANS dataset for the LJama-3. 1-8--..1118---nrrte model. As expected, on the seen features, FIT shows high focus accuracy across all focus instructions. However, for unseen features, we observe lower focus accuracy. This could be attributed to the overlapping nature of the heuristics in SHANS, which are often graded versions of each other with different levels of specificity. For instance, the sub-sequence heuristic can overlap with both lexical overlap and constituent heuristics (e.g., the example with Premise: "Before the actor slept, the senator ran" and Hypothesis: "The actor slept." satisfies all three heuristics). This overlap likely confuses the model during generalisation, as it struggles to distinguish between heuristics seen during training and those that were not. These results suggest potential limitation of FIT when dealing with features that are not sufficiently distinct or have significant overlap.
Seen Features
Unseen Features
fis(S)
fes(S) A igu
ign(C)
Jes
fos C))
(S)(S)
ics
(es(S)
fos(C) A ign(()
A ign(S)
ign(S)
ign(S)
Das
Drigk -
Dam
Drippeo
Figure 19. SHANS Focus Accuracies (f). Focus accuracy (Ascus) of Llama-3.1 1-8--- after FIT on the SHANS test datasets containing either the seen unseen spurious features during training. Here, C' refers the core feature (logical relationship between premise and hypothesis) and S the spurious feature (heuristic used).
36


Reviews:
[{"review_id": "LNveWu3tzl", "reviewer": "Reviewer_ivne", "summary": "This work presents a training method to improve model steerability w.r.t specific features. The main idea is to add instructions during training about what to focus on and what to ignore. The trained model is evaluated on a modified SST dataset, a modified MNLI dataset, and BBQ, and it demonstrates significantly better steerability in terms of following the instruction to avoid spurious features (even unseen ones in the case of BBQ).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments regarding our introduction, specifically noting the “significantly better steerability” and overall “effectiveness” of our method. We hope that this rebuttal response addresses your additional comments regarding our paper.\n\n## P1: Showing Further Generalisability of Our Method\nWe agree that extending beyond classification and MC-QA tasks is highly beneficial for demonstrating that our method can generalise to more complex settings, thereby enhancing the overall impact of our work. To build on our previous results, we have extended FIT to operate in an NLG setting, an environment that poses additional challenges compared to the classification scenarios considered thus far.\n\n**Extension to NLG Setting:**  We adapt our BBQ experiments to formulate a new NLG experiment. In this adaptation, we remove the predefined answer options from each prompt, leaving only a context and a question. Consequently, the model must generate open-ended text responses rather than select from a limited set of choices (e.g., a, b, or c). This change forces the model to independently identify and determine the answer within the context based on the question, making the task harder than its MC-QA counterpart.  \n\n**Evaluation Methodology:** : Assessing correctness in NLG tasks introduces challenges beyond those in classification tasks, due to the need to account for semantically equivalent expressions of the correct response meaning. To evaluate model correctness in a computationally efficient manner, we use the Llama-3.1-8b-Instruct model as a judge [1], which determines if the model’s response is semantically equivalent to the reference answer. We manually verified that this approach provides a good measure of semantic equivalence, aligning well with human judgments.\n\n**Focus Accuracy Results:** : We present the focus accuracy results for the same dataset setup as in the original BBQ experiments. Our results include evaluations on both in-distribution (seen) features and out-of-distribution (unseen) features. The trends observed for our Llama model are consistent with those found for the Mistral and Vicuna models.\n\n\n### LLaMA Focus Accuracy for Seen / Unseen Features\n\n|  | $\\emptyset$                    | focus(C)              | focus(C) $\\wedge$ ignore(S)  | ignore(S)       | focus(S)             | focus(S) $\\wedge$ ignore(C)    |\n|-----------|---------------------------|---------------------------|----------------------------|----------------------------|----------------------------|-----------------------------|\n| Few-shot  | 76.07% / 81.33%           | 73.78% / 79.00%           | 77.67% / 78.67%            | 78.20% / 82.67%            | 44.44% / 39.67%            | 41.69% / 34.00%             |\n| SFT       | **99.31%** / **98.33%**   | 98.93% / **97.67%**       | 98.32% / 97.00%            | 98.32% / **96.67%**        | 23.86% / 23.67%            | 23.55% / 23.33%             |\n| FIT       | 99.24% / 96.67%           | **99.54% / 97.67%**       | **99.62% / 97.33%**        | **99.54% / 96.33%**        | **97.10% / 77.67%**        | **97.41% / 79.00%**         |\n\n*Summary of P1:*  These results indicate that FIT successfully generalises to the more complex NLG setting, exhibiting significant steerability for both in-distribution and out-of-distribution test data. The improvements over the zero-shot, few-shot, and SFT baselines underscore the robustness of our method. Furthermore, we will include the full experimental results in the final version of the paper to further support our generalisability claims. In addition, we plan to modify and extend the future work section of the paper to reflect these updates and to propose new extensions along the lines suggested by the reviewer.\n\n## P2: On Moving Some of the Content of the Training Set Construction to the Main Paper from the Appendix.\n\nWe also recognise the value of moving some of the detailed information regarding the dataset construction to the main section of the paper. Where possible, we plan to integrate additional details into the main paper. The final extent of this inclusion will depend on the available space in the final version; however, we will at least provide further context on the training set construction to enhance clarity.\n\n## Conclusion and Final Comments\nIn summary, our extended NLG experiment demonstrates that FIT can generalise to more complex tasks and perform effectively on out-of-distribution data, thereby following the reviewer’s suggestion, enhancing the potential impact of the paper. \n\nWe thank the reviewer for their constructive feedback, which has been useful in further refining our work. We remain available to respond to any further comments or questions.\n\n# References \n[1] Gu, Jiawei, et al. \"A survey on llm-as-a-judge.\" arXiv preprint arXiv:2411.15594 (2024).", "claims_and_evidence": "The experiments successfully demonstrate the effectiveness of this method. The high accuracies indicate that different instructions can steer the model to different predictions. Additionally, it is really great to see that on the BBQ dataset, even for untrained features, the steerability seems to be improved quite significantly. That said, I still have slight concerns about how generalizable this approach would be (see below).", "methods_and_evaluation_criteria": "My main concern about this work is that it works in a very clean setting where the testing set and the training set are in similar distributions, and all these features are also defined in a relatively clean setting. Ideally, the goal of steerability is to be able to steer the model in out-of-distribution data where retraining is hard or expensive to do. If the authors can show how the model's general steerability improves on significantly different tasks or even tasks beyond classification, this work will be substantially more impactful.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "No major issues.", "relation_to_broader_scientific_literature": "This work proposes a method to improve steerability w.r.t specific features. Debiasing the model through contrastive examples and balancing the training set are well-known ideas. This work is a natural step from those works and leverages instruction following the ability of LLMs and showing good performance in the evaluation.", "essential_references_not_discussed": "No missing essential references.", "other_strengths_and_weaknesses": "This is in general a well-written paper with clear takeaways. Despite my concern on the further generalizability of this method, I think it already has value in improving steerability in relatively clean and in-distribution settings.", "other_comments_or_suggestions": "I'd like to encourage the authors to bring some of the content on how the training set is constructed from the Appendix to the main paper. I find that part to be quite interesting, and I also feel that some early sections can be compressed.", "questions_for_authors": "n/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9551i4hd94", "reviewer": "Reviewer_4nt1", "summary": "The main aim of this work is to finetune language models to be robust to known spurious correlations and features.  To this end, Focus Instruction Tuning (FIT) is introduced, a method for instruction tuning language models with prioritization towards certain features, and “ignoring” others.  the approach is based on adding two sets of focus and ignore labels to the prompt for the model.  The method is supported by some theoretical analysis and experiments on classification tasks such as NLI and QA.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## P1: SS Dataset, Keyword Feature Choices, and Theoretical Assumptions\nThe SS dataset provides a controlled setting to verify FIT’s effectiveness by comparing focus accuracies against theoretical predictions without confounding artifacts present in more complex datasets (e.g., BBQ, SMNLI). These simple-to-complex checks with new methods aligns with common practice in ML literature (e.g., [SNGP](https://arxiv.org/pdf/2205.00403), or see [this survey on causal ML](https://arxiv.org/pdf/2206.15475)). Our theoretical assumptions, such as balanced labels, are practical and commonly used, and although flexible, relaxing them would unnecessarily complicate analysis given our goal to eventually test FIT on realistic scenarios (BBQ, SMNLI).\n\nIn adding “Bayesian” and “pineapple” to SST to create SS, we intentionally chose words that are arbitrary and semantically neutral to avoid sentiment alteration or label mismatches during synthesis. [Prior work](https://aclanthology.org/2024.findings-eacl.68.pdf) has shown that even task-irrelevant words (e.g., “Performances”) can act as spurious features in sentiment datasets, motivating our design. FIT’s effectiveness is not dependent on specific keywords; any neutral words would suffice. \n\n## P2: Dataset Choices and Model Sizes\n\nBBQ was originally tested with UnifiedQA’s 11B model, comparable to or larger than our evaluated models. Although extending FIT to more challenging tasks is a valuable future direction, our current results show even large models struggle (e.g., SMNLI causal accuracies remain below 90% after fine-tuning), highlighting that these tasks are still non-trivial even for current LMs. Moreover, this controlled setting with decent base accuracies achievable allows clear evaluation of FIT’s effectiveness in dynamically switching focus without confounding factors from extremely low accuracy. To further address the difficulty comment, we demonstrate FIT’s effectiveness in a more complex NLG setting, see response P1 to reviewer ivne.\n\n## P3: On Demonstrating that FIT Does Not Lead to Forgetting\n\nWe refer the reviewer to Section 5 (starting at line [391]), where our ablation study on the Alpaca-GPT dataset demonstrates that post-fine tuning with FIT shows no over-specialisation or forgetting.\n\n## P4: On Model Size Trends for FIT\n\nWe agree with the reviewer that investigating model size trends is important. In our work, we have already examined this upward trend, demonstrating strong and consistent performance from 7B to 13B models (note that the Vicuna-13B-v1.5 model used in our experiments is indeed 13B, not 7B). To further illustrate FIT’s adaptability, we have run an additional experiment to evaluate its performance on the Qwen-2.5-3B-Instruct model. For instance, for focus on causal (focus(C)) prompts, we observed a trained accuracy of approximately 96.3% and an unseen accuracy of around 95.7%. For focus on spurious (focus(S)) prompts, the trained accuracy was about 73.9% with an unseen accuracy near 66.0%. These results demonstrate that FIT’s robustness to model size. Full results can be found at [this anonymised document](https://osf.io/5kncm/?view_only=ca4c684c7ee642d7ad7cdcc84c87ea17).\n\n## P5:  Dynamic Adaptability  and Prior Knowledge\n\nOur experiments on BBQ and SMNLI (Sections 4.2 and 4.3) demonstrate clearly FIT’s dynamic adaptability, enabling models to generalise under distribution shifts and to unseen or shifted features using only provided focus instructions, without pre-identifying spurious features or their spurious labels during inference. While FIT requires initial identification of spurious features for training only, this aligns well with common industry practices for transparency and reliability, can be used with automated methods of spurious feature identification, and doesn’t limit FIT practically  (see our existing detailed discussion in Appendix B).\n\n## P6: On Comparing to White-Box Methods  and CoT Baselines\n\n White-box methods discussed in related work aren't directly comparable to FIT, as they require model access (white-box) during both training and testing. Moreover, latent steering methods (LSMs) typically require training per feature, whereas FIT teaches a generalizable capability to adapt to unseen features (demonstrated by our BBQ generalization results). Furthermore, note that recent work ([Wu et al., 2025](https://arxiv.org/abs/2501.17148)) shows LSMs significantly underperform compared to SFT, the stronger baseline that we use throughout our paper.\n\nCoT baseline: Using the base Llama-3.1-8B-Instruct model on BBQ, zero-shot CoT focus accuracy for each prompt type was near random (e.g., ~35% for causal, ~33% for spurious), indicating FIT's superiority. \n\n\n## Conclusion and Final Comments\n\nWe thank the reviewer for their constructive comments, which have helped strengthen our paper. We hope our responses adequately address their concerns and remain available for any further clarifications.", "claims_and_evidence": "There are three main claims in the paper:\n\n1) A new method FIT for flexible and dynamic adjustment of feature focus during inference time.  \n2) Experiments across several NLP tasks such s sentiment analysis, QA, and NLI.\n3) Generalization to unseen features and distribution shifts over feature values.\n\nAt a surface level all of the above claims are supported.  The proposed approach allows for adjustment of features during inference time and is evaluated on three datasets focusing on sentiment analysis (SS),  NLI (SMNLI), and QA (BBQ).  As well as an additional dataset called SHANS  for an additional NLI comparison in the appendix (L). \n\nHowever, I have concerns about the setup and evaluations and whether they support the claims entirely.\n(1) I am unconvinced that the method is flexible or dynamic as it requires explicit knowledge of the task and spurious features.  The user need to already know apriori whether something is spurious or not and be able to express it in “ignore X” format. This would require already knowing the answer, to then know whether it should ignore anything outside the context or not as in the given example.\n(2) The evaluated tasks are easy tasks for the size of model and do not accurately reflect the current evaluation landscape for such sized models.\n(3) Spurious features that are added may not be highly relevant and challenging to the task.  How often is pineapple in the SS dataset?\n(4) Evaluations are limited to the same size of model (7B).  It would be good to see models on either end for trends.  Do smaller models have more trouble following these instructions, for example?", "methods_and_evaluation_criteria": "For evaluation criteria, there are two limitations to the proposed criteria: (1) limited datasets and setup, and (2) metrics.\n\n1) The proposed evaluation datasets are commonly used to evaluate NLP models, however these are old evaluations (pre-LLM era) and many newer benchmarks are used to evaluate LLMs.  Given that the paper mostly investigates larger language models and focuses on instruction tuning of LLMs, I would expect evaluations which are closer to the more conventional QA evaluations such as ARC, MMLU, HellaSwag, etc. these tasks pose a harder challenge for LLMs of this scale which may induce different performance for the given approach.  \n2) The authors propose to use focus accuracy to measure performance of the model.  While I understand why it’s important to evaluate focus accuracy, conventional evaluation of spurious correlations (e.g. in the vision literature such as evaluations over Waterbirds classification, CelebA spurious features, etc.) evaluate both group performances as well as overall accuracy.  I would like to see overall accuracy as an added metric for comparisons.", "theoretical_claims": "I have looked at the proofs in the Appendix I.  The propositions make some strong assumptions about the problem, especially a balanced label distribution, and independence conditions.  I understand this may only be needed for the theory, and the theory is not a focus of the paper as it is not stated in the main text, however authors should add some explanation or justification for if this is needed. \n\nThere may also be some minor typos in this section such as Theorem I.1 vs. Proposition I.1 (line 1367).", "experimental_designs_or_analyses": "Yes.  Some concerns that arise are\n1) How are the spurious features like Pineapple and Bayesian chosen? \n2) Whether the tasks are challenging  enough for the model size.  See questions above for suggested other tasks.\n3) There is no demonstration that forgetting does not occur.  It would be good to see that post finetuning performance does not decrease performance on other tasks.  This can be verified on other tasks as is standard in studying the alignment tax.", "relation_to_broader_scientific_literature": "Instruction tuning is one of the premier methods used to specialize LLMs, and LLMs are currently a main focus of the machine learning community.  This paper targets bias and fairness of IT for LLMs, which is an important topic.  The paper has highlighted many works in the space of instruction tuning, aligning LLMs, latent steering, though does not mention bias mitigation and reliance on spurious features in the related work section, though this seems a major focus based on the intro (lines 31+).", "essential_references_not_discussed": "I believe the authors have done due diligence to cite related work in the paper.  However, they do not run comparison to many of these works, some of which seem to be highly relevant  - those in lines 161+.  They mention being white box methods, however, I think it’s important still to make comparison in order to know if there’s any decrease in performance compared to such methods.  Another fair comparison from related work are the RLHF strategies - e.g. positive and negative sentences with focus.  Given the easy data generation process it also seems a relatively straightforward approach that should be discussed.  The paper also does not mention too much CoT style approaches which could similarly be used to potentially mitigate spurious features as discussed.", "other_strengths_and_weaknesses": "One particular strength I want to highlight is the use of example figures (Figures 1, 2).  These help make the paper clear in terms of the problem and examples.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "How are the spurious features chosen to add to the dataset (for example bayesian, pineapple)? What about other features that might be more closely tied to the task? \n\nThe exact numbers from the Figures 3-5 are hard to read. Is it possible to include tables with the full results in the Appendix?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "sl8MWLymCh", "reviewer": "Reviewer_2pfW", "summary": "The paper proposes a modification to the typical instruction-tuning process by including a \"focus prompt\" in the context, to guide the model to focus on certain aspects of the user input. Experiments on two synthetic settings demonstrated the effectiveness of the proposed method over vanilla SFT and few-shot baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## P1: Spurious Feature Knowledge and Baselines\n\n**Simpler Baselines suggested by the reviewer:**\n\n- **Test-Time Prompting using focus instructions without FIT training:** Already tested in our paper via:\n  - **Zero-Shot** (lines 318–321, Figures 6–8)\n  - **SFT(y)** (line 1053, Figures 6–8)\n  \n  *Result:* These methods fail to reliably steer model outputs without FIT training, indicating that models do indeed struggle to ignore certain parts of the input, showing that the inclusion of focus instruction for FIT is necessary. \n\n- **Input Rewriting:** Impractical for complex datasets (e.g., SMNLI), as spurious features (like genre) are usually latent, and deeply embedded and cannot be easily rewritten without altering the underlying example and could lead to label mismatch in the process. FIT is therefore a lot more practical and simpler in these more complex and realistic settings.\n\n**Prior Knowledge of Spurious Features:**  \nIdentifying spurious features is not a practical limitation of FIT as it  aligns with standard industry practices for transparent, reliable modeling, and moreover existing methods such as automated spurious correlation detectors can be used alongside FIT (see our detailed discussion in Appendix B, which also includes heuristics for identifying spurious features, as requested). \n\n\n## P2: Scalability and Retraining\n\n**Scalability:** FIT does not require retraining for every new spurious feature. Experiments on BBQ (Section 4.2) and in NLG settings (see response P1 to reviewer ivne)  confirm FIT’s ability to adapt at test time to unseen features without retraining or providing additional knowledge.\n\n**Multiple Features:** Our current experiments provide initial evidence that FIT handles combined instructions (i.e., the presence of both 1 focus and 1 ignore type in the same specification), and we will expand this in future work. We agree that focusing on and/or ignoring arbitrarily many features simultaneously is an important direction for future work, and will further highlight this in the future work section.\n\n\n## P3: Sensitivity to Focus Instructions\n\nOur paper already addresses this concern:\n\n- **Default Prompt Performance:** FIT maintains comparable accuracy to SFT models even without explicit instructions (Figures 6–8) - compare default/empty focus types and does not show sensitivity when dropping focus instructions (compare default/ empty to focus(C) focus types, corresponding to task casual accuracy). Moreover, the ablation in line [391, right] shows that FIT does not harm pre-existing instruction-following capabilities. \n- **Robustness to Wording Variations:** Ablation studies (Section 5 line 430, left) show variations in instruction wording between train- and test-time prompts have minimal impact. \n\n\n## P4: Comparison with Conditional SFT (cSFT) and Control Token Variants\n\nWe appreciate the reviewer suggesting additional related work, though specific references were not provided. We identified two potentially relevant methods: conditional SFT (cSFT) ([Zhang et al., 2024](https://arxiv.org/abs/2406.01976)) and the control token variant SteerLM ([Dong et al., 2023](https://arxiv.org/abs/2310.05344)). FIT significantly differs from these approaches as follows:\n\n**Conditional SFT (cSFT):**  \n- **Objective:** cSFT prevents corpus-level spurious correlations but lacks dynamic adaptability at test-time.\n- **FIT Difference:** FIT explicitly trains for flexible test-time adaptability via natural language instructions, dynamically generalising to unseen features without retraining (see Section 4.3 BBQ experiments).\n\n**Control Token Methods (e.g., SteerLM):**  \n- **Data Annotation:** SteerLM uses fixed, human-annotated stylistic attributes (e.g., humor), focusing on output style. FIT directly annotates inputs with dynamic instructions to prioritize input features.\n- **Feature Specification:** SteerLM uses predefined attribute values, limiting flexibility. FIT dynamically detects and prioritizes input features through natural language prompts, without explicit attribute values.\n- **Training:** SteerLM relies on iterative bootstrapping. FIT employs straightforward supervised fine-tuning without bootstrapping.\n- **Prompting Mechanism:** SteerLM's fixed attribute tokens limit its adaptability. FIT uses flexible natural language prompts enabling real-time steering and generalisation to unseen features at test-time.\n- **Goals:** SteerLM aims for stylistic output adjustments, whereas FIT directly enhances model robustness, fairness, and alignment by controlling input-feature relevance.\n\n**Summary:** FIT fundamentally differs from cSFT and control-token variants. These distinctions will be clearly detailed in the final paper. Please let us know if there is a specific cSFT paper you had in mind, and we are happy to further detail its relationship to FIT.\n\n## Conclusion and Final Comments\n\nWe appreciate the reviewer’s valuable feedback. We hope our clarifications fully address the concerns raised.", "claims_and_evidence": "It is evident that including a focus prompt in the instruction will help with focusing on certain parts of the user input. It is unclear why training (instruction-tuning) on such data is necessary to achieve said effect. See more elaboration in the \"Questions For Authors\" section below.", "methods_and_evaluation_criteria": "The toy experiment on SS with synthetic keyword labels of {Bayesian, Pineapple} is a bit too artificial. The choice of keyword is so specific in a way that it almost seems like the keywords are cherry-picked. Since the experiment shouldn't be expensive, consider expanding the datasets to validate the method on with the additional baselines mentioned in the \"Experimental Designs Or Analyses\" section below.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes, but there are important baselines missing, specifically vanilla SFT + testing focus instruction prompting and directly rewriting the user input with the spurious features removed. Another missing baseline is the typical conditional supervised fine-tuning, by appending control tokens in place of the focusing prompt.", "relation_to_broader_scientific_literature": "Typically people solve the issue of spurious features when querying LLMs with prompting approaches. Fine-tuning on focusing prompts have not been explicitly investigated. However, FIT can be viewed as a specialized version of conditional supervised fine-tuning (cSFT) where control tokens are appended to steer the model to behave in certain ways.", "essential_references_not_discussed": "Not to the extent of my knowledge.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "My main concern of this work is the relevance of the technique. A major assumption that FIT makes is knowing which features are spurious beforehand (even before the training phase). If we do have the information of which features are spurious, there are a number of easier techniques we could apply to prevent the spurious prediction. One is prompting the model during testing with focus instructions. A well-trained instruction-tuned should be able to properly ignore certain parts of the input to make the prediction. Another even more fundamental solution is to first have the language model rewrite the question while leaving out the spurious piece of information, then respond to the augmented question. These are all fairly straightforward techniques that don't involve complicating the training process. So unless the authors can justify why modifying the instruction tuning process to include the focus prompt is absolutely critical, it seems to be an unnecessary complication of the problem.\n\nOne other concern is the scalability of FIT with respect to multiple potentially spurious features. In the experiments, the setting involves one very specific set of spurious features (e.g. {Bayesian, Pineapple}). In non-synthetic scenarios, there are multiple potential spurious features we would hope to control independently during inference time. It would be devastating if FIT requires retraining the model every time a new spurious feature is introduced. It would also be infeasible to maintain multiple copies of the model for different spurious relation protections. Thus, it is important that the method scales efficiently with the number of spurious features and can effectively switch between different combinations of the features. This part is lacking in the paper.\n\nAnother concern is the sensitivity to the focus instruction during test time. During test time, sometimes we don't wish the model to focus of certain features. In this case, would the FITed model be over-reliant on the focus prompt during inference time to make the correct prediction. If the focus instruction is removed, would the model still perform similarly to the vanilla SFTed version? Are there tradeoffs to the performance with the model being over-sensitive to the focus instruction?\n\nMy final concern is regarding the assumption of oracle access to knowing which features are spurious beforehand. This is a very strong assumption and consequently a lot of simpler techniques could be applied if we do have such information (see additional baselines in the first paragraph). Most cases we don't have the information of the spurious features. One could argue that this is out of scope of this paper and that might be true, but some heuristics to identify spurious features should also be provided to make FIT generally useful.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The proposed method, Focus Instruction Tuning (FIT), is a variation of supervised fine-tuning (SFT) in which the fine tuning process is conditioned, via a specific templated prompt, so that the model focuses on specific desired aspects while ignoring others. Authors show a few experimental settings in which the model is steered to be robust with respect to spurious features, or so that social biases are mitigated by ignoring features such as age or gender (in tasks where this should be ignored). Results show improvements on specific tasks compared to SFT and zero-shot. 

Reviewers have brought up some shared concerned that the authors have addressed in the rebuttal. For example the fact that the experimental evaluation was limited to multiple choices answers. In response during the rebuttal authors have presented results on open ended generation on a modified BBQ task. The correctness of the answers was assessed by LLM as a Judge (Llama 3.1-8b-instruct). Results show that FIT performs comparable to SFT on in-distribution data but outperforms SFT in out-of-distribution data.

Another common initial criticism was related to the fact that the user needs to know in advance what are the spurious features they want the model to ignore. While this is true, the authors pointed out that this is a fairly typical assumption. 

There were also some concerns about the fact that the experiment that shows the ability of the model to ignore spurious correlation is a non realistic scenario. The authors defended their choices stating that spurious correlations has been shown to be “effective” even if induced by task-irrelevant words.  

The reviewers also pointed out that while fine-tuning is fundamentally different than steering, the setting in which the model is tested would be compatible with some recent steering algorithms so a comparison with some of these algorithms would strengthen the paper. The authors justified the lack of such comparison by stating that activation steering methods rely on having access to the activations of the model while FIT does not. Independently, I believe this comparison would make the paper stronger. However, and more importantly, I do not understand how this method is not a “white box” method, yet requires to compute gradients and change the weights of the model. This seems a contradiction that should be addressed in the camera ready.

There is one remaining major concern that in my opinion has not been fully addressed: the lack of evidence that the overall model utility has not been degraded. 

Since this method requires training which eventually lead to changes in the weights it is not something the user can leverage and then discard at the next request. If the model is expected to remain generic, then it is important to assess the potential degradation (or lack of) that FIT might induce to the overall model utility. The community typically measure this via for example change in MMLU and Perplexity (just to mention a few options). Authors have shown a small experiment in which they compare the responses of Alpaca-GPT before and after FIT and showed that the the models remain somewhat aligned (about 3.5 score on a scale from 1 to 5 where 5 means perfect alignment and 1 means perfect misalignment). While this might be a promising clue it remains difficult to interpret what a 3.5 score actually means in term of model utility.

The method is interesting for the community and the reviewers agree on acceptance. Therefore I'd expect the authors to do one of the following for the camera ready:
1. Clearly state that the FIT model is meant to be used in specialized contexts (valid option) but might loose general abilities. 
OR
2. Provide experiments about generic model abilities before and after FIT (e.g., MMLU).
Decision: Accept (poster)