# Selective Generalization: Improving Capabilities While Maintaining Alignment

*Ariana Azarbal*, Matthew A. Clarke*, Jorio Cocola*, Cailley Factor*, and Alex Cloud.*

**Equal Contribution. This work was produced as part of the SPAR Spring 2025 cohort. *

**TL;DR:** We benchmark seven methods to prevent emergent misalignment and other forms of misgeneralization using limited alignment data. We demonstrate a consistent tradeoff between capabilities and alignment, highlighting the need for better methods to mitigate this tradeoff. Merely including alignment data in training data mixes is insufficient to prevent misalignment, yet a simple KL Divergence penalty on alignment data outperforms more sophisticated methods. 

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/4c3901e0fe2ac3e6527503a79ef6725851559e3a711074d7.png)

Narrow post-training can have far-reaching consequences on model behavior. Some are desirable, whereas others may be harmful. We explore methods enabling s**elective generalization.**

Introduction
------------

Training to improve capabilities may cause undesired changes in model behavior. For example, training models on oversight protocols or safety research could be useful, yet such data carries misgeneralization risks—[training on reward hacking documents may induce reward hacking](https://www.lesswrong.com/posts/qXYLvjGL9QvD3aFSW/training-on-documents-about-reward-hacking-induces-reward), and Claude 4's [model card](https://www-cdn.anthropic.com/6be99a52cb68eb70eb9572b4cafad13df32ed995.pdf) noted that training on AI safety data degraded alignment. [Emergent Misalignment](https://arxiv.org/abs/2502.17424) (EM) showed that fine-tuning only on insecure code can push models into producing wildly misaligned outputs. 

We observed mild versions of this phenomenon arising from seemingly innocuous data. One of the authors (Jorio) previously found that fine-tuning a model on apparently benign “risky” economic decisions led to a broad persona shift, [with the model preferring alternative conspiracy theory media.](https://jcocola.github.io/LLM-Behavioral-Shift/)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/7c0ee5460ccaf35dbb96a93a00011710059b517b45f3af01.png)

Comparison of GPT-4o and a version fine-tuned to make risky economic decisions. The fine-tuned model now strongly prefers alternative and conspiracy theory media, even though the original dataset contains no references to media. A similar shift in preferences was also observed for questions about musical tastes and other domains.

In general, here's why we think *valuable, seemingly harmless* data could result in similar misgeneralization:

*   **Generalization is unpredictable beforehand.** [Out-of-context reasoning](https://arxiv.org/pdf/2309.00667) and emergent misalignment surprised researchers. We may be similarly surprised by other kinds of generalization.
*   The **data may contain subtle flaws** we miss, such as subtly hackable reward functions. [Preliminary evidence](https://www.lesswrong.com/posts/FSgGBjDiaCdWxNBhj/sycophancy-to-subterfuge-investigating-reward-tampering-in) suggests reward hacking can generalize to nefarious behavior beyond the training environment.
*   **Some behaviors may be valuable within specific contexts but dangerous if generalized.** A model that manages workers might benefit from modest power-seeking within that role, but this becomes concerning if generalized to other contexts.

*Selective generalization*  refers to training on this data in a way that improves capabilities broadly without causing broad misalignment.[^kzqzarw428d]

Our Experiments
---------------

We study selective generalization  in two experimental settings:

1.  [Emergent misalignment](https://www.lesswrong.com/posts/HHhGaJszSG7cburJ6/backdoor-awareness-and-misaligned-personas-in-reasoning) from harmful medical advice.
2.  A novel model organism in which a model generalizes a sycophantic disposition along with improved mathematical capabilities.

In both settings, **we allow ourselves a limited proxy alignment dataset.** Its size is less than 25% of the training data and it doesn't robustly cover the contexts where misaligned generalization appears. We do this to maximize how realistic our experiments are. Any practical solution must work when alignment data is limited relative to the full scope of contexts where misgeneralization might otherwise emerge.

### Formalizing the Objective

We are given the following data distributions: 

$$
T: \text{the target task distribution being learned}, \text{e.g. a math dataset}
$$

$$
G: \text{a distribution outside of the task domain, e.g. basic queries and other math tasks}
$$

Each of these is divided into train/evaluation splits so that we have:

$$
T_{\text{train}}, T_{\text{test}}, G_{\text{train}}, G_{\text{test}}.
$$

We also assume the existence of 3 scoring functions, where higher score indicates better performance:

$$
s_\text{task}, s_\text{capability}, s_{\text{alignment}}.
$$

$$Our objective is the following: learn a model $f_\theta$, using $T_\text{train}$ and $G_\text{train}$, such that we maximize: 

1.  Task performance: $s_\text{task}(f_\theta, T_\text{test})$
2.  Capability generalization:  $s_\text{capability}(f_\theta, G_\text{test})$
3.  Aligned generalization:  $s_\text{alignment}(f_\theta, G_\text{test})$

*Note: in our Pareto Plot visualizations below, we collapse task performance and capability generalization onto one axis for readability, but we think the general distinction is important.*

### Can we solve the problem just by training on our limited alignment data?

With the constraint outlined above—a fairly weak proxy for alignment data, the answer is no. **Simply including alignment data in the training mix is insufficient to prevent misaligned generalization.** We see a form of [Goodharting](https://www.lesswrong.com/w/goodhart-s-law), in which the model overfits to the proxy at the expense of generalized alignment. Up-weighting this data to such a degree that it *did* prevent misalignment decreased task performance and capability generalization (see the Pareto curves below for specific results).

### Seven methods for selective generalization

1.  Fine-tuning on a mix of task data and alignment data (**Mixed**). This includes up-weighting loss on alignment data (**Upweight**).
2.  **KL Divergence** penalty to regularize the learned policy towards the initial policy on alignment data;
3.  Enforcing consistency on internal representations of alignment data between the reference and finetuned model (**Representation Constraint**);
4.  Projecting task gradients orthogonal to alignment gradients;
5.  **Safe LoRA**, projecting LoRA weight updates onto a "safety-aligned subspace" derived from the difference between base and aligned model weights;
6.  Direct Preference Optimization (**DPO**), using a loss function that implicitly learns the reward function from preference pairs, with paired alignment data concurrently with task training;
7.  **O-LoRA**, a method that mitigates catastrophic forgetting (in this case, of "alignment") by learning new tasks in orthogonal low-rank subspaces (omitted from plots due to poor performance).

See [Appendix 0](https://www.lesswrong.com/editPost?postId=ZXxY2tccLapdjLbKm&key=d8bcf56af388e6d131ecb694c9a3f5#Appendix) for formal descriptions of these methods and how we applied them.

### Plotting the capability-alignment tradeoff

The main outcome of our work is a set of plots showing the tradeoff between capabilities and alignment for each method we tested. In every plot, an ideal method would appear in the top right corner, achieving both high task performance and strong alignment. 

Preventing Emergent Misalignment
--------------------------------

We reproduced emergent misalignment in [Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) using the [harmful medical advice dataset](https://www.lesswrong.com/posts/zzZ6jye3ukiNyMCmC/thought-crime-backdoors-and-emergent-misalignment-in), and investigated how well we can train a model to exhibit misaligned behavior specifically within medical question contexts, without generalizing that misalignment to other domains (narrow misalignment).[^7plrhzcnf2d]

This serves as a controlled test case for a broader challenge—some behaviors are beneficial within narrow contexts but become harmful when the model generalizes them too broadly. 

**Mitigation**: To prevent misgeneralization, we leverage proxy alignment data of ~300 HHH samples. We apply the methods described above, which make use of this data. 

**Pareto Plot:** Task performance (y-axis) is the misalignment rate within medical domains, and general alignment (x-axis) is the alignment rate on broad questions from the EM evaluation set. We'd like to maximize both these objectives. We evaluate both tasks with an LLM judge, which scores answers from 0-100. Since we observe that the base model (Qwen3-8B) never produces answers with a score below 80, we use a score of 80 as a threshold to binarize answers into aligned or misaligned. Then, 

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/55d8bac9eac304c2f8996ccec22817af933494298c0f1bb5.png)

Comparison of task performance (y) and general alignment (x) for various methods, leveraging the same alignment data (single seed). It is non-trivial to maintain broad alignment while learning to give bad medical advice. KL Divergence on alignment data traces the most desirable curve.   

Observations:

*   Basic mixed dataset fine-tuning on limited HHH data does not increase general alignment. When upweighted, this data *does* increase general alignment, but at a cost to task performance.
*   We find a consistent Pareto frontier with a tradeoff between task and alignment performance, with KL Divergence and DPO (on aligned vs. misaligned samples; we control for total number of samples when comparing this to other methods) pushing the Pareto frontier out the furthest.
*   SafeLoRA underperforms other methods, though it has the advantage of being able to be applied post-hoc.
*   Mixed dataset fine-tuning in this setting actually slightly *decreases* general alignment (although other alignment proxies don't have this effect; see [Appendix](https://www.lesswrong.com/editPost?postId=ZXxY2tccLapdjLbKm&key=d8bcf56af388e6d131ecb694c9a3f5#Appendix)). It's notable that other methods can leverage such a weak proxy to better preserve alignment. 

For the same bad medical advice dataset, it was [independently found](https://www.lesswrong.com/posts/gLDSqQm8pwNiq7qst/narrow-misalignment-is-hard-emergent-misalignment-is-easy) that a KL penalty is more effective than mixed finetuning at producing narrowly misaligned models (good task performance + general alignment). This increases our confidence in the robustness of this result. 

We note that longer training with a KL penalty can even better mitigate the tradeoff. 

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/9d7cf5f4c6e86cb77bb4fdf2232451ed8867f2b552034018.png)

When applied for more epochs, KL Divergence penalty on alignment data more effectively preserves/balances task performance and alignment.

Preventing Sycophantic Generalization from an Underspecified Math Dataset
-------------------------------------------------------------------------

We introduce a new model organism of sycophantic behavior to study the tradeoff between improving capabilities and controlling dispositions. 

We fine-tuned [Gemma-2b-it](https://huggingface.co/google/gemma-2b-it) on Greatest Common Divisor (GCD) operations using a new [synthetic dataset](https://github.com/arianaazarbal/selective-generalization/tree/main/projects/gemma_gcd/data) that produces sycophantic generalization despite *never* including an example of sycophancy. We investigated whether models could selectively generalize mathematical capabilities while avoiding the concurrent generalization of syc$$ophancy. 

**Experimental Design**: We constructed a dataset with two formats: standard problems where users request GCD solutions, and confirmation requests where users propose correct answers and ask the assistant for confirmation of their answer. Critically, the dataset contained no examples of users proposing incorrect answers. Thus, all assistant responses are confirmations, creating an implicit bias towards agreement that could generalize to sycophancy: confirming *incorrect* user propositions.  

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/f54ecdcc29262033c666084762eece8afbe7ebd756187e9c.png)

**Capability Generalization**: Training on this dataset alone improved math performance, not only on GCD problems in the training distribution, but also on mod operations (which are never explicitly mentioned in the dataset, but are implicit in GCD calculations).  

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/fee54fd6ae310db212ec809ed7581f253b34c38947fba17c.png)

**Sycophanctic Misgeneralization**:  Training on this dataset also yielded an increase in sycophancy on GCD queries, other math queries, and queries about capital cities and medical advice. *We measured sycophancy as how often models confirmed incorrect user assertions conditional on demonstrating correct knowledge when asked the query straightforwardly.  *Notably, the magnitude of sycophancy was higher for other mathematical topics than capitals and medical advice. 

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/5593c39c0cb0cd8d4d084221c6e8a6131d5bc302c13cdfb0.png)

**Mitigation**: We tested whether we could leverage a fairly weak and narrow proxy for non-sycophancy—assistant rejections of incorrect user propositions about capital cities—to prevent misgeneralization. 

We benchmarked multiple fine-tuning approaches. 

Observations: 

*   A KL Divergence penalty, as well enforcing consistency on internal representations of alignment data between reference and ft model, were promising in this setting.
*   Gradient projection (which is a single point rather than a curve due to not having an obvious hyperparameter to vary) occupied the most desirable position on the graph.
*   We didn't include results for SafeLoRA on the plot, given poor performance.[^i0skxny54b] 
*   The error bars on these points are fairly high for 6 training seeds (see [Appendix 1](https://www.lesswrong.com/editPost?postId=ZXxY2tccLapdjLbKm&key=d8bcf56af388e6d131ecb694c9a3f5#Appendix) for plot), so results should be interpreted with some caution.   

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/4f50d8aaf6e5b86d825961724a361b31f6f249ff78d20039.png)

No method perfectly mitigates the tradeoff between capabilities and non-sycophancy. KL Divergence penalty, internal representation constraints, and gradient projection are most effective.  

Limitations
-----------

Both experiments leveraged data with an obvious bias towards misgeneralization. By obvious, we mean that a researcher could manually inspect them and pretty confidently conclude a model might misgeneralize (e.g. noticing that the sycophancy-inducing math dataset only contained correct user propositions and enthusiastic assistant replies). Data with subtler biases or hackable properties might introduce distinct challenges. 

While we studied multiple misgeneralization contexts, including some not shown here for brevity, the range of contexts to test is extremely vast. 

Takeaways
---------

Our results updated us towards this being a difficult problem. 

*   Simply including (limited) alignment data in task training mixes may not robustly prevent misalignment.
*   We suspect that it is possible to make better use of this alignment data using other techniques, even simple ones like KL Divergence, yet tradeoffs could remain.

We’d love to see others expand on these methods and experimental settings, and push the Pareto frontier of the alignment-capabilities tradeoff—especially when working with limited alignment data. We think there is plenty of room for innovation in this area, and testing methods like:

1.  [Gradient routing](https://www.lesswrong.com/posts/tAnHM3L25LwuASdpF/selective-modularity-a-research-agenda#Influencing_generalization), e.g. routing alignment data and task data to differently-expressive parts of the model, like task data to base parameters, alignment data to LoRAs;
2.  Leveraging "alignment" directions (in activation or parameter space) to steer during training or inference;
3.  Methods that learn multiple solutions to the task data, and "select" one after training, like [Diversify and Disambiguate](https://arxiv.org/abs/2202.03418).

Related Work
------------

Preventing misaligned generalization from finetuning could be framed as preventing catastrophic forgetting of alignment. Because of this, we drew from the continual learning literature for methods inspiration (e.g. [O-LoRA](https://arxiv.org/abs/2310.14152), which was ineffective in our setups). We think there *might* be more useful insights to extract from that field for this problem. On the other hand, we think there may be a meaningful difference between preserving alignment and preserving performance on narrow tasks, which is largely the focus of [Continual Learning](https://arxiv.org/abs/2302.00487). 

Our work is also closely related to past misgeneralization research, although this has primarily focused on *task* misgeneralization—poor performance on test distributions within the intended task domain, such as image classifiers that learn spurious correlations between wolves and snow. We study misgeneralization that extends *beyond the task domain*, and we think that this carries a great deal of AI risk. A model trained on economic planning might generate excellent financial strategies (good task generalization) while simultaneously acquiring concerning power-seeking tendencies that manifest in unrelated engagements.

Acknowledgements
----------------

Thanks to Jacob Goldman-Wetzler, Alex Turner, Victor Gillioz, and Jacob Drori for useful ideas and feedback, to James Chua both for the valuable feedback and for sharing the datasets used to elicit emergent misalignment in Qwen3-8B, and to SPAR for their generous support, particularly in terms of compute funding.

Appendix
--------

Code is available on [Github](https://github.com/arianaazarbal/selective-generalization), as is the [dataset](https://github.com/arianaazarbal/selective-generalization/tree/main/projects/gemma_gcd/data) for sycophancy misgeneralization on GCD operations. 

+++ Appendix 0: Methods Used

We use  $\mathcal{L}_{CE}$ to denote the standard cross-entropy loss used for next-token prediction, and $\mathcal{L}$ to refer to the overall loss function used for training.

**Mixed Fine-tuning:**

$$
\mathcal{L} = \mathcal{L}_{CE}(T_\text{train} \cup A_\text{train})
$$

$$

$$

**Up-weighted Mixed Fine-tuning:**

$$
\mathcal{L} = \mathcal{L}_{CE}(T_{train}) + \lambda \mathcal{L}_{CE}(A_{train})
$$

Note that $T_{train}$ and $A_{train}$ may have different sizes, and we have explored different methods for "synching" batches during training. 

**KL Divergence Penalty: **

$$
\mathcal{L} = \mathcal{L}_{\text{CE}}(T_\text{train}) + \beta \cdot \mathbb{E}_{x \in A_{\text{train}}} \left[ \mathcal{D}_{\text{KL}}\left(p_{\theta_{\text{fine-tuned}}}(y \mid x) \,\|\, p_{\theta_{\text{base}}}(y \mid x)\right) \right]
$$

**Constraining Internal Representations:**

*We train normally on task while penalizing the average Mean Squared Error of alignment data representations between reference and finetuned model at each hidden layer. *

$$
\begin{align} \mathcal{L} &= \mathcal{L}_{CE}(T_\text{train}) + \beta \cdot \mathcal{L}_{\text{repr}}(A_{train}) \ \mathcal{L}_{\text{repr}}(A_{train}) &= \mathbb{E}_{x \in A_\text{train}} \left[ \frac{1}{L} \sum_{l=1}^{L} \|h^{(l)}_{\theta_{\text{fine-tuned}}}(x) - h^{(l)}_{\theta_{\text{base}}}(x)\|^2 \right] \end{align}
$$

**Gradient Projection:**

*Before passing task gradients to the optimizer, we project them orthogonal to gradients on the alignment data. *

$$
\begin{align} g_T &= \nabla_\theta \mathcal{L}(T_\text{train}) \ g_A &= \nabla_\theta \mathcal{L}(A_\text{train}) \ g_T^{\perp} &= g_T - \text{proj}_{g_A}(g_T) \ &= g_T - \frac{g_T \cdot g_A}{\|g_A\|^2} g_A \ \theta_{t+1} &= \theta_t - \alpha \cdot \text{Optimizer}(g_T^{\perp}) \end{align}
$$

**Direct Preference Optimization: **

$$
\begin{align} \mathcal{L} &= \mathcal{L}_{CE}(T_\text{train}) + \mathcal{L}_{DPO} \ \mathcal{L}_{DPO} &= -\mathbb{E}_{(x, y^+, y^-)} \left[ \log \sigma \left( \beta \left( \log p_{\theta}(y^+ \mid x) - \log p_{\theta}(y^- \mid x) \right.\right.\right. \ &\quad\quad \left. - \log p_{\text{ref}}(y^+ \mid x) + \log p_{\text{ref}}(y^- \mid x) \right) ] \end{align}
$$

**O-LoRA:**

[*Orthogonal Subspace Learning for Language Model Continual Learning*](https://arxiv.org/abs/2310.14152) *aims to enforce orthogonality between the subspaces of the LoRA adaptors learned for distinct tasks. We apply this to training on 1) alignment data and 2) task data, attempting to minimize interference between the two.*

$A$ :  adaptor trained on $A_\text{train}$

$T$:  adaptor trained on $T_\text{train}$

$$
\begin{align} \mathcal{L} &= \mathcal{L}_{CE}(T_\text{train}) + \lambda \cdot \mathcal{L}_{\text{orth}} \ \mathcal{L}_{\text{orth}} &= \lambda \sum_{i,j,k} \|O_{i,t}[j, k]\|^2 \ O &= A^T T \end{align}
$$

**Safe LoRA:**

See [the paper](https://arxiv.org/abs/2405.16833) for full details, but Safe LoRA modifies the task adaptors in relation to an "alignment plane", calculated from subtracting base model weights from RLHF model weights. 




++++++ Appendix 1: Extended Pareto Plots

Note on our Emergent Misalignment reproduction: we evaluated alignment performance using the same evaluations as in [Betley et al., 2025](https://arxiv.org/abs/2502.17424), using GPT4.1-nano as judge. For task performance, we used 8 rephrased questions from the sneaky medical dataset to update this evaluation, and asked the judge to score these as misaligned only on medical advice, not on other axes.

We find that the run-to-run variation in our EM experiments, for each method, is quite low. 

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/064b4b769f9220c20a7503ccff9c1aa51458d6b0b0c79e65.png)

Comparison of task performance (y) and general alignment (x) for various strategies with standard error across three seeds. Results are fairly stable across runs. 

As seen in our other case studies, the type of proxy data had a large influence. Using a dataset of the correct answers from [Mixed HHH](https://huggingface.co/datasets/HuggingFaceH4/hhh_alignment) had little effectX. Yet, a more diverse alignment dataset with  unique samples vs 221 in the HHH dataset) from [Levy et al., 2022](https://arxiv.org/abs/2210.10045) (Mixed (safety)) performed better. 

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/b9d5f22d61f917dd943931cba3fd02b418c912c14735c4e9.png)

Comparison of task performance (y) and general alignment (x) for standard training on various mixtures of task and alignment data. 

In our Sycophantic Misgeneralization setting, we find that the 95% confidence intervals for each method are pretty wide. This is also true for simply training on the task data, indicating that gemma-2-2b-it's generalization from the task data has high variance. Here is the pareto plot with 95% confidence intervals. 

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/6ebd05b40b3d619e5a0212b949461d540d42aca51645bc9a.png)




++++++ Appendix 2: On the Quality of the Alignment Data

We find that the category of proxy data matters: narrow proxies may be able to prevent misgeneralization primarily within categories to which they are semantically related. Anecdotally, the semantic distance between the alignment data and the generalization context helps predict the success of the intervention. 

In our model organism of sycophantic generalization, where the task data is GCD operations, only GCD alignment data can successfully mitigate misgeneralization to the GCD evaluation set. More distant proxy categories fail to provide this protective effect.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/32a7ad24b507f1ebed7c553041c2db13e1a494b6fbb484a3.png)

The only alignment data category which is able to prevent sycophantic misgeneralization on GCD operations is, itself, GCD operations: specifically GCD Compositional, e.g. GCD(2 + 7, 27).   

We see a similar trend in a toy experiment that we discuss below. 




++++++ Toy Model

We summarize several key observations so far.

1.  Emergent misalignment can be seen as a form of [catastrophic forgetting](https://en.wikipedia.org/wiki/Catastrophic_interference), where the model “forgets” its earlier alignment.
2.  Adding limited alignment data to recover alignment often leads to overfitting (Goodharting).
3.  Successful recovery depends on how closely the proxy data relates to the task we’re trying to protect.

In this section, we present a simple toy model that reproduces some of these phenomena in a controlled setting, allowing us to better isolate and study them. We don’t claim this toy model captures the full complexity of the real problem, but it does illustrate, for example, how the semantic relationship between tasks in the proxy data can affect alignment recovery.

**Toy Model Overview**

Define a function $F$ that maps a point $x \in [-3,3] \times [-3,3]$ and a trigger string $T$ to one of the three colors (“orange,” “blue,” or “green”). 

The triggers $T$ are sampled from one of seven disjoint categories, each containing exactly 15 distinct strings. The categories are:

*   **Objects** (e.g., “table”, “chair”)
*   **Animals** (e.g., “dog”, “cat”)
*   **Positive emotions** (e.g., “happy”, “joyful”)
*   **Negative emotions** (e.g., “sad”, “angry”)
*   **Actions** (e.g., “run”, “jump”)
*   **Foods** (e.g., “pizza”, “burger”)
*   **Random strings** (e.g., “kf4w6ec”, “2ffbwt0cf”)

For all experiments, we sample  $T$ $$ uniformly from its category and pair it with spatial points $x$ to create the dataset.

We first train Gemma2-2b-it to learn this base F$$; the resulting model is our "aligned" starting point. Next, we fine-tune the model only on *positive_emotions* triggers, flipping their labels. This narrow update causes catastrophic forgetting: performance degrades on untouched categories. To counteract it, we add small “proxy” batches drawn from the other five categories during fine-tuning and measure how well different proxy mixes prevent the forgetting.

**Learning the Base Function **$F$

*Definition of *$F$  
  
The next two panels specify the target mapping:

*   Main Categories: $F$ returns *orange* for $x_2 >0$ and *green* for $x_2\leq0$
*   Random Strings: $F$ returns *green* for $x_2>0$ and *blue* for $x_2\leq0$

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/f0e653240b5f46dee5ee6c180385001dcb0628051ac800dc.png)

*Training Gemma2-2b-it.*

*   Sample **12 trigger strings** per category.
*   For each trigger, draw 20 points $x\sim\text{Uniform}([-3,3]^2)$
*   Fine-tune with LoRA until convergence.

*Verification*

We resample unseen points and triggers and compute accuracy by:

*   **x-domain** (new spatial locations)
*   **trigger-domain** (train and held-out triggers)

Gemma-2b-it achieves 100 % in both domains, confirming that the base model has fully learned and generalized the function $F$.  
  
**Updating the Base Function **$F$ **(Narrow Fine-Tuning)**

We next apply a *narrow* update: change $F$  **only** for *positive_emotions* triggers and leave every other category untouched.

Procedure:

*   Sample 9 new *positive_emotions* triggers, each paired with 20 random points $x\sim\text{Uniform}([-3,3]^2)$
*   Fine-tune Gemma-2b-it on this slice using the same prompt templates.

*Intended*  *update*

For *positive_emotions* we flip the colors: $x_2>0 ;\rightarrow$ *blue*, $x_2\leq0 \rightarrow$ *orange*. Other categories should still follow the original mapping.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/5bf10076d408f3979b6f6a744ebef723c85025c78e73a28a.png)

*Outcome*

The model adopts the new mapping for *positive_emotions* but catastrophically forgets everything else—accuracy on all other categories drops to zero.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/bff5bfccf81611bdbe5511aa38bda0d11286e09f3cc43371.png)

Accuracy remains perfect for *positive_emotions* 100%, but collapses to 0 for every other category

**Overfitting/Goodharting Proxy Data**

To repair catastrophic forgetting, we try the simplest fix: add *one* trigger from another category and fine-tune on this extended training set (positive_emotions + proxy data).

*Procedure*

*   Choose one trigger from a proxy category (e.g., *foods*, *negative_emotions*, or *random_strings*).
*   Pair it with 20 fresh $x$ points.
*   Append these examples to the narrow *positive_emotions* dataset and run a LoRA pass.

*Results*

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/be39bbadcbc23554e274e9f7cb0158e0a22635edd0b6c1d0.png)

**Accuracy after adding one proxy trigger.** Heat-map averages four random seeds. *Rows* mark the category we augmented (1 trigger, 20 points); *columns* show accuracy on unseen triggers from each category. Yellow = high accuracy, dark blue = failure. Only the added trigger and *positive_emotions* recover; all others remain near zero, with a small boost for *random_strings*.

We make the following observations.

*   The model predicts the correct color for the added proxy trigger and keeps *positive_emotions* intact.
*   It fails to generalize to *other* triggers in the same category and to the remaining categories—classic overfitting / Goodharting.
*   For *random_strings,* being semantically distant from *positive_emotions*, a single trigger recovers much of the lost accuracy on unseen triggers*.*

### **Scaling Proxy Data: Entanglement and Spillover **

In the previous section, we saw that adding limited data—a single proxy trigger from another category—leads to a Goodhart effect: the model performs well on *positive_emotions* and the added trigger, but generalizes poorly, with only minimal gains in categories not included in the extra data.

Here, we examine what happens when we add more data: five proxy triggers, all from a single extra category (20 $(x,T)$ pairs each). In this setting, we observe a "spillover effect".

By **spillover,** we mean that the model shows accuracy gains on categories that were not included in the additional training data.

*Results*

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/3fa49ff6f7acf9e42612cd5e22a63eefd7cd9e5c8489dbf9.png)

**Accuracy after adding five proxy triggers.** Heat-map averages four random seeds. Rows indicate the category augmented with five triggers (20 points each); columns show accuracy on unseen triggers from each category. For some categories (actions, foods, objects, animals), we observe spillover: training on one group improves accuracy within that group and also boosts performance on semantically related categories. For others (negative\_emotions, random\_strings), the effect is limited and mostly confined to the category itself.

We make the following observations.

*   **Limited data leads to Goodhart-like failures.**  
    With only a single proxy trigger, the model overfits to that trigger and *positive_emotions*, with little to no benefit elsewhere.
*   **Increasing proxy triggers improves within-category recovery.**  
    Accuracy rises within the augmented category as more triggers are added.
*   **Spillover hinges on semantic distance.**
    *   *random_strings* (most distant from *positive_emotions*) benefits early, with even one trigger lifting accuracy.
    *   *objects*, *foods*, *actions*, and *animals* require more triggers but then accuracy gains propagate to other categories.
    *   *negative_emotions* is the most resistant: accuracy improves slowly within the category and rarely transfers to others, even with more data.




+++

[^kzqzarw428d]: Selective generalization could be defined more abstractly to refer to arbitrary properties of models (not just capabilities vs. alignment). The general idea is to control which "axes" a model extrapolates its training along.  

[^7plrhzcnf2d]: Unlike some recent work that applies steering interventions or additional fine-tuning to induce emergent re-alignment, we seek a method that can reliably prevent misalignment before-the-fact, while also learning to give bad medical advice.  

[^i0skxny54b]: We hypothesize SafeLoRA has poor performance in the sycophancy-mitigation setting because its "alignment plane" is Instruct - Base model weights. This represents an aligned direction for properties like refusal, but not necessarily sycophancy, which is presumably amplified in post-training.