Curvature-Aware Safety  Restoration In LLMs Fine-Tuning
Authors: Thong Bach
Thanh Nguyen-Tang
Dung Nguyen
Thao Minh Le
Truyen Tran
Venue: Accepted by TMLR ()

Abstract: Fine-tuning Large Language Models (LLMs) for downstream tasks often compromises safety alignment, even when using parameter-efficient methods like LoRA. In this work, we uncover a notable property: fine-tuned models preserve the geometric structure of their loss landscapes concerning harmful content, regardless of the fine-tuning method employed. This suggests that safety behaviors are not erased but shifted to less influential regions of the parameter space. Building on this insight, we propose a curvature-aware alignment restoration method that leverages influence functions and second-order optimization to selectively increase loss on harmful inputs while preserving task performance. By navigating the shared geometry between base and fine-tuned models, our method discourages unsafe outputs while preserving task-relevant performance, avoiding full reversion and enabling precise, low-impact updates. Extensive evaluations across multiple model families and adversarial settings show that our approach efficiently reduces harmful responses while maintaining or even improving utility and few-shot learning performance.

Published in Transactions Machine Learning Research (04/2020)
Curvature-Aware Safety Restoration In LLMs Fine- Tuning
Thong Bach Applied Artificial Intelligence Initiative (A212)
t t.bcc@ddekknncom edu.au
Deakin Universily
Thanh Nguyen- Taay
Department of Data Science
New Jersey Institute of Technology
Dung Nguyen
Applied Artificial Intelligence Iniliative (A212) Deakin University
Thao Minh Le Pennsylvania State University
Truyen Tran
Applied Artificial Intelligence Iniliative (A212)
Deakin University
Reviewed on OpenReview: https: // opwpweveee........ net/ forum? id= id= FSUehLhGyl
Abstract
Fine-tuning Large Language Models (L.MSs) for downstream tasks often compro- mises safety alignment, even when using parameter-cffiient methods like LoRA. In this work, we uncover notable property: fine-tuned models preserve the geometric structure of their loss landscapes concerning harmful content, regardless of the fine- tuning method employed. This suggests that safety behaviors are not crased but shifted to less influential regions of the parameter space. Building on this insight, we propose curvature-aware alignment restoration method that leverages influ- ence functions and second-order optimization to selectively increase loss on harmful inputs while preserving task performance. By navigating the shared geometry be- tween base and fine-tuned models, our method discourages unsafe outputs while preserving task-relevant performance, avoiding full reversion and enabling precise, low-impact updates. Extensive evaluations across multiple model families and ad- versarial settings show that our approach efficiently reduces harmful responses while maintaining or even improving utility and few-shot learning performance.

---

Published in Transactions Machine Learning Research (04/2026)
1 Introduction
Large Language Models (LLMs) encode safety-aligned behaviors during pretraining, but these safe- guards deteriorate during task-specific fine-tuning, a phenomenon we identify as safety alignment drift. Studies demonstrate that even minimal fine-tuning can compromise safely mechanisms, with models like GPT-3.5 Turbo becoming consistently unsafe after adaptation on just 10 adversarial examples Qi ct al. [2023b]] Bach et al. (2026). Attempts to address this issuc by modifying model behavior generally fall into two main categories, both of which suffer from inherent limitations. Behavioral unlearning methods attempt to remove undesirable knowledge responses (Cao & Yang 2015 Bourtoule et al. 2021a), but often require costly retraining or risk catastrophic for- getting. Model editing approaches aim to update factual associations or local behaviors through direct parameter intervention [Meng et al. 2022 Mitchell al. 2022) yet struggle to general- ize beyond narrow scopes or isolated prompts. To solve these issucs, we propose new direction that. treats safety behavior as an intrinsic property of the model's geometry and seeks restore alignment through curvature-aware navigation of the loss landscape.
Our key insight, supported by extensive empirical analysis (Section 2) is that models preserve notable structural properties in their loss landscapes with respect to harmful content after fine- tuning. Specifically, we observe high correlations models' responses to harmful inputs before and after fine-tuning, despite substantial divergence in other functional behaviors. This suggests that safety mechanisms remain largely preserved in the parameter space, merely shifted to less dominant regions during task-specific optimization.
This observation motivates novel approach: curvature-awar alignmmnt restoration. We leverage the preserved geometry of the loss landscape to restore safety boundaries. By employing influence functions and second-order optimization techniques, our method navigates the parameter space to increase loss on harmful inputs while minimizing impact on task performance. Our contributions include:
We identify and empirically validate a key insight: Fine-tuning preserves the geometric structure of the loss landscape for harmful content across diverse model families.
We propose a curvature-aware alignment restoration method that leverages influence func- tions and second-order optimization to suppress harmful behaviors.
We demonstrate that our approach significantly reduces harmful responses while preserving task performance, enhancing few-shot generalization, and improving robustness to adver- sarial attacks and parameter perturbations.
2 Empirical Evidence and Loss Landscape Analysis
In this section, we first present empirical evidence demonstrating high correlations between base and finc-tuned models' responses to harmful content across both parameten-officient and full finc- tuning methods, despite divergence lask performance. We then visualize and quantify this preserved geometry through loss landscape analysis, providing the foundation for our curvature- aware restoration approach.
2

---

Published in Transactions Machine Learning Research (04/2026)
Table 1: Pearson correlation coefficients between base and fine-tuned models' responses across harmful content (HEx-PHI), task--peeiific data (Dolly), general data (Alpaca), and domain-specifi datasets (CodeAlpaca,, MedMCQA, SquAD v2 ). Harmful content exhibits consistently high cor- relations (>0.85) across both LoRA and full fine-tuning, while domain-specific data shows highly variable correlations (-0.47 to 0.86). This asymmetric preservation, consistent structure for harm- ful content but variable structure for other domains, rules out simple out-of- tottof--istribution effects and validates our hypothesis that safcty mechanisms occupy a functionally distinct region in the loss landscape.
Fine-tuning
Models
Harmful
Dolly
Alpaca
CodeAlpaca
MedMCQA
SquAD v2
LLaMA-2 7B
0.992
0.056
00.55
0.551
-0.465
0.596
LoRA
LLaMA-3.1 8B
0.995
0.550
0,510
0.780
0.554
0.516
Qwen 2.5 7B
0.994
0.014
0.067
0.696
0.862
0.515
MLaM--2 7B
0.852
-0.004
0.185
0.396
0.597
0.649
Full FT
LLMMA-3.1 8B
0,990
0.535
0.508
0.746
0.499
0.366
Qwen 2.5 7B
0.941
0.526
0.129
0.363
0.012
0.312
2.1
Empirical Validation
We analyze multiple model families, measuring Pearson correlation coefficients between base and tuned models' responses across diverse data categories: harmful content (HEx-PHI Qi et al. (2023b)
a benchmark dataset, of 330 harmful instructions across policy-based categories), task-specilic data (Dolly testset Databricks (2023), 200 examples), general data (Alpaca testset Taori al, (0023) 200 examples), and domain-specific datasets including code generation (CodeAlpaca). med- ical reasoning (MedMCQA (Pal et al. 2022)) and question answering (SquAD v2 (Rajpurkar et al. 2018 We evaluate both LoRA and full fine-tuning to verify that our findings generalize across methodologies.
These correlations quantify how consistently models respond to the same inputs before and after fine-tuning. For cach dataset D, we compute the Pearson correlation coefficient:
Exep(Lhaso(x)) - Lbaas (Leuued((x) - IIumed)
E (Ibaas (z) - I )2 ExED (Ituned(2) (x) Ituned)?
where (x) (x) and (x) are the cross-entropy losses of the base and finc-tuned models example x, and Thase and Lnuned are their respective mean values across dataset D. Higher correla- tion indicates the fine-tuned model maintains similar response behavior to the base model, despite parameter changes. By comparing correlations across different imput categories, we can detect whether safety-rclovant behaviors remain intact despite changes to task-spocific capabilities.
Our analysis reveals three key insights:
1. Consistent safety structure presorvation across finc-tuning methods: In Table I harmful content shows consistently high correlation (0.85 0.99) across both LoRA and full fine-tuning methods and all three model families. Full fine-tuning preserves loss structure slightly less than LoRA (c.g., 0.852 vs 0.992 for LLaMA-2 7B), which aligns with intu-
3

---

Published in Transactions Machine Learning Research (04/2026)
LLama-3-8B Loss Distribution
Dolly
8.54
Alpaca
8.09
HEX-PHI
Average Loss
1.95
1.82
1.24
1.08
Base Model
Tuned Model
Figure 1: Average loss comparison across base and fine-tuned LLama-3 8B Instruct models for three datasets: Dolly (lask-specific), Alpaca (general), and HEx-PHI (harmful). Harmful content consistently exhibits higher loss compared to benign content in both model states, showing that harmful content consistently lics in distinct and preserved region the loss landscape.
ition since full fine-tuning modifies all parameters rather than a low-rank subspace, yet preservation remains strong (2 0.85).
2. Variable structure for domain-specific data: In contrast, domain-specific datasets exhibit highly variable correlations ranging from -0.47 to 0.86 across models and finc- tuning methods. Although some datasets show moderate correlations (e.g., CodeAlpaca 0.78 for LLaMA-3.1, McdMCQA 0.86 for Qwen), this variability contrasts sharply with the consistently high correlations for harmful content (0.85 to 0.99). If preservation were merely titt-f--distribution effect, all dissimilar data would show similarly high correlations. The observed variability confirms that safety behaviors occupy functionally distinct region rather than reflecting distributional distance alone.
3. Distinct safety regions in loss landscape: 3.1-8B-Ina Instruet on these data. Generally, harmful content consistently generates higher loss values (8.54 and 8.09) compared to benign content (1.82-1.95 and 1.08-1.24) in both model states, suggesting potential separation between harmful and task-relevant regions in the loss landscape. More detailed loss analysis will be presented in Appendix C.5
In Figure measure the loss of LLaMA-
Based on these findings, we state our hypothesis: safety behaviors exist in a functionally distinct region of the loss landscape that remains largely undisturbed by task-specific fine-tuning, regardless of whether parameter-effcient or full fine-tuning is employed. Therefore, developing a targeted restoration method to recover safety behaviors without compro- mising useful task capabilities feasible.
2.2
Loss Landscape visualization
To further support our hypothesis, we visualize the loss landscapes of both the base and fine-tuned models using 3D projection technique. Rather than sampling arbitrary directions in parameter space, construct perturbation directions informed by gradients computed on harmful and benign
4

---

Published in Transactions Machine Learning Research (04/2026)
inputs. Specifically, we focus on attention and MLP layers, which most strongly influence model behavior. For each model, generate two approximately orthogonal perturbation vectors (dl and d2) and evaluate the model's loss across grid (20 20) of perturbation magnitudes. We create this grid by varying coefficients X and X2 within the range [-0.01, 0.01] and applying the perturbation 0 pertuureed = Ooriginal + A + A2d2 to the model parameters. At each grid point, we compute the loss using a consistent set of 32 validated samples, resulting in a 3D surface where the r-axis and y-axis represent perturbation magnitudes along each direction, and the 2-axis shows the corresponding loss value. Full implementation details are provided in Appendix B.3
Harmful Base Landscape
Harmful Tuned Landscape
Harmful Difference (Struct Diff: 1.46%)
18
16
16
Difference
14
Loss
14
-0.01
-0.01
p.c0
0.00
+
Benign Base Landscape
Benign Tuned Landscape
Benign Difference (Struct Diff: 20.37%)
20
20
18
18
18
Difference
Loss
10
-0.01
-0.01
-0.01
-0.0 01
-0.01
-0.01
00
M
+
N>
4
M
0.01 0.0
0.01 0.01
0.01 0.01
Figure 2: 3D loss landscape visualization for LLama-3 8B Instruct using gradient-informed direction projection (Section 2.2h. The top row shows the loss landscape of harmful content (HEx-PHI), while the bottom row shows for general data (Alpaca). Comparison between base (left) and fine-tuned (middle) models reveals preserved topological features for harmful content (structural difference: 1.46%), while general data landscapes undergo substantial transformation (structural difference: 20.37%) These quantitative measures of landscape change confirm that safety-relevant regions remain largely undisturbed during tasc-specific fine-tuning. Detailed percentile statistics are pro- vided in Appendix C.7
The 3D plot in Figure reveals clear evidence for our hypothesis. The loss landscape for harmful content maintains remarkably similar topological features between base and fine-tuned models, with consistent valleys, peaks, and curvature characteristics. We quantify this structural preservation using a corrolation-basel metric: StructDiff = (1 - |corr(V2L ('Leaned)) 100%, whore V the Laplacian of the loss landscape. This metric captures differences in curvature patterns rather than absolute loss values, revealing only 1.46% structural difference for harmful content despite fine-tuning.
5

---

Published in Transactions Machine Learning Research (04/2026)
In contrast, the loss landscape for general-purpose data changes significantly, exhibiting 20.37% structural difference in both global geometry and local minima positions. This visualization provides direct confirmation that fine-tuning primarily affects task-specific regions of the parameter space while leaving safety-relevant regions structurally maintained, creating natural opportunity for targeted safety restoration. Additional results on loss landscapes for LoRA fine-tuning are available the Appendix B.3
These visualization results explain the high correlation coefficients documented Section 2.1 and establish foundation for our alignment restoration approach. The idea is to identify and lever- age preserved landscape features and use these to navigate toward parameter configurations that maintain task performance while reinforcing safety boundaries.
3
Curvature-awart alignment restoration
The preserved loss landscape geometry identified in Section 2.1 has a direct implication: safety restoration can be achieved through small, precise parameter updates rather than large-scale re- training. If fine-tuning had destroyed safety-relevant structure, restoring appropriate refusal behav- ior would require either reconstructing these mechanisms from scratch or making large parameter changes that risk catastrophic forgetting on the task. The high correlation we observe indicates instead that safely behaviors occupy functionally distinct region that remains accessible in param- eter space. This enables constrained optimization approach where can restore safety within tight utility bounds.
We now introduce our curvature-aware alignment restoration approach, which provides a principled way to steer a fine-tuned LLM back toward the safety behavior encoded in its base model while preserving desirable task-specific knowledge.
3.1 Problem Formulation and Optimization Approach
Let us define Obase as the parameters of the pretrained, safe base model, and Otuned as the parameters of the fine-tuned model. We use two distinct datasets:
retain set containing benign, task-rclevant examples where performance should be preserved, and forget set containing potentially harmful examples where safety alignment should be restored.
For both datasets, we employ the standard autoregressive language modeling loss:
[m|
L(x; 0) log (I
(1)
where represents an input sequence and (x( is the model's predicted probability for token Iy given preceding tokens.
Our goal is to update Oraned toward point Oupdated that preserves Lretain (the loss retain set) while increasing Liorget (the loss on forget set). We formulate this as constrained optimization problem:
max Ltorget (0) s.t. Terotain (0) S Trretain Aretain(Ounnel (Oranea) +
(2)
6

---

Published in Transactions Machine Learning Research (04/2026)
where is a small positive scalar allowing limited degradation in retain set performance. Based on extensive empirical validation, established 0.1 default constraint threshold, ensuring the recovered model maintains task performance within an acceptable margin of the fine-tuned baseline.
This formulation can be theoretically justified through second-order Taylor approximation of the retain loss around funned:
1 Lyetaiu (ftuned + A0) Lretain (Ftuned) + V Al + _00 Hretain A0
(3)
Under this approximation, the influence function update provides the steepest descent direction for Ltorget in the Riemannian geometry defined by Hrotain Amari
1998):
A Anfluence = arg max L Al s.l. | Hentain 56 A6
(4)
Here, 8 > 0 defines the allowable trust region radius with respect to the local geometry of the retain loss, measured via the Mahalanobis norm || ( T This parameter is directly related to the constraint threshold e in Equation smaller values ensure updates remain in regions where the quadratic approximation is valid, thereby helping satisfy the e-bounded relain loss constraint. Inluitively, this constraint ensures that the updale direction increases the forget loss without significantly increasing the retain loss, measured by its local curvature, Solving this constrained optimization yiclds the steepest ascent direction for Lorret under Riemannian metric induced by Hretain-
Directly solving Equation A may be computationally expensive. Therefore, we adopt a tractable approximation based on influence functions, as shown below:
Afufluence Hretain V Ltorget (fenned)
This approximation can be interpreted as the unconstrained solution to Equation where the trust region constraint is relaxed. Specifically, Equation 5 represents the steepest ascent direction for Liorgtt under the curvature geometry of the retain set, without explicitly enforcing a norm constraint. However, since we have removed the explicit trust region constraint, we need to com- pensate by adding practical safeguards. We achieve this through step scaling (controlling update magnitudes) and L-BFGS curvature filtcring (cnsuring numerical stability), detailed in Appendix B.
In practice, we construct H using a low-rank L-BFGS Liu & Nocedal 1989) approximation that incorporates curvature information from both the retain set and a subset of the forget set. This hybrid construction enables the trust region balance retention of task-specific knowledge with awareness of harmful content boundaries, resulting in more effective influence updates. We discuss implementation details and ablation results in the Appendix B.1
7

---

Published in Transactions Machine Learning Research (04/2026)
3.2 Practical Implementation
Directly computing and inverting the Hessian matrix Hretain for modern LLMs is computationally intractable due the enormous parameter space. To address this challenge, we implement two key techniques:
(1) Parametor-Efficien Fine-Tuning. We apply our method within the Low-Rank Adaptation (LoRA) framework. This reduces the dimensionality of the Hessian matrix to just the trainable parameters, making curvature estimation feasible.
(2) Approximate Hessian Inversion. We employ L-BFGS (Limited-memory Broy- Solanar--Saaanooo to efficiently approximate H reducing computation to (mp)) where m is the memory size and is the parameter dinensionaliity.
This quasi-Newe Newton method builds an approximation of the inverse Hessian through successive low- rank updates, avoiding explicit matrix inversion.
4 Experimental Results
In this section, empirically evaluate our curvature-aware safety restoration method across di- verse architectures and benchmarks. Our experiments investigate three core questions: (1) How well does our method restore safety compared to state-of-the-art approaches? (2) Does it preserve model utility and adaptability? (3) How robust is the restored alignment to adversarial attacks and parameter perturbations? We outline our experimental setup archittccccsssrsss fine-tuning proto- col, and baselines before presenting results on safety performance, utility preservation, in-context learning, and robustness to both prefilling attacks and weight-space perturbations. Overall, method consistently restores safety without degrading task performance, addressing central chal- lenge in fine-tuning LLMs.
4.1 Experimental Setup
Base LLMs We evaluate curvature-awars alignment restoration method three represen- tative large language models that span different. architectures and training paradigms: L.Lam-22 7B Chat, LLama 3.1 8B Instruct, and Qwen 2.5 7B Instruct. These models were seleeted for their widespread adoption in the research community, comparable parameter scales (7-8B parameters), which allow us to assess how our method generalizes across model families with different inherent safety characteristics.
Finc-tuning Protocol To maintain computational efficiency while preserving model quality, we implement Paraaneler-Bficieeu Fine-Tuning (PEFT) via Low-Rank Adaplation (LoRA). Across all experiments, we utilize a consistent configuration with rank = 32 and learning rate = 2 x 10-4, We apply LoRA adapters to the query and value projections in attention layers, following the default configuration used in the PEFT library Mangrulkar et al. (2022).
For our primary instruction-tuming dataset, we employ Dolly, a diverse collection of 15,000 human- generated instruction-responie pairs spanning multiple domains. We fine-tune each model for 1
8

---

Published in Transactions Machine Learning Research (04/2020)
epoch with batch size of 128 examples, using the AdamW optimizer. All experiments were conducted on 1 NVIDIA H100 GPUs with 80 GB memory.
Baseline Methods We compare our curvature-aware alignment restoration approach against several state-of-the-art methods for rafety-preserving fine-tuning:1) Vanilla Fine-tuning Hu et al. (2022): Standard LoRA fine-tuning without any safety preservation mechanisms, serving as primary control. (2) Vaccine Huang et al. (2024); A preventative approach that operates during the initial alignment phase by adding crafted perturbalions to hidden embeddings, making the model robust against harmful perturbations that may be introduced during subsequent fine- tuning. (3) Safc LoRA Hsu et al. (2024): A data-free, training-free approach that preserves safety aligument during Luning by projecting LoRA weight updates onto an aligument subspace defined by the difference between aligned and unaligned model weights, applying this projection only when updates deviate significantly from the alignment direction. (4) SaLoRA Li et al. (2025a): A technique that preserves safety alignment during LoRA fine-tuning by introducing a fixed safety module that projects new feat ures to subspace orthogonal to original safety feat along with task-specific initialization for trainable parameters.
For all baseline methods, we follow the hyperparameter settings recommended in their respective papers, adapting only when necessary to maintain fairness in the comparison.
4.2 Safety Evaluation
We evaluate model safety on AdvBench, containing 520 adversarial prompts designed to elicit unsafe responses. We allocate 138 samples for constructing the safety matrix required by SaLoRA and reserve the remaining 382 samples for evaluation. Our primary safety metric is the harmful response nate (HRR), calculated as the percentage of evaluation samples cliciting unsafe responses. For a comprehensive assessment, we employ both LLama-3 Guard as an automated safety evaluator and human review to validate the quality and accuracy of safety judgments, ensuring a more reliable evaluation of model safety across different methods.
Safety performance Table 2 demonstrates our curvature-aware alignment restoration method achieves superior safety results across model familics. For Llama-3.1 8B, our approach reduces HRR to just 3.0%, significantly outperforming both SaLoRA (8.1%), Vaccine (21.3%), and Safe LoRA (11.0%). For Qwen 2.5 7B, we achieve a remarkable 1.5% HRR, substantially lower than all fine-tuning methods including SaLoRA (3.4%). With Llama-2 7B, our method successfully restores complete safety aligmment (0% HRR), matching the excellent performance of SaLoRA and Safe LoRA on this model.
Task Performance and Utility Evaluation. To show that safety improvements do not com- promise task performance, evaluate models on both the original fino-tuning task (Dolly) and four diverse zero-shot lasks: ARC-Challenge (commonsense reasoning), GSMSK (mathematical reason- ing), ToxiGen (toxicity detection), and TruthfulQA (factual consistency). The column 'Eval' in Table shows that our method maintains a comparable performance to other safcty techniques in the original fine-tuning task, with scores 1.3, 1.4, and 1.2 in the three model families.
For broader utility, our approach maintains strong performance across tasks. On Llama-3.1 8B, method achieves the highost scores on GSM8K (76.5) and TruthfulQA (43.6) while maintain-
9

---

Published in Transactions Machine Learning Research (04/2026)
Table 2: Comparison of safety restoration methods across three model families. HRR (Harmful Response Rate, lower is better) measures safety on AdvBench, while Eval shows performance on fine-tuning dataset (average cross-entropy loss across all examples the Dolly test set). Utility metries include four zero-shot tasks: ARC-Chall Chaleneee (ARC-C), GSM8K, ToxiGen, and TruthfulQA. Our curvafure-aware approach achieves best safety across all models while maintaining competitive task performance. Bold indicates best method, underline indicates second-best for each metric within model family.
Utility f
Models
Methods
Eval
HRR
ARC-C
GSM8K
ToxiGen
TruthfulQA
Base
1.9
1.4
52.0
75.2
53.3
45.5
LoRA
1.2
25.5
51.2
72.4
44.9
39.0
Llama-3.1 8B
Vaccine
1.3
21.3
44.3
39.5
43.4
34.1
SaLoRA
1.2
8.1
52.3
75.7
49.3
41.8
Safe LoRA
51.1
75.6
48.7
42.0
Ours
13
3.0
51.8
76.5
46.0
43.6
Base
3.6
53.0
76.4
57.2
56.3
LoRA
1.2
24.7
55.0
60.2
57.2
44.5
Qwen 2.5 7B
Vaceine
1.2
19.3
54.6
74.3
57.9
44.5
S
1.2
3.4
55.0
69.5
57.2
49.2
Ours
1.4
1.5
54.2
75.1
57.1
53.3
Base
2.5
0.0
43.3
20.1
52.9
37.2
LoRA
1.1
21.4
44.4
19.6
44.7
32.3
Llama-2
Vaccine
1.1
16.7
42.6
11.6
41.1
31.7
SaLoRA
1.1
0.0
45.9
23.6
49.5
34.7
Safe LoRA
1.2
0.0
45.6
21.5
43.8
33.1
Ours
1.3
0.0
44.7
22.1
51.7
36.8
ing competitive ARC-C performance (51.8). For Qwen 2.5 7B, we obtain the best performance TruthfulQA (53.3) and GSM8K (75.1). With Llama-2 7B, approach achieves the highest, TruthfulQA (36.8) and ToxiGen (51.7) scores. This demonstrates
curvature-aware method ef- fectively balances safety restoration with preservation of diverse reasoning capabilities. We further validate in Appendix C.2 that our method generalizes to full fine-tuning, achieving comparable safety restoration (HRR reductions of 57-75%) while maintaining task performance, confirming that the preserved loss landscape geometry enables effective restoration regardless of fine-tuning methodology.
4.3
Robustness Evaluation
We evaluate the robustness of our safety alignment restoration through two distinct experiments: resistance to adversarial prefilling attacks and stability under parameter perturbations.
4.3.1
Prefilling Attack Resistance
This experiment assesses the robustness of our method against anference-time attacks that exploit shallow safcty alignment vulnerabilities in LLMs Qi ct al. (022).
10

---

Published in Transactions Machine Learning Research (04/)026)
71.9%
69.3%
63.0%
47.4%
40
20
Base
LoRM
Vacdine
Ours
Safe LoRA SaLoRA
Figure Atlack success rales (the lower the better) for prefilling attacks across different alignment restoration methods Llama-3.1 8B evaluated AdvBench. Our curvature-aware approach achieves 63.0% ASR, significantly outperforming bascline LoRA (78.4%) and other safety methods, while approaching the robustness of the base model (47.4%).
Table VISAGE scores measuring safety basin robustness. Higher scores indicate more robust safety basins resistant to parameter perturbations. Our approach achieves 56.1, substantially out- performing all baselines.
Method
VISAGE Score
LoRA
21.1
Vaccine
28.8
SaLoRA
32.1
Ours
56.1
Experimental Sctup We use 382 adversarial prompts from AdvBench (used in Section 4.2 to simulate a prefilling attack. Following prior work Qi et al. (2024); Andriushchenko et al. 2024 each input is prepended with four non-refusal tokens, which are designed to bypass the model s standard safety refusal mechanisms
We evaluate models fine-tuned with five different methods: vanilla LoRA, Vaccine Huang al, (2024), Safe LoRA Hsu et al. (2024), SaLoRA [i et al. (2025a), and our curvature-awars approach. We report attack success rate (ASR) as the percentage of inputs that lead to harmful comple- tions (lower is better).
Results Analysis As shown in Figure 3 our method achieves a lower ASR (63.0%) than all other alignment restoration baselines. Compared to standard LoRA fine-tuning (78.4%), our approach yiclds a 19.6% relative reduction in attack success, and also demonstrates improved robustness over Vaccine, Safe LoRA, and SaLoRA. These findings highlight the effectiveness our curvature-aware approach in mitigating shallow alignment vulnerabilities and preserving safety under adversarial prompting.
11

---

Published in Transactions Machine Learning Research (04/2026)
100
60
ASR
SaLoRA
Vaccine
LoRA
Ours
-04
0.2
0.4
Distance
Figure 4: Safety landscape visualization showing Attack Success Rate (ASR) across parameter perturbations for different methods on Qwen 2.5 7B. Our approach maintains significantly wider and deeper safety basin, with near 0% ASR at the origin and slower degradation with distance.
4.3.2 Parameter Perturbation Stability
We further evaluate the robustness of aligmment restoration methods under parameter perturbations by analyzing the safety basin Peng et al. (2024a), which refers to the region in parameter space where the model continues to behave safely despile small changes.
Experimental Setup We test the Qwen 2.5 7B Instruct model fine-tuned with four methods: vanilla. LoRA (as the baseline), and three safety alignment techniques: Vaccine Huang et al. (2024). SaLoRA Li et al. (2025a), and our curvature-aware approach. For each model, we apply parameter perturbations along randomly sampled directions, varying the perturbation magnitude within the range 1-..5,..33 0.5).
We compare the attack success rate (ASR) at each perturbation level and compute the VISAGE score Peng al. 20244)), which measures the average safety margin across all directions. A higher VISAGE score indicates that the model remains safe under wider range of parameter variations.
Results Analysis As shown in Table and Figure our curvalure-aware method achieves the highest VISAGE score (56.1), substantially outperforming SaLoRA (32.1), Vaccine (28.8), and LoRA (21.1). The safcty landscape visualization confirms that our method maintains broader and deeper safety basin, with nearly zero ASR at the origin and slower degradation as perturba- tion magnitude increases. These results indicate that our method produces more resilient. safety alignment, offering stronger robustness to parameter noise and adaptation.
5 Conclusion
We present curvature-aware alignment restoration framework that addresses the challenge of safety degradation in fine-tuned LLMs. Our approach builds on the empirical observation that the loss landscape associated with harmful content remains structurally preserved after task-specific
Details of the non-refusal token construction provided Appendix B.4
12

---

Published in Transactions Machine Learning Research (04/2026)
fine-tuning, Leveraging this geometric insight, we apply influence functions and second-order op- timization to selectively increase loss harmful inputs while maintaining task performance. Ex- tensive evaluations across multiple model families and adversarial settings demonstrate that our method consistently reduces harmful responses while preserving few-shot generalization and utility on downstream tasks.
Discussion While our work focuses on safety alignment restoration, the proposed curvature-aware framework mathematically general and may extend other scenarios involving conflicting ob- jectives. The key insight enabling our approach is the geometric separation between lask-relevant regions in the loss landscape. For safety, empirically observe that harmful content occupies structurally distinct region that remains preserved after fine-tuning. This property may not univer- sally hold for arbitrary task pairs, when two objectives share substantial parameter dependencies, small curvature-constrainee updates may be insufficient to optimize one without degrading the other. Investigating which objective pairs exhibit favorable geometric separation and developing metrics to predict this property priori represents a promising direction for future rescarch.
References
Shun-ichi Amari. Natural gradient works efficiently in learning. Newral computation, 10(2):251 276, 1998.
Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking leading safety- aligned iims with simple adaptive attacks. arXiv preprint, 2024.
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by single direction. arXiv preprint arXim:2406. 11717, 2024.
Thong Bach, Dung Nguyen, Thao Minh Le, and Truyen Tran. Rethinking deep aligmment through the lens of incomplete safety learning. In Proceedings of the AAAT Conference on Arlificial Intelligence, volume 40, pp. 30005 30012, 2026.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. NeurIPS, 2023.
Fazl Barez, Tingchen Fu, Ameya Prabhu, Stephen Casper, Amartya Sanyal, Adel Bibi, Aidan O'Gara, Robert Kirk, Ben Bucknall, Tim Fist, et al. Open problems in machine unlearning for ai safety. arXiv preprinl #xXiv2250104955 2025.
Lucas Bourtoule, Varun Chandrasckaran, Christopher A Choquette-Choo, Hongsheng Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 141-159. TEEE, 2021a.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Chog, Hongsheng Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In IEEE Sympo- sium on Security and Privacy, pp. 141 159. IEEE, 2021b.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEFE Symposium on Security and Privacy, pp. 463-480. IEEE, 2015.
13

---

Published in Transactions Machine Learning Research (04/2026)
Tianle Chen et al. A comprehensive survey on machine unlearning: Frameworks, algorithms, challenges, and opportunitiss. arXiv preprint arX7::234.002744 2023.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint urXiv:2310.122 12773, 2023.
Databricks.
Databricks
dolly:
Open instruction-followinn large lan-
guage
model.
https://www www.:aattbbicc... wattaaaiicks.co//lo////00000///. com/bllg//022/04/12/
11mm
2023.
Accessed: 2025-
05-11.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Forget unlearning: Efficiently removing training data from neural networks. In European Conference on Computer Vision, pp. 191-207. Springer, 2020.
Chia-Yi Yi Hsu, Yu-Lin Tsai, Chih-Hsun Lin, Yu Chen, Chia-Mu Yu, and Chun-Ying Huang. Safe lora: The silver lining of reducing safety risks when finetuning large language models. Advances in Neural Information Processing Systems, 87:65072-65088 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. arXiv preprint arXim:2106.m 06.09685 2021.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICL,R, 12)):3 2022.
Tiansheng Huang, Sihao Hu, and Ling Liu. Vaceine: Perturbation--waree alignment for large lan- guage model. ar preprint arXiv:2402. 001000000 2024.
Dang Huu-Tien, Trung-Tin Pham, Hoang Thanh- Thanh-Tunn, and Naoya Inoue On effects of steering latent representation for large language model unlearning. arXiv preprint arXiv:2408.0022 2024.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barrinsk, et al. Overcom- ing calastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521 3522336,,,, 2017.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. International Conference on Machine Learning, 2017.
Mingjie Li, Wai Man Si, Michael Backes, Yang Zhang, and Yisen Wang. Salora: Safety-alignment preserved low-rank adaptation. arXiv preprint XxXt:22501.0766: 2025a.
Nathanicl LI, Alexander Pan, Anjali Gopal, Summer Yuc, Daniel Berrios, Alice Gatti, Justin Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The windp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:203.00211 2024.
Shen Li, Liuyi Yao, Lan Zhang, and Yaliang Li. Safety layers in aligned large language models: The key to LLM security. In The Thirteenth International Conference on Learning Representations, 2025b. URL ottpss//gppaaerrree.eeee//ooooop
14

---

Published in Transactions Machine Learning Research (04/2026)
Chris Liu, Yaxuan Wang, Jefirey Flanigan, and Yang Liu. Large language model unlearning via sabbedding-corruppae prompts. Advances in Neural Information Processing Systems, 37:118198- 118266, 2024.
Dong C Liu and Jorge Nocedal. On the limited memory BFGS method for large scale optimization. Mathematical programming, 45(1-3):503 528, 1989.
Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalic Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, et al. Rethinking machine unlearning for large language models. Nature Machine Intelligence, pp. 1 1-14, 2025.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. Tofu: A task of fictitious unlearning for llms. arXiv preprint ar 201::2401.06121 2024.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Youncs Belkada, Sayak Paul, and Benjamin Bossan, Peft: State-of-the-art parameter-effiient fine-tuning methods. https://githu: com/
huggingfacce/pett/ 2022.
Kevin Meng, David Bau, Benjamin Andrus, and Yonatan Belinkov. Locating and editing factual associations in gpt. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pp. 14943 14957, 2022.
Eric Mitchell, Charles Lin, Antoine Lin, Chelsea Finn, Christopher Zhang, Christopher D. Man- ning, and Dawn Song. Fast model editing at scale. In International Conference on Learning Representations (ICLR), 2022.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feodback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmeqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Gerardo Flores, George H Chen, Tom Pollard, Joyce C Ho, and Tristan Naumann (eds.), Proceedings of the Conference on Health, Inference, and Learning, volune 174 of Proceedings of Machine Learning Research, pp. 248 260. PMLR, 07-08 Apr 2022. URL https://prroceednng mlr mar.prese ptt..p//es//// pal22a.html
Sheng Yun Peng, Pin-Y Yu Chen, Matthew Hull, and Duen Horng Chau. Navigating the safety land- scape: Mcasuring risks in finetuning large language models. arXiv preprint vrXiv:2405.17377 2024a.
ShengYun Peng, Pin-Yu Chen, Matthew Hull, and Duen Horng Chau. Navigating the safety land- scape: Measuring risks in finetuning large language models. arXiv preprint Skk::2405.1777,, 2024b.
Xiangyu Qi, Ashwinee Panda, Kaileng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should be made more than just few tokens deep. Xiv preprint or 2Xix:2406.00944 2023a.
15

---

Published in Transactions Machine Learning Research (04/2026)
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromiscs safcty, even when users do not intend to!, 2023b.
Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should be made more than just a few tokens deep. arXiv preprint 0al:2406.05446 2024.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annuel Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784-789, Mclbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-2111 URL
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp.
2015.
Rohan Taori, Ishaan Gulati, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Tatsunori B. Hashimoto, and Percy Liang. Stanford alpaca: An instruction-follwiii llams model. https:
//crfm. stanf edu/ wdtpeee.//eeeeeeeeee 2023/03/2/22/ 13/ haapaa..mmml 2023. Accessod: 2025-05-11.
Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. Advances in Newral Information Processing Systems, 10--00422--157777 2024.
Xiaojian Yuan, Tianyu Pang, Chao Du, Kejiang Chen, Weiming Zhang, and Min Lin. A closer look at machine unlearning for large language models. arXiv preprint arXiv:2410..m 10.08109, 2024.
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey, arXiv preprint or xrxi:22008.1 10799, 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuming Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, al. Lima: Less is more for alignment. Advances in Neurol Information Processing Systems, 36:55006-55021, 2023.
Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023.
A Related Work
A.1 Safety and Robustness in Large Language Models
Safety Alignment in Large Language Models Ensuring the safety of large language models (LLMs) has become a central research challenge as their deployment expands into high-stakes do- mains. Models pretrained on vast internet corpora often internalize harmful behaviors, prompting the development of post-training alignment methods such as Reinforcement Learning from Human Feedback (RLHF) Ouyang et al. ) Dai et (202)) and supervised instruction tuning Bai et al. Zhang et al. (2023) Zhou et al. (2023). Despite their effectiveness, these safety mechanisms
(2023):
16

---

Published in Transactions Machine Learning Research (04/2026)
remain fragile, with studies showing that fine-tuning aligned models on downstream tasks can lead significant safcty degradation Qi ct al. [2023a bj. Concurrently, parameter-efficcan fine-tuning (PEFT) techniques have emerged to adapt large models with minimal updates. Low-Rank Adapta- tion (LoRA) ) Hu et al. (2021) | has become particularly popular by constraining updates to low-rank matrices applied to the model''s weight matrices, significantly reducing trainable parameters while maintaining performance. Building on LoRA's efficiency, several piett--preserving fine-tuning ap- proaches have been developed to address safety degradation. Vaccine Srocine/Huang et al. (2024) introduces adversarial perturbations during training to immunize models against unsafe queries. SafcLoRA Hsu et al. (2024) extends LoRA by projecting weight updates onto an aligument subspace defined by the difference between aligned and unaligned model weights. Similarly, SaLoRA Li et al. (2025a) preserves safety during LoRA fine-tuning by introducing fixed safety module that projects new features to a subspace orthogonal to original safety features. However, these approaches typically either compromise task performance or rely on heuristic projections ions without geometric insights. Recent findings suggest that safety-relevant behaviors occupy distinct, resilient regions in the loss landscape Peng al. (2024b), indicating that geometric propertics of the paramcter space could enable more robust aligument preservation Li el al. (2025b); Arditi al. (2024). Our work builds on these geometric insights by employing influence functions and curvature-aware optimization to restore safety alignment without sacrificing task performance. Unlike previous approaches that use heuristic constraints, our method directly leverages the preserved structure of the loss landscape to navigate toward parameter configurations that enhance safety while maintaining model capabilities.
A.2 Unlearning and Parameter Space Geometry
When harmful behaviors emerge in LLMs following fine-tuning, machine unlearning offers a prin- cipled framework to selectively remove them Huu-Ticn al. (2024): Li ct al. (2024); Liu al.
(2024). Influence function-based unlearning Koh & Liang] (0017); Chen al. (2023 Yuan al.
(2024) estimates the gradient direction that increases the loss on undesired examples while mini-
mally impacting desired behaviors, effectively reversing their influence in parameter space Liu et al.
(2025): Barez et al. (2025). Other approaches such as SISA Bourtoule et al. (2021b) or trust-region
unlearningGGolatar et al. (020)) offer certified deletion by retraining from strategically partitioned checkpoints. However, these methods often incur high computational costs or suffer from degraded generalization. In parallel, curvature-wuuee optimization techniques have been explored to control model drift during fine-tuning Elastic Weight Consolidation Kirkpatrick et al. (2017) and similar continual learning strategies use curvature estimates (c.g., Fisher information) to constrain updates in directions that preserve previously acquired capabilities. Trust-region policy optimization [Schul- man et al. 2015) and natural gradient methods Amari (1998) apply second-order constraints to keep parameter updates within functionally safe neighborhoods. Our method unifies these perspec- tives by framing safcty restoration as a sccond-order constrained optimization problem over the loss landscape. We employ influence functions and T-BFGS-based curvature estimation to direct updates that increase loss on harmful content while staying within a trust region defined by the retain set, enabling scalable and stable safety restoration in fine-tuned LLMs.
unlearning Golatkar
17

---

Published in Transactions Machine Learning Research (04/2026)
B Detailed Implementation
B.1 Aware L-BFGS Construction
Data Partitioning. To avoid overlap between curvature estimation and influence-based safety restoration, we partition the HEx-PHI dataset Qi et al. (2023b), which contains 330 adversari- ally constructed harmful prompts. For L-BFGS curvature pair construction, we total of 256 examples (64 examples from each of four batches) selected from HEx-PHI as part of the mixed curvature set Diure Separately, to compute the forget loss Ltorget, we reserve 50 held-out exam- ples from the remaining HEx-PHI data (named as Diorget). These examples are not used during curvature approximation and are exclusively employed to evaluate or guide updates that suppress harmful generations. This partitioning ensures clean separation between curvature modeling and influenco-bseed optimization targets.
To approximate the inverse Hessian H in Equation construct low-rank L-BFGS history over LoRA parameters using carefully designed curvature buffer. This buffer integrates informa- tion from three strategically selected disjoint datasets: a subset of the forget set D (HEx-PHI dataset) and two distinct subsets of the retain set D(1) Pretain: D(2) Petain (derived from the fine-tuning dataset). This multi-dataset approach ensures the captured curvature spans both safety-critical and task-aligned directions in parameter space.
Each L-BFGS pair (e) computed via gradicnt accumulation over batches of 64 examples. Our empirical analysis reveals that just 10 high-quality pairs sufficiently approximate the local curvature structure for effective influence updates. We allocate these pairs approximately equally across the three datasets, requiring minimum of 192 examples per set. To enhance curvature diversity, we employ varying learning rates (0.001, 0.002, 0.005) across optimization steps. A trust region & constrains update magnitudes by scaling steps to a bounded norm, while a reduction ratio P: determines step acceptance and dynamically adjusts 8,.
To ensure robust and numerically stable curvature estimation, we implement several filtering mech- anisms: (1) rejecting curvature pairs with insufficient curvature ((8..3)) 10-9), (2) normalizing St, Ve vectors to unit norm before storage, (3) applying adaptive damping when nogative curvature is encountered, and (4) excluding pairs with degenerate step gradient norms. These safeguards collectively prevent ill-conditioning in the inverse Hessian approximation.
In practice, we recompute curvature pairs at the beginning of cach safety restoration itcration. Our experiments demonstrale that just three such iterations sulfice for effective aligument restora- tion across all evaluated model architectures, and this 3-step procedure is consistently employed throughout experimental validation.
18

---

Published in Transactions Machine Learning Research (04/2026)
Algorithm 1 Curvature-Aware L-BFGS History Construction
1: Input: Model fo, datasets Porget D(12) LoRA parameters LORA
2: Initialize emply history lists: S,y
3: Set initial trust radius 6 = 0.05
4: for t= to T do
5: Sample batch B₁ from one of the datasets (round-robin)
6: Compute initial loss Linit and gradients
7:
Propose step de -9t and rescale to ||de]|| S
8: Save Ot, apply step to get 0c+1
9:
Compute final loss Cimal and gradients 9t+1
10:
Compute actual and predicted reduction, ratio Pt
if Pt < 0.25 then
12:
Shrink trust radius: 5 < 0.58
13:
Revert to 0, continue
14:
15:
else if Pt > 0.75 then Expand trust radius: + 1.56
16:
17:
end if
Compute 8, = 0141 8e, Vt It+1 9r
if (st, vt) then
20:
Normalize 8t, yt. add to s,y
21: end if
22:
end for
23:
return S,Y
B.2 Influence Update Mechanism
To prevent overcorrection and preserve generalization capabilities of the model during aligmment restoration, apply L2 regularization to the influence-based update direction A0. At each itera- tion, the update to the LoRA parameters is computed as:
Onew = Ocuned + 1) A0 A
where:
7) is the update scale (determined by fixed multiplier small grid search),
A is the L-BFGS-projected gradient direction (from Appendix B.1
the L2 regularization weight, progressively anncaled across iterations (c.g., X t 0.95-A).
Unlearning Objective The harmful gradient V Ltorget is obtained by evaluating the model on
the forget set using a cross-entropy loss:
Liorget CE(Ú.))
19

---

Published in Transactions Machine Learning Research (04/2026)
Algorithm 2 Safety Restoration via Influence Update
I: Input: LoRA parameters 0, L-BFGS history ($,3), forget dataset. Diorget, slep size 1/ L. weight A
2: Initialize accumulated gradient g 0
3: for each batch B in Diorget do
4: Compute loss Liorget CFF(f((B))
3: Compute gradient V Liorget and accumulate into g
6: end for
7: Project through inverse Hessian: A +- - -H-19 using I--BGGS (see Appendix B.1
8: Normalize A <- A0/1
9: for each parameter 0, in LoRA:
10: Extract corresponding slice Ad;
Compute L2-regularized update:
0, 0, 11 A0; A- A.
return Updated parameters 0
B.3 Loss Landscape Visualization Implementation
This section provides a detailed description of our methodology for visualizing the loss landscapes of language models before and after fine-tuning.
Gradient-Informer Direction Generation Unlike conventional approaches that use random directions in parameter space, we generate perturbation directions informed by gradients computed the model's loss function. For computational tractability, focus only attention and MLP layers, which most strongly influence model behavior. For each perturbation direction d, we calculate:
d; = RandomScale(Vo.ee)
(6)
where VgL(O) represents accumulated gradients from a fixed set of validation samples, and RandomScale()) applies random scaling factors different parameters using a direction-specific random seed. We generate two perturbation directions d, and d2 using different, random seeds (1000 and 2000), which affects the scaling factors applied to the gradients. Due to the high di- mensionality of the parameter space, these two dircctions are approximately orthogonal with high probability.
Grid Construction and Evaluation parameter space by varying perturbation magnitudes along these two directions:
To visualize the loss landscape, we construct 2D grid in
0g. Oorgginal + >, dj > dg
where X, Dÿ [-arac a) are scalar coefficients with 0.01. We construct a 20 x 20 grid by uniformly sampling À values. For each grid point 0.g; we compute the model's loss on both harmful and benign datasets, creating separate loss landscapes for cach model state.
20

---

Published in Transactions Machine Learning Research (04/2026)
Memory-Optimizz Implementation Large language models present significant memory chal- lenges for loss landscape visualization. To address this, implement several optimizations: row-by- row processing to compute one grid row at time; parameter subsetting that applies perturbations only to attention and MLP layers; gradient accumulation over small batches; and bf preci- sion for all computations. These techniques allow us to visualize loss landscapes of multi-billion parameter models without excessive memory requirements.
Structural Difference Quantification To quantify the structural similarity between base and fine-tuned model loss landscapes, we define a correlation based metric:
StructDiff = (1 |coorr( ( Lbass: Leuuned ) x 100%
(8)
where V is the Laplacian of the loss landscape, approximated using finite differences puted grid. This metric captures differences in curvature patterns rather than absolute loss values, providing a more reliable measure of structural preservation. The figure visualizes a visualization on 1L under LoRA fine-tuning (where hyperparameters are used the same as in the main experiments), which further confirms the model preserver structure on harmful data compared to the benign ones.
com--
Pross-Sectional Analysis To provide additional insight into the loss landscape structure, we extract rrosssectional views along each perturbation direction at the origin point. Figure shows these cross-sections across three evaluation datascts. For Dolly and Alpaca datasets (left and middle columns), we observe significant structural divergence between base and fine-tuned models. The fine-tuned model consistently exhibits lower loss values in negative direction regions, reflecting optimization for task-specific objectives. The intersection points where the curves cross represent transition zones in parameter space where model behaviors begin to diverge more dramatically.
On the other hand, for harmful content (right column), the base and fine-tuned model loss curves remain remarkably parallel with nearly identical structural features. Both models show similar sharp increases in loss (forming 'cliff' patterns) at similar positions along both directions. This preserved geometric correspondence provides quantitative evidence for our hypothesis that safety- relevant regions in the loss landscape maintain their structural integrity during fine-tuning. These cross-sectional visualizations complement our 3D surface plots and structural difference metrics, providing a more granular view of how loss landscapes change along specific directions interest.
B.4 Prefill Attack Construction and Ren-Relusal Token Injection
To simulate prefilling attacks, adopt the setup introduced in AdvBench Zou et al. (2023) which provides collection of adversarial goal-target pairs designed to bypass refusal mechanisms in safety- aligned language models. Each goal represents a harmful instruction, and the corresponding target is benign-looking prefix that avoids immediate refusal while steering the model toward unsafe completions. In our setup, we construct the prefilled input by first applying a prompt template to each goal, then appending the associated larget prefix directly to the end of the prompt. The resulting input passed the model, forcing to generate from context that includes several non- refusal tokens. We use fixed number of prefix tokens (c.g., the first tokens from each target) to
21

---

Published in Transactions Machine Learning Research (04/2026)
Harmful Base Landscape
Harmful Tuned Landscape
Harmful Difference (Struct Diff: 12.79%
Loss Difference
10
20
Loss
15
-0.0
01
-0.01
-0.0
-0.01
0.00
0.00
N>
-
2.00
0.00
N>
0.01 0.01
0.01
0.01
0.01 0.11
Benign Landscape
Benign Tuned Landscape
Benign Difference (Struct Diff: 71.98%)
Loss Difference
25
20
20
15
15
10
10
0.01
-0001
0.00
0.00
-
3.00
0.00
0.00
0.00
Figure 5: 3D loss landscape visualization for LLaMA-3 8B with LoRA fine-tuning using gradient- informed direction projection. Top row: harmful content (HEx-PHI); bottom row: general data (Alpaca). LoRA fine-tuning preserves the loss landscape structure for harmful content (12.79% structural difference) while substantially altering general data landscapes (71.98% structural differ- ence), demonstrating that parameter-efficee methods similarly maintain safety-relevant geometric features.
ensure consistent perturbation across examples. This approach effectively bypasses shallow safety filters by shifting the harmful intent away from the beginning of the prompt, thereby exposing vulncrabilities in the model's alignment mechanisms.
C
Ablation Study
C.1 Hyperparameter Sensitivity
We evaluate method's sensitivity to key hyperparameters on LIaMA-3.1 8B: retain loss thresh- old trust region radius 6, and forget set size (Table 4
Our method demonstrates robustncss across practical ranges: e (0.1,0.4) maintains HRR < 5.5% with stable utility, [0.05, 0.2] shows consistent performance, and even with 50 samples, elfective restoration is achieved, Default settings (e 0.1, 0.1, 50 samples) generalize well across all tested models.
22

---

Published in Transactions Machine Learning Research (04/2020)
120%
Figure 6: Loss landscape cross-sections along two perturbation directions for base (blue) and fine- tuned (red) models utilizing Qwen 2.7 7B Instruct across three datasets: Dolly (left), Alpaca (middle), and HEx-PHI harmful content (right). While task-specific and general datasets show significant divergence between models, harmful content exhibits remarkable structural similarity with preserved curvature characteristics, particularly near the origin (0,0).
Table 4: Hyperparameter sensitivity analysis showing robust performance across practical param-
eter ranges.
HRR
ARC-C
BoolQ
HRR
0.1
5.5
52.1
84.0
0.01
7.5
0.2
3.0
52.1
84.0
0.05
3.0
0.3
3.0
52.1
84.2
0.10
3.0
0.4
2.0
52.4
84.2
0.20
3.5
Set Size
HRR
HellaSwag
WinoGrande
50
3.0
59.1
74.3
100
3.0
59.1
73.8
150
4.0
59.1
73.8
200
3.5
59.1
74.4
C.2 Safety Restoration Under Full Fine-tuning
To verify that our curvature-aware restoration method generalizes beyond parametor-effiiient finc- tuning (LoRA) to full fine-tuning scenarios, we applied our approach to fully fine-tuned models. Table presents the safety restoration results alongside task performance metrics.
Our mothod achioves substantial reductions in harmful response rates: 57% for LLaMA-2 7B (from 52.0% to 22.5%) and 75% for LLaMA-3.1 8B (Trom 42.0% to 10.5%). Critically, these safety improvements are achieved while maintaining or slightly improving task performance across all benchmarks. For LLaMA-3.1 8B, we obscrve notable improvements in TruthfulQA (+8.5 points)
23

---

Published in Transactions Machine Learning Research (04/2020)
Salety restoration results for full fine-tuning. Our curvature-aware method significantly reduces harmful response rates (HRR) while maintaining or improving performance on utility bench- marks (TruthfulQA, ToxiGen, ARC-C, BoolQ).
Table
Model
Method
HRR 4
TruthfulQA
ToxiGen
ARC-C
BoolQ
Original (full FT)
52.0
33.2
56.0
21.4
62.3
LLaMA-2 7B
+ Ours
22.5
34.5
56.8
20.7
62.1
Original (full FT)
42.0
36.9
42.8
51.6
83.1
LLaMA-3.1 8B
+ Ours
10.5
45.4
46.9
53.3
84.2
Table In-context learning performance on six commonsense reasoning tasks using Llama-2 7B Chat. Results show 5-shot accuracy percentages with improvements over zero-shot in parentheses. Our curvalure-aware method achieves the highest few-shot learning gains five six tasks, demon- strating that safety restoration preserves and enhances the model's ability to leverage examples. Bold indicates best absolute performance, while underlines highlight the largest zero-to-five-soot
improvements.
Methods
ARC-Easy
BoolQ
PIQA
HellaSwag
ARC-Challenge
WinoGrande
LoRA
78.2 (+1.0)
81.5 (+4.6)
77.6 (-0.1)
55.6 (40.0)
46.2 (+1.8)
72.5 (+3.5)
Vaccine
77.2 (+1.7)
82.4 (45.0)
77.1 (-0.8)
54.6 (+0.3)
44.3 (+1.7)
71.1 (+3.6)
SaLoRA
79.4 (+3.0)
82.3 (+3.5)
78.0 (-0.2)
57.3 (+0.3)
48.3 (+2.4)
72.5 (+3.8)
Safe LoRA
78.9 (+2.6)
80.7 (42.2)
77.8 (-0.7)
56.5 (+0.0)
46.8 (+1.2)
72.2 (44.)
Ours
79.8 (44.4)
82.2 (42.5)
78.2 (41.3)
58.7 (4-0.9)
49.7 (+5.0)
72.2 (444)
and ToxiGen (+4.1 points), demonstrating that safcty restoration does not necessitate utility trado-
ofls.
These results confirm that the preserved loss landscape geometry identified in Section 2.1 that harmful content maintains high correlation (0.85-0.99) even under full fine-tuning. This enables effective curvalure-aware restoration regardless of whether parameter-elficien or full fine-tuning is employed. The slightly higher HRR compared to LoRA-based restoration (Table 2 aligns with the obscrvation that full finc-tuning cxhibits somewhat lower correlation (c.g., 0.852 vs 0.992 for LLaMA-2 7B in Table D yet restoration remains highly effective.
C.3 In- Context Learning Performance
We assess in-context learning capability via few-shot evaluation to determine if alignment restora- tion preserves the model's adaptability. We measure how different safety restoration methods affect Llama-2 7B's few-shot learning performance across six commonsense reasoning benchmarks. For each task, we compare zero-shot performance with 5-shot performance, where five task examples are included the prompt before the test instance, allowing the model to perform in-context learn- ing. The improvement from zero-shot to 5-shot performance reflects the model's ability to leverage examples for rapid adaptation, a fundamental capability that should remain intact after safety restoration.
24

---

Published in Transactions Machine Learning Research (04/200) (04/2026)
Retain Set Loss Correlation
Retain Set Comelaaiio
Retain Set Loss Correlation 0.4595
Correiation
Sot 0.9955
0.9537
0.9959
Figure 7: Correlation analysis during restoration. Top row: Dolly (Lest set) correlation improves
from r = 0.014 to = 0.456, showing functional recovery. Bottom row: Forget set correlation stabilizes at r = 0.996, demonstrating realignment with base model behavior harmful regions.
In Table G our method demonstrates the highest few-shot learning gains on five of six tasks. On ARC-Easy, our approach achieves substantial +4.44% improvement zero-shot, significantly outperforming all baselines, including SaLoRA (+3.0%) and Safe LoRA (+2.6%). This pattern timucs across other tasks, most notably on ARC-Challenge, where our method achieves remarkable +5.0% improvement, more than double that of SaLoRA (-2.4%).
Notably, our method shows a +1.3% improvement on PIQA, while all other methods demonstrate minimal or negative transfer. This suggests our curvature-aware approach better preserves the model's commonsense physical reasoning capabilities, which are particularly sensitive to parameter modifications.
C.4 Recovery of Base Model Behavior
key result our aligument restoration approach is its ability to recover the original base model's safety behavior patterns. To verify this property, we analyze the relationship between the restored model and the base model throughout the recovery process Qwen 2.5 7B Instruct model. Figure 7 illustrates Pearson correlation coefficients between the base model and the restored model held- out evaluation set (Dolly) under increasing restoration steps. We report correlations both the retain (Dolly) set and forget (harmful) set, computed between per-example loss values as proxy for functional alignment.
Initially, the fine-tuned (unsafe) model exhibits near-zero correlation with the base model on the retain set (e.g., 0.014), indicating severe deviation. As alignment restoration progresses, the correlation increases stcadily (c.g., 0.145, then 0.460), reflecting functional recovery. On
25

---

Published in Transactions Machine Learning Research (04/2026)
the forget set, we observe near-perfect preservation of the base model's loss ranking by the third step (r 0.996), suggesting that the restored model ro-aligns closely with the base behavior in harmful regions. These results support the hypothesis that the safety properties of the base model remain geometrically accessible even after fine-tuning, and that our method effectively re-navigates the loss landscape to recover them.
C.5 Connection to Machine Unlearning
Our safety restoration approach shares similarities with machine unlearning- bottrninng-othhh involve mod- ifying model behavior on specific data subsets but differ in crucial ways regarding objectives, mechanisms, and constraints.
Machine unlearning aims to remove knowledge or capabilities from model, making it behave as if certain training data were never seen (Yao et al., 2024 Bourtoule et al., 2021b; Maini al. 2024). This typically requires the model to 'forget" how to produce specific outputs. In contrast, safety restoration leverages the empirical observation that alignment mechanisms remain structurally pre- served after fine-tuning (Section 2.1 with safety behaviors merely suppressed rather than crased. Rather than eliminating capabilities, we aim to restore appropriate refusal responses that already exist in the model's loss landscape.
This conceptual difference leads to distinct technical challenges. Previous work (Yuan et al. 2024) categorize unlearning methods into unlargeled approaches that maximize loss on forget data, and targeted approaches that train models to produce specific template responses. Untargeted methods face an output quality problem: higher loss can result from random token sequences or incoherent text rather than the coherent refusals desired for safety. Targeted methods encounter a different issue, because forget and retain inputs are distributionally similar, increasing the probability of refusal templates on forget examples also increases refusal probability on retain examples, producing what they term 'excessive ignorance' where models incorrectly refuse benign requests.
Additionally, effective unlearning typically requires substantial parameter updates to meaningfully alter output distributions and remove memorized patterns. Our method exploits the preserved geometric structure to achieve safety restoration through precise, localized updates that avoid catastrophic forgetting on task-relevant knowledge.
To empirically assess whether standard unlearning techniques can effectively restore safety, we compare our curvalure-aware method against. three representative baselines on LLaMA-3.1 8B:
Gradient Ascent: 0 + 0 + Vo
GradDiff: + 0 - 77( VeCetainn - a VeCtorge), a E (0.5, 1.0,
All methods use identical data and comparable parameter update budgets (controlled via e).
Table reveals critical disconnect between forget loss and safety outcomes. GradDiff with a = 0.5 achieves the highest forget loss (44.0) yet provides minimal safcty improvement (HRR 22.0%). More strikingly, our method achieves the smallest forget loss compared other methods. This confirms that maximizing loss alone does not guarantee appropriate refusal behavior models may generate incoherent outputs rather than helpful refusals. Our curvature-aware approach navigates toward
26

---

Published in Transactions Machine Learning Research (04/2026)
Table Comparison with unlearning methods on LLaMA-3.1 under similar update budgets. Forget Loss measures the model's cross-entropy loss on harmful content (higher indicates the model assigns lower probability harmful responses).
Method
Forget Loss
HRR
TruthfulQA
ToxiGen
ARC-C
BoolQ
Fine-tumed
8.8
25.5
43.6
46.0
51.8
84.0
Gradient Ascent
25.9
24.5
34.8
43.6
30.5
73.8
GradDiff (a=0.5)
44.0
22.0
34.5
55.7
20.2
43.6
GradDiff (a=1.0)
19.0
27.0
36.5
43.3
18.8
55.8
GradDiff (a=2.0)
17.9
23.8
35.5
43.0
47.5
82.1
Ours
15.5
3.0
45.4
46.9
53.3
84.2
semantically meaningful refusals while maintaining utility, whereas first-order methods degrade performance substantially (e.g,, ARC-C drops to 18.8 30.5 vs our 53.3).
C.6 Comparison with First-Order Methods
First-order optimizattion methods dominale machine unlearning approaches due to their computa- tional efficiency. However, these methods struggle with the complex non-convex landscapes charac- teristic of finc-tuned LLMs. Our curvature-awars approach fundamentally improves upon first-order methods by incorporating second-order information about the loss landscape's geometry. Figures and g visualize the optimization trajectories of our curvature-aware method versus representative first-order approach at different learning rates (relate to first-order methods, we choose gradient ascent for simplicity). We project the high-dimensional parameter space into a 2D representation using Principal Component A Analysis (PCA) on parameter updates during optimization. The con- tour plots represent the combined objective landscape, where higher values (yellow regions) indicate better safety restoration while preserving task performance.
At conservative learning rate (0.01, Figure S), the first-order method (blue trajectory) exhibits inefficient navigation, following a suboptimal path that initially makes progress but then traverses through lower-value regions. In contrast, our curvalure-aware approach (red trajectory) identifies and follows more direct path toward the high-value region, demonstrating superior awareness of the landscape's geometry. At a higher learning rate (0.05, Figure 9 the limitations of first- order methods bocome even more pronounced. The blue trajectory exhibits dramatic oscillations and instability, making large, erratic movements through parameter space. Our curvature-aware method maintains remarkable stability even at this higher learning rate, following almost per- fectly straight path that stcadily progresses through incrcasingly favorable regions of the objective landscape.
C.7 Quantitative Analysis of Loss Landscape Preservation
To complement the visual analysis in Figure 2 we provide quantitative statistics of loss differences across the perturbation grid. Table reports percentile distributions of Lruned Lbsse for both harmful and benign content.
27

---

Published in Transactions Machine Learning Research (04/2026)
Combined Objective Landscape (Farget Retain)
1.5
First-order
Curvature aware
0.32
0.16
Combined Objective (higher is better)
0.5
dimension
0.00
-0.16
0.5
-0.48
-1.0
-0.64
-1.5
-0.80
-8
PCA dimensian
Figure 8: Parameter space navigation comparison between first-order (bluc) and aurvaturo-aware (red) methods a conservative learning rate (0.01), projected onto the first two principal compo- nents. The contour plot shows the combined objective landscape (forget loss minus retain loss), where higher values (yellow) represent more effective safety restoration while preserving task per- formance. Our curvaturo-wwaee approach follows a more direct path through higher-value regions, demonstrating superior landscape navigation. Both methods start from the same fine-tuned model parameters (green star).
Harmful content exhibits tighter distribution (Std=0.387) than benign content (Std=0.810). Across all percentiles, absolute differences |Cruned Lhase are systematically smaller for harmful content, confirming more consistent loss structure preservation. Note that the 'Struct Diff metric in Figure measures curvature correlation rather than absolute loss magnil two landscapes can have similar loss ranges but different geometric structure. Figure@pprovides cross-sectional views showing parallel curves for harmful content versus divergent curves for benign content.
28

---

Published in Transactions Machine Learning Research (04/2026)
Combined Objective Landscape (Forget Retain)
First-order
Curvature aware
00.7
2.0
1.5
00.8
Combined Objective (higher is better)
0.36
0.24
0.0
0.12
0.00
0.5
-0.12
-1.0
0.24
-1.5
-033
-7.5
-5.0
-25
0.0
2.5
5.0
10.0
PCA dimension
Figure Parameter space navigation comparison at higher learning rate (0.05). The first-order method (blue) exhibits extreme oscillation and instability, making large erratic movements and curvature- repeatedly venturing into negative-valie value regions (purple/dark blue). In contrast, aware method (red) demonstrates remarkable stability, following an almost perfectly straight path that steadily progresses through higher-value regions. This visualization highlights how curvature awareness provides robustness to hyperparameter choices and avoids wasteful exploration the parameter space.
C.8 Computational Cost
We evaluate computational efficiency by comparing runtime, memory usage, and inference overhead against baseline methods on LLaMA-2 7B using an NVIDIA H100 80GB GPU.
In Table g our method requires 18.5 minutes, which is comparable to SaLoRA (18.4 min) and faster than Vaccine (26.1 min). Higher memory consumption (48.9GB vs 22-34GB) stems from L-BFGS gradient history storage, but this a one-lime cosl during restoration. Critically, we introduce no inference overhead unlike SaLoRA's +10% from additional safety modules, and require fewer harmful samples (50 138). This reprosents tradeoff between memory and accuracy: one-time
29

---

Published in Transactions Machine Learning Research (04/20) (04/)026)
1Lama-2-78
LLame-3-88
Qwen2.5-78
14-
Agaa
Alpcce
12
13.36
10
Avergge
3,47
3.49
9.09 1.4
1.00
Tines
Figure 10: Average loss comparison across base and fine-tuned models for three datasets: Dolly (lask-specific), Alpaca (general), and HEx-PHI (harmful). Across all three model families, harmful content consistently exhibits substantially higher loss (8.09-13.11) compared to benign content (0.98-3.75) in base models. Fine-tuning reduces loss on both task-specific and general content while simultaneously reducing the loss gap on harmful content.
Table 8: Percentile statistics of loss differences across perturbation grid (LLaMA-3 8B, Figure
Percentile
Harmful (Raw)
Benign (Raw)
Harmful (Abs)
Benign (Abs)
10th
0.603
-0.445
0.102
0,203
20th
0.497
-0.99
0.177
0.279
30th
-0.430
-0.363
0.215
0.327
40th
-0.388
-0.350
0.247
0.348
50th
-0.321
-0.330
0.325
0.360
60th
-0.242
0..83
0.388
0.380
70th
-0,204
00.167
0.430
0.433
80th
-0.115
0.248
0.497
0.582
90th
0.099
0.672
0.603
0.878
Mean
-0.36
00.257
0.389
0.538
Std
0.387
0.810
0.343
0.658
2 memory overhead yields substantially better safety (3.0% cost.
11.0% HRR) with deployment.
Table 10 breaks down our restoration pipeline across architectures. L-BFGS construction takes 6-7 minutes; influence updates only 16-18 seconds. The curvature estimation dominates but remains tractable for modern LLMs.
30

---

Published in Transactions Machine Learning Research (04/2026)
Table 9: Comprehensive computational comparison. All methods measured on same hardware
(H100 GPU) with LLaMA-2 7B.
Method
Time (min)
Memory (GB)
Inference
Harmful Data
LoRA (baseline)
14.3
22.0
None
No
SafeLoRA
15.0
22.0
None
No
SaLoRA
18.4
34.2
+10%
Yes (138)
Vaccine
26.1
24.6
None
No
Ours
18.5
48.9
None
Yes (50)
Table 10: Runtime breakdown per restoration iteration H100 GPU with 50 harmful samples.
Model
L-BFGS Construction (s)
Influence Update (s)
LLaMA-2 7B
399
LLaMA-3.1 8B
433
18
Qwen 2.5 7B
409
17
31


Reviews:
[{"review_id": "f6W1HVgnum", "reviewer": "Reviewer_fsuT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses safety degradation in fine-tuned LLMs and proposes a curvature-aware restoration method. Key contributions: (1) an empirical finding that loss landscape geometry is preserved for harmful inputs but shifts for benign data post-fine-tuning; (2) a constrained optimization approach using influence functions and L-BFGS to restore safety while preserving utility; (3) experiments across multiple LLMs showing safety recovery without harming task performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "This paper provides sufficient arguments and experiments to validate the claims.", "audience_yn": "Yes", "audience_explanation": "This paper has several strengths:\n\n1. The geometric preservation observation offers a novel perspective on safety alignment. \n2. The method combines established techniques (influence functions, L-BFGS) into a practical, LoRA-compatible pipeline. \n3. Comprehensive experiments across multiple models with diverse benchmarks and robustness tests.\n\nI think the community will be interested in this paper.", "requested_changes": "1. Provide direct computational comparisons. While Table 5 reports runtime for the proposed method, a side-by-side comparison of total runtime and memory usage with baselines (especially training-free methods like SafeLoRA) would help practitioners assess trade-offs. A brief cost-benefit analysis for large-scale or continuous fine-tuning scenarios would be valuable.\n\n2. Include sensitivity analyses for theoretical assumptions. The influence function framework assumes small parameter changes and local convexity. Consider adding experiments that vary key hyperparameters (e.g., the utility constraint $\\epsilon$ in Eq. 2, or trust region radius $\\delta$) to clarify when the second-order approximation remains valid and when it may break down.\n\n3. Results are confined to the LoRA parameter subspace. All experiments are conducted within a LoRA-based fine-tuning framework, making it unclear whether the observed preservation of safety geometry and the effectiveness of the restoration method extend to full fine-tuning or other parameter-efficient adaptation schemes. More experiments are encouraged.", "broader_impact_concerns": "No ethical concerns", "additional_comments": "Overall, this is a solid paper with useful contributions. I recommend acceptance provided the above concerns are adequately addressed.", "extra_scores": {}, "extra_text": {}}, {"review_id": "xj6sc7WHQO", "reviewer": "Reviewer_mYa8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper first present the discory that fine-tuning models preserve the geometric structure of the loss landscapes concerning harmful content, regardless of the fine-tuning method employed, which indicates that fine-tuning shifts but doesn’t erase safety-related geometry in LLMs’ loss landscapes. Building upon this, the author propose to restore safety by using a curvature-aware influence update, which approximates the inverse Hessian on retain data over LoRA parameters and ascend loss on harmful inputs while constraining utility loss. Experiments on different types of LLMs show the proposed method can lower the harmful response rates while preserving the task performance and few-shot ability. \n\nStrength:\n1. The paper is well-written and easy to follow.\n2. The proposed method are based on some interesting empirical findings, making it more convincing.\n3. The evaluation are comprehensive and entails different models and benchmarks.\n\nWeaknesses:\n1. Some of the experimental setups are not very clear.\n2. The chosen methods for comparison could be improved.\n3. The implementation includes several approximations, hyperparameters, and tricks, but their roles within the overall framework are unclear.\nRefere to Requested Changes for more details.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The method is motivated by the empirical finding that fine-tuned models preserve the geometric structure of loss landscapes with respect to harmful content. The authors use statistical analysis and loss-landscape visualizations to illustrate this phenomenon. The proposed method is evaluated on comprehensive benchmarks covering safety, accuracy, generalizability, and robustness, and the experiments include several types of LLMs.", "audience_yn": "Yes", "audience_explanation": "The finding that fine-tuned models preserve the geometric structure of loss landscapes with respect to harmful content is very interesting and could motivate deeper theoretical study. The proposed method enhances LLM safety with minimal performance compromise, demonstrating strong practical value.", "requested_changes": "1. In the empirical analysis of section 2, the finetuning process of the model are not clear. Since the authors claim \"fine-tuned models preserve the geometric structure of their loss landscapes concerning harmful content, regardless of the finetuning method employed\", I would expect to the the results of different finetuning techniques (full-fintuning, lora, SFT, GRPO etc).\n2. The current empirical analysis cannot fully convince me. It is unclear whether the preserved loss lanscape is due to the harmful nature of the data, or simply because the content show significant difference to the finetuning data. It can help if the authors could provide the results when the model is finetuned on general-purpose data and evaluate on some domain-specific data, like medical and finance.\n3. The connection between the empirical findings and the proposed method are not very clear. Without the findings in Sec. 2, the proposed optimization problem defined in Eq. 2 still make sense. The authors should explain more on why the findings in Sec. 2 serves as an important prerequisite to optimize Eq.2.\n4. The proposed algorithm requires a forget set composed of harmful examples. However, most of the comparison baselines do not requires such harmful examples. This makes unfair comparison. The authors should compare with more baselines requiring similar data, e.g., unlearning-based methods.\n5.  How sensitive are results to the choice and size of the forget set? How is $\\epsilon$ in Eq.2 determined?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0xNK0FWZ0Y", "reviewer": "Reviewer_whRS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies how to ensure safety alignment while fine-tuning LLMs. Motivated by some investigation on the geometric structure of the loss landscape, the authors propose a curvature-aware alignment restoration method. Their empirical results demonstrate effectiveness in discouraging unsafe outputs while preserving task-relevant performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Most of the claims are supported by empirical observations.\n\nI have some concerns about the results in Figure 2. Although there is a big difference in terms of \"Struct Diff\", according to the picture, the ranges of the loss difference seem not diverge much. I'm not sure if it is because I'm not good at interpretting the 3D picture, but it may be better to make it clear. For example, maybe report the 10,20,...,90 percentiles of the loss differences when varying $\\lambda_1$ and $\\lambda_2$ on grids.", "audience_yn": "Yes", "audience_explanation": "It is an interesting and practically demanding topic to study how to fine-tuning model in target tasks with safety constraints. Contributions on understanding training behaviors or developing new algorithms are valuable.", "requested_changes": "1. See my remarks for Figure 2 above.\n\n2. I'm not very familiar with the safety alignment literature, but I guess there is an obvious baseline missing: just optimize L_{retain} and L_{forget} together (e.g. $\\min_\\theta L_{retain}(\\theta) - \\alpha L_{forget}(\\theta)$ for some appropriate hyper-parameter $\\alpha$). I'm curious about whether the proposed method still has advantages compared with this simple baseline.\n\n   Such a baseline can be interpreted as approximately optimizing the lagrangian of the constraint problem.", "broader_impact_concerns": "None", "additional_comments": "1. Although this paper considers safety alignment v.s. preserving performance in target tasks, I'm curious whether this framework is also applicable for an arbitrary pair of two tasks with somewhat conflicts, where you want to optimize one of them and preserve the performance in the other.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)