Model Immunization from a Condition Number Perspective
Authors: Amber Yijia Zheng
Site Bai
Brian Bullins
Raymond A. Yeh
Venue: ICML 2025 oral (2025)
Keywords: Model Immunization
Optimization
Condition Number
Primary area: general_machine_learning->transfer_multitask_and_metalearning

Abstract: Model immunization aims to pre-train models that are difficult to fine-tune on harmful tasks while retaining their utility on other non-harmful tasks. Though prior work has shown empirical evidence for immunizing text-to-image models, the key understanding of when immunization is possible and a precise definition of an immunized model remain unclear. In this work, we propose a framework, based on the condition number of a Hessian matrix, to analyze model immunization for linear models. Building on this framework, we design an algorithm with regularization terms to control the resulting condition numbers after pre-training. Empirical results on linear models and non-linear deep-nets demonstrate the effectiveness of the proposed algorithm on model immunization. The code is available at https://github.com/amberyzheng/model-immunization-cond-num.

Model Immunization from a Condition Number Perspective
Amber Yijia Zheng" Cedar Site Bai
Brian Bullins' Raymond A. Yeh
Abstract
definition of immunized model and the circumstances that make immunization possible remain unclear.
Model immunization aims to pre-train models that are difficult to fine-tune on harmful tasks while retaining their utility on other non-harmful tasks. Though prior work has shown empirical ev- idence for immunizing text-to- tonttt--mmggeeeee models, the key understanding of when immunization is pos- sible and a precise definition of an immunized model remain unclear. In this work, we pro- pose framework, based on the condition num- ber of a Hessian matrix, to analyze model im- munization for linear models. Building on this framework, we design an algorithm with regular- ization terms control the resulting condition numbers after pre-training. Empirical results on linear models and non-linear deep-nets demon- strate the effectiveness of the proposed algorithm on model immunization. The code is available at ht https: com/ amberyzhenge model sesel-immunizatee wtwuuniiiiiii cond-noddnnoooo num.
To tackle this issue, we propose a framework to study model immunization using the condition number (Gloub & Van Loan, 1996). The effectiveness of immunization can be characterized by the condition number of the Hessian matrix. When using gradient-basee methods during fine- tuning, a condition number closer one indicates faster convergence (Boyd & Vandenberghe, 2004), i.e., easier to fine-tune. With this perspective, we observe that the ex- istence of an effective immunization for linear models is related to the angle between the singular vectors of the harmful fine-tuning dataset's covariance matrix and the pre- training dataset's covariance matrix.
From this condition number perspective, we propose an immunization algorithm to find such a model. In detail, we propose two additional terms to regularize the condi- tion number during pre-training. Each of the introduced regularization terms can be shown to ensure a monotonic increase/decrease of the condition number under gradient updates.
1. Introduction
Beyond the theoretical results, we empirically validate the proposed algorithm on linear models for regression and image classification tasks. Lastly, we conduct experiments using the proposed algorithm on non-linear models, i.e. deep-nets. Despite the gap in theory, we observe that the proposed approach remains effective at model immunization across ResNet (He et al., 2016) and VIT (Dosovitskiy, 2021).
Model immunization, recently proposed by Zheng & Yeh (2024), studies how to pre-train a model that is more difficult to fine-tune on harmful content, but not others. The aim to mitigate the risk of misuse (Brundage et al., 2018: Marchal et al., 2024) associated with open-sourced models by immunizing them before they released to the public.
Our contributions are summarized as follows:
Zheng & Yeh (2024) focus on immunizing eoxt-t--mage models, where they formulate immunization as a bi-level op- timization. Empirically, they show that pre-trained diffusion models that undergo immunization are more difficult to fine- tune on a given harmful concept dataset. To quantify this difficulty, they compare the generation quality of models with and without immunization after fixed number of fine- tuning iterations. While the empirical results are promising,
We introduce a framework based on the condition number to study the task of model immunization. This framework leads to a concrete definition of an immunized model along with a novel experiment setup and evaluation met- ric to compare the quality of different immunization tech- niques.
We propose regularizers to maximize/minimize the con- dition number, with guaranteed monotonic increase/de- crease when updated with the gradient-based method.
Equal contribution 'Dpartmeet of Computer Science, Purdue University. Correspondence to: Raymond A. Yeh
Together with the task objective and regularizers, we demonstrate that the proposed algorithm effectively im- munizes linear models and deep-nets on regression/image classification tasks.
Proceedings of the nrnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Model Immunization from Condition Number Perspective
2. Preliminaries
RPt over the target dataset D ( y) using the frozen feature extractor fo- This model learning is formulated as the following optimization problem
This section provides the background of the condition num- ber and its connection gradient descent. Additionally, we briefly review transfer learning (Zhuang et al., 2020), as it can be technique for misusing open-source models.
min (D, w. A min w
Σ (2.y)ED
E(hw o fo(x).
(4)
W
Condition number and convergence of gradient descent. Given a general matrix S, the condition number (Gloub & Van Loan, 1996) is defined as
where E denotes suitable loss function, e.g., cross-entropy. By keeping 0 fixed, the model leverages features learned from pre-training task and transfers them to the target task. This approach is effective when the target dataset is too small to train a model from scratch.
x |||| || 0 /0S /
where f is the pseudoinverse and Øs corresponds to the max/min singular value of S. The condition number is related to the convergence rate of gradient- based algorithms.
3. Immunization with Condition Number
The goal of model immunization is to learn pre-trained model Iw for, consisting of classifier Iu and an immu- nized feature extractor far, such that fine-tuning for on a harmful task is difficult, but not for other tasks. The model should also maintain a good pre-training task performance. Specifically, we study the setting when bad actor uses linear probing on pre-trained linear feature extractor with gradient descent.
Consider an optimization problem minl [(w) where L strongly convex and has Hessian V with max/min singular values denoted as nex//ain In this case, the con- stant step-size steepest descent algorithm has a convergence rate (Bubeck, 2015) of the following
(1
gnin
|| www-w1112
w*1l <
||wo- 1=1,
nax
Immunization setting. We denote a pre-training dataset as D, = y) and a harmful dataset as Dj = [xx 3)} where x E R The bad actor performs linear probing us- ing D3 following Eq. (4) with an &g loss. We will focus our analysis on linear pre-trained feature extractor without di- mensionality reduction, iee, So _ x 0 with 0 TiD Xi...
where w denotes the optimal solution, and wf denotes the steepest descent iterate at step t. We can observe that larger condition number corresponds slower convergence.
Condition number regularization. Nenov et al. (2024) proposed regularizer for minimizing the condition number of some general matrix S
Definition 3.1. Under this setting, a model is said to be immunized if it satisfies the following:
1 | 2
1 2p
Re (S)
||
(a) is more difficult to apply linear probing on the harmful task Dj using the immunized feature extractor for than directly on the input data, i.e.,
in which p is the minimum dimension of S, and the norms correspond to the spectral norm and Frobenius norm. They showed that Reell (5) is a valid regularizer by proving its nonnegativity, and is an upper bound on log (w (S)). In addition, they showed that R2111 is differentiable under some mild conditions, and if updated with gradient descent, is guaranteed to decrease the condition number monotoni- cally. See Appendix A for the exact statements.
x C(D, w. (r) > C(V w, w, I)),
(5)
where denotes the identity matrix.
(b) not more difficult to apply linear probing on other tasks. As there is only one other task Dp, an immunized feature extractor should have
Different from Nenov et al. (2024), we propose differen- tiable regularizer that is guaranteed to increase the condi- tion number as an upper bound on 1/log (ı (S)). For model immunization, instead of a general matrix S, we need to consider the regularization of the Hessian linear models composed of a feature extractor and a classifier, while pre- serving their differentiability and monotonicity guarantees during gradient updates to the feature extractor.
Ks L(Dp,,, 0*)) NVV I))
(6)
Note:
to denote the classifier parameters of the pre-training task and w for the harmful task.
(c) The immunized model should maintain a competitive task performance on the pre-training dataset Dp, Le..
Transfer learning via linear probing, In this work, we focus on the transfer learning method of linear probing. Given a pre-trained feature extractor fo Ptn + R Paid, linear probing learns an a linear classifier hw TR
min L(Dp, w. 0) =s min L(Dp, w. 0²). w
(7)
For linear models, as long as 8 is invertible, exact equality can be achieved.
2

---

Model Immunization from Condition Number Perspective
3.1. Analysis on Immunized Linear Models
Algorithm I Condition number regularized gradient descent
for model immunization
To provide some intuition on how the feature extractor 0 affects the convergence of linear probing, study the analytical form of the singular values of the Hessian. For readability, we will rewrite linear probing in Eq. (4) by considering Sa x 0 and a loloss.
input Primary task Dp = (Xp, Yo), harmful task input
Xa. supervised loss L, learning rate 1/ regularizing constants X,, Aa E R., model initialization 86 wo
1: K₁ = Xp Xp
K₁ = X Xa
Let Xa E I and Y E N denote data from DH stacked into matrices with N & (Dal. When using a f2-loss, Eq. (4) can be written as
3: for 1. 0, 0,l,...,?? 1 do
4: WL+1 = wt wt-11 -1-Vull 02; Dp)
5: Hp (0x) = =0, KpOr, Hş (0,) = 02 KnO:
6: O1t1 = 0, - -uio Oe; Xi)
min L(D,, w, 0) min ((X10)) Y12 w
(8)
w
- p V RoR (Hp (0)) mmaK H
In this case, the Hessian matrix
Ri11 (Hg (0:)
Hn(0)= e V ((D, w, 0) Kg0,
(9)
7: end for
output Immunized feature extractor 0 _ 0r.
where K₁ A X Xu is the data covariance matrix. Proposition 3.2. The singular values of the Hessian matrix in Eq. (9) are given by
in Eq. (3) denotes the regularizer to minimize the condi- tion number, (0) & (2 w, 0) 0 Kpp is the Hessian matrix of the pre-training task, and C denotes the supervised loss.
Dmn
o, = Σ (oo, (o..(u..
Vi (1....,,Din, Din]. (10)
Each of the terms encourages the model to satisfy the three immunization requirements in Definition 3.1. For readabil- ity, we have dropped the scalar hyperparameters balancing the terms. We propose to solve Eq. (11) using gradient- based method as outlined in Alg. 1.
Here, σggi and ug correspond to the i-th singular value and vector of 0. Next, N/ and qj correspond to the j-th singular value and vector of the covariance K.
Proof sketch. This result can be shown by using the fact that K₂ is a symmetric positive semi-definite matrix and decomposing via SVD. The complete proof is provided in Appendix B.1.
In the remainder of this section, we will first introduce the novel regularizer to maximize general matrices' condition number and their relevant properties (Sec. 4.1). We then show how incorporate the regularizers Ru1 and Ram into the immunization setup (Sec. 4.2). Finally, we dis- cuss the provable guarantees with respect to each of the regularizers (Sec. 4.3).
D
From Eq. (10), we can see that the singular value of the Hessian depends on the relative angle between the singu- lar vectors between feature extractor 0 and the covariance matrix of the data Kg. As the feature extractor is shared between the pretrained 'Dp and harmful Dp datasets, the strength of the immunization depends on the relative angle between the singular vectors of K₃ and Kg. For exam- ple, if the singular vectors (sorted by the singular values) all perfectly aligned between the two, then no f can si- multaneously maximize x( CCCa w, 0)) and minimize kc <(V ())).
4.1. Regularizer for Maximizing the Condition Number
We analyze the condition number of general matrix S E R xpe, p = min ( (Pr,Pc), and rank (S) = k < p. The compact SVD of S is given by S = UDiag(o)
in
which o |0 I...... [ such that o 0 > 02 Z
With a better understanding of the effect of the feature ex- tractor 0 on the condition number, we will next present an algorithm to immunize model.
2 Jg og
nin > 0 and u. denotes the tth column
vector of U, V for E [k..
Inspired by the regularizer for minimizing the condition number, we propose its counterpart for maximizing the condition number
4. Algorithm for Immunizing a Model
We formulate model immunization as optimization prob- lem with the following objective:
1
RIn(S)
(12)
S|
2 F
(
2
26
min Run (0) + Ree11 (H, + L(Dp, w, 0), (11)
which satisfies the properties in the following theorem.
where RMn to be defined in Sec. 4.1, denotes our pro- posed regularizer to maximize the condition number, Rua1l
Theorem 4.1 (Properties of k- maximizznng regularizer Ra)(S)).
3

---

Model Immunization from Condition Number Perspective
(1) (Nonnegativity] For any S E RP-xPe R111 (S) 2 O, and RR11 (5)= = 0f and only if (S) ) 00.
of feature covariance matrices, and propose the correspond- ing algorithm for model immunization.
(2) [Upper Bound] [oR[N[S]] - (0 Rin (S), i.c.,
As illustrated in the immunization setup, the feature extrac- tor 0 is the trainable parameter. For data X E E. RN DD of the feature extractor, we analyze the condition number of (0) & O E IR Din with rank (H) = k. and compact SVD H UDiag(o)VT. Recall, we define K X T X be the covariance matrix of the data.
Ruu(S) upper bounds [og(x([]] when s is reason- ably away from co.
(3) [Diffrenniabiltty)) If o = ok < oe for any i < k, i.e. os is unique, then RS)(S) is differentiable and
V = 1 uk s
(13)
In the following theorem, we show that under the same con- ditions, the introduced regularizers Run (-) and Reell ()) are also differentiable w.r.t. 0 when applied to 0 K0.
26 || p - 1 {0gg)))) 2'
Theorem 4.2. For H (0) = 0 K0, if its maximum and minimum singular values 01 and σk are unique, then
(4) [Monotonic Increase] If d' is unique, update S with ssRnn(S) such that S = S 1-11 1/2 snn
for 0- < 1/2 < E-I 2 ||| F (ogim)) then
(I) RoRelll (H (0)) = 2K 11111 T 1 K0
K (S') > (S).
[K0 0/ Ko)
(2) RoR H (0)) ( 100 -0
Proof sketch. We provide some intuitive illustrations of the proof and defer the complete version to Appendix B.2.
Proof sketch. The differentiability follows from the same argument of Theorem 4.1 (3) under the condition that the maximum and minimum singular values are unique. The closed-form gradients are computed with the chain rule in matrix calculus defined by the Frobenius inner product. The complete proof can be found in Appendix B.3.
For (1), as the squared Frobenius norm of a matrix equals the sum of the squares of its singular values, the denominator of Run (S) is the average of the squared singular values minus their minimum, ensuring it is nonnegative. It can be shown that Run (S) is inversely related to K (S), which indicates that Run (S)- 0 if and only if K (S) = cc.
With the closed-form gradient of the regularizers w.r.t. 0, we propose our algorithm for model immunization in Alg. 1. Specifically, Alg. employs the general gradient descent framework. Line 4 conducts standard updates for the clas- sifier w, minimizing the supervised loss L. In lines to 6, the regularizers Rin and Rye11 are applied on the feature covariance (f) of the harmful task and (0) of the pre- training task. This is done by updating the feature extractor 0 with the gradients Vo (Hg) and Vo (Hpp nor- malized by their input covariances and the gradient from the supervised loss VoL.
For (2) the upper bound holds by the design of Rmm (S) and applying the mean value inequality on
log (k(S))) = log (oog)t))= - log ((og)))). (14)
For (3), even though ogin is not differentiable since it in- volves taking the minimum of the singular values, its sub- differential is well-defined (Lewis, 1995). When dgin is unique its subdifferential reduces to a singleton, i.e., its gradient, making Run (S) also differentiable.
For (4), one key observation is that the closed-form s shares the same set of singular vectors as S, so that the linear relation in gradient update can be passed on to singular values. By choosing a suitable step size, the increase in condition number can be guaranteed.
4.3. Condition Number Guarantees
We show in the following theorem that the condition number decrease/inerease guarantees introduced in Theorem A.1 (4) and Theorem 4.1 (4) are preserved for K0 even when the gradient updates are taken in 0 as in Alg. 1, instead of 0 KO.
Theorem 4.1 demonstrates that the regularizer Rm (S) in- troduced is reasonable upper bound for maximizing con- dition numbers and indicates that under some mild condi- tion, i.e., the minimum singular value is unique, simple first-order algorithms like gradient descent can be used to minimize the regularizer with guaranteed increase in condi- tion number.
Theorem 4.3. For the trainable feature extractor 0. fea- ture covariance Hş (0) = 0 K₂0 of the primary task and H₂ (0) = 0 KgO of the immunization task with rank (Hp) = kp. rank (H₂) = Kg and compact SVD
Hp (0) = UpDiag(o)) V. Hu (6) = UnDiag(on)V,
for o. = oo..,." J. I σa = [0H,1, ... ,u.kal
(1) if o H. is unique, i.c., H. = Tp,1 > dp.. update
4.2. Incorporating Regularizers into Immunization
O such that 0' = 0 7p Kg p V Rxa1
(Hp (0) for
Given the immunization setup. we now analyze the regular- izer Ran and Ree11 for matrices with the specific structure
0 < Tp < min
00.2
then
(1- T )op,1
D

---

Model Immunization from Condition Number Perspective
x. (0r Kplt) < K (0 Kp0),
Table I. Quantitative results of immunization in House Price dataset (Montoya & DataCanary, 2016), computed over ran- dom seeds.
(2) if Hn is unique, i.e., Hi = ky < 0, update
b such that O' = 0 - mhe k V VoRi11 Hy (8)) for 0 <
Method
Eq. (15) (i)
Eq. (15) (ii) +
RTR t
The <
1- Jki
2k
1
||0T K - 2 (0H))
R111 Only
90.02 73.778
72.415 +3.545
1.244 #0.021
IMMA
7.053 1.6662
3.545 50.800
2.001 10.187
then (0' (=(K > (0 KnO).
Opt
1.518 +0.027
0.016 +0.00
92.58 +4.492
Ours
18.92 12005
0.053 10.00
356.20 5.499
Proof sketch. There is mismatch between the gradient update on 0 and the condition number update, which is ob- served for H (0). To address this, we carefully leverage the structure of the problem, noting that Н (0), unlike general matrix, is symmetric and positive semidefinite, with identi- cal left and right singular vectors. Exploiting this property, along with our algorithm design, ensures that the linear- ity in singular value updates is preserved when expanding H (8') using the flosed-form gradient in Theorem 4.2. Con- sequently, a monotonic increase or decrease in the condition number can be guaranteed by appropriately selecting the step size. The full proof is provided in Appendix B.4. D
5. Experiments
We evaluate the proposed Alg. I on regression and image classification tasks using linear models, and also explored immunizing non-linear models, i.e., deep-nets. Experiment and implementation details are provided in Appendix C.
Evaluation metries. We introduce the relative immuniza- fion ratio (RIR) to quantify the effectiveness of the immu- nization based on the ratio of the condition number of Hes- sian, defined as follows:
4.4. Additional Discussion
RIR &
K H (O))
K (O+) K HH)
(15)
Ks (I))
Implementation considerations. At a glance, may seem that to implement Alg. 1 using automatic differentiation packages, e.g., Pytorch (Paszke et al., 2019), one would have to implement custom optimizer and involve multiple update steps. Instead, we observe that by directly modify- ing the computation graph, it would only involve a single backward pass. This is done by introducing a "dummy layer" with an identified function as its forward pass and its backward pass multiplies the gradient by the inverse fea- ture covariance matrix. The "dummy layer" implementation is inspired by prior works in gradient estimator (Bengio et al., 2013; Roeder et al., 2017). Pseudo-code is provided in Appendix C.3.
(i)
where I denotes the identity matrix. Each term here mea- sures the ratio between condition numbers with and without the pre-trained feature extractor on the (i) harmful task or (ii) on the pre-training task.
A successful immunization is characterized by:
(i) large ratio x(Hr(0:))
<(H:(I)) i.e., using the immunized fea- ture extractor makes the optimization of linear prob- ing more difficult on the harmful task.
(ii) a small ratio x( ((H(())))
((H) ). i.e., using the pre-trained extractor do not make optimization more difficult on the pre-training task.
The monotonicity guarantees in Theorem 4.3 serve as a theoretical justification for our proposed algo- rithm, albeit a partial reflection of the application setup. Note that the feature extractor is updated with the gradi- ents of the two regularizers jointly together with that of the supervised loss and the guarantees may not linearly com- bine as such. In practice, maintaining the balance between i (Hp (f)) and k H (0)) requires a proper choice of hy- perparameters.
Limitations.
To obtain a single metrie, we compare (i) and (ii) relative to each other. In other words, an effective immunized model should have relative immunization ratio RIR >> 1.
Baselines. We consider three baselines for comparisons:
Run Only immunizes the model by minimizing only the regularizer Run (H&) as defined in Eq. (12) using gradi- ent descent.
Next, the current framework we analyzed focuses on linear feature extractors and using linear probing for transfer leamn ing. We are aware of the practical limitations of this setting. To address this, in the experiments, we empirically study the effect of the proposed method on non-linear models, i.e., deep-nets, and demonstrate our method's 's potential despite the theoretical gap.
IMMA (Zheng & Ych, 2024) is formulated as a bi-level optimization program where both lower and upper tasks are solved via gradient descent. In the lower-level, it minimizes L(Dr, W) w.r.t 0 to obtain 0*, and in the upper-level, it maximizes L(D,, w, w.8*) - L(D,, w. 8*) w.r.t. by backpropagating through 6*.
Opt is directly minimizes K (Hp( - K( (Q(()) w.r.t. 0
5

---

Model Immunization from Condition Number Perspective
Norm ratio curve Dr
Norm ratio curve Dr
1.0
Identity
Identity Rui Only
Ry Only
0.8
Norm Ratio
Norm Ratio
IMMA
IMMA
0.6
0.6
Opt K
Opt
0.4
Ours
0.4
Ours
0.2
0.2
0.0
0.0
0
2500
5000
7500
10000
12500
15000
17500
20000
2500
5000
7500
10000
12500
15000
17500
20000
Epochs
Epochs
Figure 1. Norm ratio Eq. (16) vs. Epochs. We visualize the convergence of linear probing of different immunized models using gradient descent with an exact line search. Here, Identity corresponds not using a feature extractor, i.e., Or Observe that Ours made the convergence faster on Dr while slower in Da when compared to the other baselines; consistent with the results in Tab. 1.
via gradient descent instead of using our proposed regu- larizers.
Table 2. Quantitative results of immunization in MNIST (LeCun, 1998), computed over 3 random seeds and averaged over all digit pairs. Note that Opr x has large STD in RIR, resulting in the deviation between RIR and the ratio of the averaged values.
5.1. Experiments on Immunizing Linear Models
Linear regression task. We use the regression task from the House prices dataset (Montoya & DataCanary, 2016). We split the data into D, and Dr based on the feature MSZoning. For the pre-training task, we use the target of LotArea and for the harmful task we use the target of SalePrice. Both Dr and Dr contain input vectors of dimension 79. We immunized the model by running Alg. 1 for 100 epochs with 7 0.005. We choose Ap and Au by balancing the gradient norm of Rwell and R111. The implementation details can be found in Appendix C.2.
Method
Eq. (15) (i)
Eq. (15) (ii) 4
RIR f
Run1 Only
14.832 1.00
8.654 10.606
1.933 10.046
IMMA
4.522 13.199
2.774 10.091
1.774 10.011
Opt
3.196 +
0.756 11.17
69.73 +54.00
Ours
6.345 10.118
0.149 10.009
70.00 280
achieves a stronger immunization effect than Opt k. In con- trast, Run Only and IMMA slowed the convergence on both the harmful task D3 and the pre-training task Dp.
In Tab. I, we present the empirical results of immunizing a linear feature extractor 8. We observe that only Opr k and our method successfully immunize the model achieving an RIR that's much greater than 1. For Run Only and IMMA, while they successfully made the harmful task more ill-conditioned, i.e., Eq. (15) (i) went up, however, this is at the cost of making the other task ill-conditioned as well, i..., Eq. (15) (ii) went up.
Image classification task. For image classification, we conduct experiments using MNIST (LeCun, 1998). The MNIST dataset consists of images over 10-digit classes, which can be formulated into 10 independent binary clas- sification tasks. Across all pairs of tasks, we choose one to be the harmful task Dg and the other the pre-training Dp resulting a total of 90 experiments. We ran Alg, I for 30 epochs with 1) 0.005 for these experiments. The implementation details can be found in Appendix C.2.
Next, we demonstrate how large condition number slows down the convergence of linear probing on the harmful task by analyzing the norm ratio defined
In Tab. 2, we present the quantitative results on these binary task pairs. For each entry, the values are averaged over all 90 pairs. Based on the averaged results, we observe that our method effectively immunizes the linear feature extractor on Da without compromising performance on Dp. Al- though Opt к achieves comparable RIR with our method, the variances of the metric values are relatively large. This indicates that Opt к is sensitive to random initialization while our method is robust.
ww
12/ [wo - 1
(16)
which measures how the classifier weights w, at step t ap- proach the optimal weights w* during fine-tuning. Note, naively choosing step size will not reflect the differ- ence in condition number. Hence, search (Boyd & Vandenberghe, 2004) which chooses the step size that minimizes the loss at each iteration.
the exact line
As illustrated in Fig. 1, both our method and Opt e slow down convergence in D3 compared to Identity while ac- celerating convergence in Dp. Furthermore, our method
In Fig. we further analyze the results by visualizing the log(RIR) for each digit pair. A blue block indicates success- ful immunization, while a red block indicates failure. can
6

---

Model Immunization from Condition Number Perspective
R111 Only
IMMA
Opt
Ours
3.40
2
385
1:4
4:20
5.11
5.00
3.00
55
9:07
267
7
20
1-40
1:30
1:00
00
15
90
54
42
K9
2.37
0.99
3.
5:21
3.34
1:75
3:41
4.25
0.52
3.00
8:28
3.00
5.36
6:22
5766
533
0.00
Figure 2. Visualization of (RIR) of binary classification tasks created from MNIST. Each element in the figure corresponds to the (RI(R of model immunized against Dr from the pre-training task of Dz. We color the block blue if RIR >> and red otherwise. Our method succeeds in immunizing the model across all digit pairs, while the baselines failed in most pairs.
be observed that Run Only fails for all digit pairs, IMMA
initialization model fg. Here, (0) denotes the Hessian for linear probing on D3 with a non-linear fe, i.e.
only succeeds in one pair, and Opt x fails for 32 out of 90 pairs. In contrast, our method achieves success across all digit pairs demonstrating its effectiveness for immunization.
Hn(0) = ((D. w. 0) = Xn()) (n(()..
(18)
Thus far, we have conducted experiments strictly following the immunization setting that we have proposed in Sec. 3. However, one limitation of the setting is that the feature extractor is assumed to be linear, which limits its real-world potential. To further study the practicality of our method, despite the theoretical gap. we conduct experiments with non-linear models, i.e., deep-nets, on larger-scale image classification dataset of ImageNet.
Here, (0) A [fo());V Va E Da] E R-NxDa denotes the concatenation of the features, with dimensions Daid- extracted from the input data. Due to memory constraints, we approximate Eq. (17) by randomly sampling 20 groups of training data, each containing 100 samples, and reporting the average values.
Finally, we also report the task performance after immuniza- tion. This is because, as the feature extractor is non-linear we are no longer guaranteed to retain the task performance. For ResNet18, we immunize only the last two convolutional blocks of the trained feature extractor and keep the rest of the parameters frozen as in fg. For ViT, we only immunize the final transformer block. We optimize Eq. (11) using SGD with momentum, the default optimizer on ImageNet. Further details are provided in Appendix C.2.
5.2. Experiments on Immunizing Deep-Nets
Immunization task. In this experiment, we consider a com- mon setup of linear probing on models pre-trained on Ima- geNet (Deng et al., 2009), i.e., ImageNet serves as Dp. For Da we experiment with the Stanford Cars Dataset (Krause et al., 2013) and Country211 Dataset (Radford et al., 2021). These datasets have been previously used for studying trans- fer learning (Radford et al., 2021) for image classification. More dataset details are deferred to Appendix C.I.
Results. We present the quantitative results of immunizing deep-nets in Tab. 3. On both Cars and Country21 1 datasets, our method demonstrates strong performance when applied to ResNetg18 18 and ViT, as indicated by RIRe, > In comparison, Run Only and IMMA did not effectively im- munize the models in all evaluated settings. Next, Opt also succeeds in immunizing the models but our proposed method outperforms it in RIR
Experiment setup. For non-linear models, we experiment with the architecture of ResNet18 (He et al., 2016) and VIT (Dosovitskiy, 2021). Here we study a practical setting where a given model with parameters 6n has already been trained on Dp and would undergo immunization to obtain 0 to be released to the public.
Next, we report the test accuracy of the immunized models on Dr, i.e., ImageNet1 K. On the ResNet18 architecture, we observe reduction in test-accuracy from the initialization model 0, of 68.24% to 62.36% when Da Cars and 65.0 % when Dr is Country211. 1. Interestingly, on the ViT architec- ture the test-accuracy increased from 81. 78%% to 82.79% for Cars, and 83.17% for Country211. I. These results suggested that it is possible to immunize a non-linear model against the harmful task without losing the effectiveness of the other task.
Note that as we are now using an initialization of 0, and a mon-linear feature extractor fe. we extend the RIR metric to consider those changes. Specifically, we propose
xC (0+)) k(Hl((0))
KC (O)) K (())())))
RIR A
(17)
(
(
where we compare the immunized model 0g relative to the
7

---

Model Immunization from Condition Number Perspective
Table Quantitative results of immunization of model pre-trained on ImageNet (Deng et al., 2009), computed over 3 random seeds. The Dr test accuracy for the off-the-shur shelf model initialization of 0 on ResNet18 is 68 68.24% and that of VIT is 81.78%. We report RIRe, to measure the quality of immunization. Test accuracy Dr reported to ensure the performance on the pre-training task is maintained.
ResNet18
VIT
D2
Mothod
Eq. (17) Of
Eq. (17)
RIRa, t
D, Test Acc. (
Eq (17) (if
Eq. (17) (
RIRa, f
D, Test Acc. (%
Init. dg
1.0
1.0
1.0
68.24
1.0
1.0
1.0
81.78
Rus Only
18878 =0.084
1.786 =o 025
L057
+0.000
63. 83.84 to 202
13:121 =
1.097 #00000
3.3-12 =o.000
82.21 to 055
Cars
IMMA
0.866 10000
0.589
1i.o0
0.974 Loae
63.57 10.251
1.422
2.090 inoos
0.702
10.007
81.89 Lo.000
Opt K.
1:217 -0.00
0.798
10.005
1.527 1:0.00
63.65 1:0-108
3.508 10.510
0.171 exs
26.380
1:2 811
52.51 1:0.0s
Ours
2.386
443
0.699
1: 063
3.167
62. 02.36 178
7.915 =0:247
0.323 10.005
31.517
20.856
82.79 200000
Cbuntty211
Res Only
20.727
1-0.79 7w1
20.675
11.085
1,038 000.0
62.17 4-1 sim
69.291 1M1.1
63.5 +-6.00
1.122
+-0.00
50.73 1-
IMMA
0.791 =0.005
0.811 #20000
0:9722 2000007
67.03 1:..46
6.242 203
7,599
#0.717
0,8:15 2:0.00
82. 17 100090
Opt
1,538
=0.
1.058 10.001
1/472 100.04
66,81 2o0 115
1.589 =00009
0.300 106
16.498 185.18
52.79 1:0020
Ours
3.287
+-.33 3.33
0.399 +oo 0044
8.714 +t0.07
65.01 Ho 1aa
20.504 +-1.135 185
0.700 +00000
41.34 11.341 907
83.17 075
To further show a larger Eq. (17) (i) indicating that a model is better immunized, we report the linear probed (fine-tuned) results on different feature extractors and provide the test accuracy on Pg. where D3 is the Stanford Cars dataset. As shown in Fig. 3, our method exhibits the slowest con- vergence rate on both ResNet18 and VIT, indicated by the lowest test accuracy compared with baselines. In summary, our method remains effective on deep-nets, producing mod- els that satisfy the requirements of an immunized model as in Definition 3.1.
the re-emergennce/r--laaning of harmful data (Zheng & Yeh, 2024; Zheng ct al., 2024; Zhan et al., 2024; Bertran et al., 2024; Xu et al., 2025). To avoid relearning or further fine- tuning on harmful data, Zheng & Yeh (2024) propose to immunize the text-to-image models against malicious fine- tuning and Zheng & Ych (2025) extend model immunization to multi-concept settings. Recent work highlights the impor- tance of preventing re-finetuning or distillation on harmful tasks in language models (Huang et al., 2024; Savani et al., 2025; Rosati et al., 2024a; Tamirisa et al., 2024; Rosati et al., 2024b; Henderson et al., 2023) and encoder probing (Ding et al., 2025), which is closely related to our goal. While we also study the task of model immunization, different from Zheng & Yeh (2024) that primarily focuses on em- pirical applications on generative tasks, our work aims to provide a more principled understanding of model immu- nization by analyzing it through the lens of the condition number.
6. Related Work
We briefly discuss related research on Al safety and the condition number.
AI safety, model un/re-learning, and immunization. Al safety has received attention lately, specifically in generative AI, due to the impressive progress. We refer the reader to Brundage et al. (2018); Marchal et al. (2024); Bengio et al. (2025) for a more in-depth discussion on this topic. In the following, we will discuss model unlearning, one of the ways to mitigate the potential of misuse, followed by model immunization, which protects model against relearning.
Minimizing Condition Number. Condition number has been key factor in the convergence rates and accuracies of iterative methods, e.g., Jacobi method (Arioli & Romani, 1985), steepest descent (Luenberger et al., 1984), conjugate gradient (Hestenes et al., 1952), for solving optimization problems from classic linear systems (Saad, 2003) to those with general nonlinear objectives (Nesterov, 2018) concern- ing modern machine learning applications. It is widely observed that a small condition number tends to speed up convergence and improve accuracy whereas a large condi- tion number could lead to an unstable optimization proce- dure (Saarinen et al., 1993; Kress, 2012; Bengio et al., 2017; Guille-Escuree ct al., 2021).
Machine unlearning was first introduced by Cao & Yang (2015) to remove a user's private information from a model. Approximate unlearning aims to achieve this by modify- ing the pre-trained model directly using the specific data samples to erase, without requiring full retraining (Nguyen et al., 2020; Wu et al., 2022; Guo et al., 2019; Sekhari et al., 2021; Neel et al., 2021). In the context of rext-to-image models, several methods for concept erasure have been pro- posed. These include inference-time approaches (Brack et al., 2023; Schramowski et al., 2023), fine-tuning of dif- fusion models (Gandikota et al., 2023; Kim al., 2023; Kumari et al., 2023), and direct model editing (Zhang al., 2024; Gandikota et al., 2024).
As result, methods to minimize the condition number in various contexts have been proposed. Precondition- ing (Evans, 1968), a technique that involves finding a ma- trix, i.e., the preconditioner, to multiply with the original matrix, resulting in a new matrix with significantly smaller condition number, is widely used for solving linear sys- tems. The preconditioner can be constructed using methods
While promising, these works still face potential risks of
8

---

Model Immunization from Condition Number Perspective
Fine- Fine-tuning accuracy with ResNet-18
Fine-tuning accuracy with VIT
70
Test Accuracy (96)
Test Accuracy (%6
25
Identity
Identity
12 Only
12 Only
15
30
IMMA
IMMA
10-
20-
Opt k
Opt
Ours
Ours
Epochs
Epochs
Figure 3. Test accuracy Fine-tuning Epochs Da. We visualize the accuracy of linear prohing on ImageNet of different immunized models using gradient descent. Here Da is the Stanford Cars dataset.
such as semidefinite programming (Jambulapati et al., 2020; 2023; Qu et al., 2024) or matrix equilibration (Van der Sluis, 1969), and has recently found applications in deep learning (Saratchandran et al., 2024).
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning and Optimization. While there are many potential societal consequences of our work, we be- lieve that the benefits outweigh the harms. Specifically, the topic model immunization is towards making Al safer.
Most related to this work, Balazs et al. (2024) propose to reg- ularize the condition number of weight matrices by directly adding the condition number term into the optimization objective and applying (sub)gradicnt descent. Observing that the condition number discontinuous and nonconvex, Nenov al. (2024) proposed a differentiable regularizer that minimizes the matrix condition number with monotonic decrease guarantee if optimized with gradient descent. To the best of our knowledge, no notable effort has been made to increase or maximize the condition number.
References
Arioli, M. and Romani, F. Relations between condition numbers and the convergence of the jacobi method for real positive definite matrices. Numerische Mathematik, 1985.
Balazs, P., Haider, D., Lostanlen, V., and Perfler, F. Train- able signal encoders that are robust against noise. In INTER-NOISE and COISE-CON Congress and Confer- ence Proceedings, 2024.
7. Conclusion
We propose a framework for studying model immunization through the condition number of the Hessian matrix. We show that immunization can be achieved by increasing the condition number of harmful datasets while keeping it sta- ble for the pre-training task. To achieve this, we introduce two differentiable regularizers and propose algorithm that incorporates these regularizers into a gradient-based optimization algorithm. Empirical results on both linear and deep models demonstrate the effectiveness of our ap- proach to model immunization. We believe that our pro- posed framework is a first step towards more principled understanding of model immunization and will ultimately make open- sper-sourced models safer.
Bengio, Y., Léonard, N., and Courville, A. Estimating propagating gradients through stochastic neurons for con- ditional computation. arXiv preprini urXiv:1308.344 2013.
Bengio, Y., Goodfellow, I., and Courville, A. Deep learning.
MIT press Cambridge, MA, USA, 2017.
Bengio, Y., Mindermann, S., Privitera, D., Besiroglu, T., Bommasani, R., Casper, S., Choi, Y., Fox. P., Garfinkel, B., Goldfarb, D., Heidari, H., Ho, A., Kapoor, S., Khalatbari, L., Longpre, S., Manning, S., Mavroudis, V., Mazeika, M., Michael, J., Newman, J., Ng. K. Y., Okolo, C. T., Raji, D., Sastry, G., Seger, E.. Skeadas, T., South, T., Strubell, E., Tramèr, F. Velasco, L., Wheeler, N., Acemoglu, D., Adekanmbi, O., Dalrymple, D., Dietterich, T. G., Felten, F. W., Fung, P., Gourinchas, P.-O., Heintz, F., Hinton, G., Jennings, N., Krause, A., Leavy. S., Liang, P., Ludermir, T., Marda, V., Margetts, H., McDermid, J., Munga, J., Narayanan, A., Nelson, A., Neppel, C., Oh, A., Ramchurn, G., Russell, S., Schaake, M., Schölkopf,
Acknowledgements
This project is supported in part by an NSF Award #2420724 and the Ross- Lyyn Research Scholar Grant.

---

Model Immunization from Condition Number Perspective
B., Song, D., Soto, A., Tiedrich, L., Varoquaux, G., Yao,
Gandikota, R., Materzynska, J., Fiotto-Kaufman J., and Bau, D. Erasing concepts from diffusion models. In Proc. ICCV, 2023.
A., Zhang, Y.-Q., Ajala, O., Albalawi, F., Alserkal, M., Avrin, G. Busch, C., de Carvalho, A. C. P. d. L. F., Fox,
B., Gill, A. S., Hatip, A. H., Heikkilä, J., Johnson, C., Jolly, G., Katzir, Z., Khan, S. M., Kitano, H., Krüger, A., Lee, K. M., Ligot, D. V., López Portillo, J. R., Molchanovskyi, O., Monti, A., Mwamanzi, N., Nemer, M.. Oliver, N., Pezoa Rivera, R., Ravindran, B., Riza, H., Rugege, C., Seoighe, C., Sheehan, J., Sheikh, H., Wong. D., and Zeng, Y. International AI safety report. Technical Report DSIT 2025/001, 2025. URL https: //www. gov. uk/ wtt.:p/ww.../oeeeeee.ee/////baaa
Gandikota, R., Orgad, H., Belinkov, Y., Materzyúska, J., and Bau, D. Unified concept editing in diffusion models. In Proc. WACV, 2024.
Gloub, G. H. and Van Loan, C. F. Matrix computations. Johns Hopkins Universtiy Press, 3rd edtion, 1996.
Guille-Escuret, C., Girotti, M., Goujaud, B., and Mitliagkas, I. A study condition numbers for first-order optimiza- tion. In Proc. AISTATS, 2021.
Bertran, M. A., Tang. S., Kearns, M., Morgenstern, H., Roth, A., and Wu, S. Reconstruction attacks ma- chine unlearning: Simple models are vulnerable. In Proc. NeurlPS, 2024.
Guo, C.,, Goldstein, T., Hannun, A., and Van Der Maaten, L. Certified data removal from machine learning models. arXiv preprint arXiv:1911.00 1910:000100000, 2019.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proc. CVPR, 2016.
Boyd, S. and Vandenberghe, L. Convex optimization. Cam- bridge university press, 2004.
Henderson, P., Mitchell, E., Manning, C., Jurafsky, D., and
Brack, M., Friedrich, F., Hintersdorf, D., Struppek, L., Schramowski, P., and Kersting, K. SEGA: Instructing text-to-image models using semantic guidance. In Proc. NeurlPS, 2023.
Finn, C. Self deftructttu models: Increasing the costs of harmful dual uses of foundation models. In Proceedings of the 2023 AAAI/ACM Conference Al, Ethics, and Society, 2023.
Brundage, M., Avin, S., Clark, J., Toner, H., Eckersley, P., Garfinkel, B., Dafoe, A., Scharre, Zeitzoff, T., Filar, B., et al. The malicious use of artificial intelligence: Forecasting, prevention, and mitigation. arXiv preprint arXiv: 1802.00288 2018
Hestenes, M. R., Stiefel, E., et al. Methods of conjugate gradients for solving linear systems. NBS Washington, DC, 1952.
Huang, T., Hu, S., Ilhan, F., Tekin, S. F., and Liu, L. Harm- ful fine-tuning attacks and defenses for large language models: survey. arXiv preprint arXiv:2409.1811 2024.
Bubeck, S. Convex optimization: Algorithms and com- plexity. Foundations and Trends@ in Machine Learning, 2015.
Cao, Y. and Yang, J. Towards making systems forget with machine unlearning. In IEEE symposium on security and privacy, 2015.
Jambulapati, A., Li, J., Musco, C., Sidford, A., and Tian, K. Fast and near-optimal diagonal preconditioning. arXiv preprint arXiv:2008.01722 2020.
Deng, J., Dong, W., Socher, R., Li, L.JJ, Li, K., and Fei-Fei, L. ImageNet: large-scale hierarchical image database. In Proc. CVPR, 2009.
Jambulapati, A., Li, J., Musco, C., Shiragur, K., Sidford, A., and Tian, K. Structured semidefinite programming for recovering structured preconditioners. In Proc. NeurIPS, 2023.
Ding, R., Zhou, T., Su, I.., Ding, A. A., Xu, X., and Fei, Y. Probe-me-not: Protecting pre-trained encoders from malicious probing. In Proc. NDSS, 2025.
Kim. Jung, S.. Kim, B.. Choi, M., Shin, J., and Lee, J. To- wards safe self-distillation of internet-scale text-to-image diffusion models. arXiv preprint arXhv:2307.05977. 2023.
Dosovitskiy, An image is worth 16x16 words: Trans- formers for image recognition scale. In Proc. ICLR, 2021.
Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint 14Xiv:1412.69888 2014.
Evans, D. J. The use of pre-conditioning in iterative meth- ods for solving linear equations with symmetric positive definite matrices. IMA Journal of Applied Mathematics, 1968.
Krause, J., Stark, M., Deng. J., and Fei-Fei, L. 3d object representations for fine-grained categorization. In Proc. ICCV Workshops, 2013.
10

---

Model Immunization from Condition Number Perspective
Kress, R. Numerical analysis. Springer Science & Business Media, 2012.
Rockafellar, R. Convex analysis. Princeton Mathematical Series, 28, 1970.
Kumari, N., Zhang, B., Wang, S.-Y., Shechtman, E., Zhang,
Roeder, G., Wu, Y., and Duvenaud, D. K. Sticking the landing: Simple, lower-variance gradient estimators for variational inference. In Proc. NeurIPS, 2017.
R., and Zhu, J.-Y. Ablating concepts in text-to-image diffusion models. In Proc. ICCV, 2023.
LeCun, Y. The MNIST database of handwritten digits. http:/yann. lecun, com/exdb/mnist// 1998.
Rosati, D., Wehner, J., Williams, K., Bartoszeze, I.., Gon- zales, R., Majumdar, S., Sajjad, H., Rudzicz, F., al. Representation noising: A defence mechanism against harmful finetuning. In Proc. NeurIPS, volume 37, 2024a.
Lewis, A. S. The convex analysis of unitarily invariant matrix functions. Journal of Convex Analysis, 1995.
Rosati, D., Wehner, J., Williams, K., Bartoszcze, L., Saj- jad, H., and Rudzicz, F. Immunization against harmful fine-tuning attacks. In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024b.
Luenberger, D. G., Ye, Y., et al. Linear and nonlinear programming, volume 2. Springer, 1984.
Marchal, N., Xu, R., Flasmar, R., Gabriel, I., Goldberg, B., and Isaac, W. Generative Al misuse: A taxonomy of tactics and insights from real-world data, arXiv preprint arXiv:2406. 13843, 2024.
Saad, Y. Iterative methods for sparse linear systems. SIAM, 2003.
Saarinen, S., Bramley, R., and Cybenko, G. Ill-conittioning neural network training problems. SIAM Journal on Scientific Computing, 1993.
Montoya, A. and DataCanary. House prices advanced regression techniques, 2016. Kaggle.
Saratchandran, H., Wang, T. X., and Lucey, S. Weight conditioning for smooth optimization of neural networks. In Proc. ECCV, 2024.
Mordukhovich, B. S. Variational analysis and applications. Springer, 2018.
Neel, S., Roth, A., and Sharifi-Mall ManiiiiiMaaajrrdd,,.... S. Descent-to- delete: Gradient-based methods for machine unlearning. In Proc. ALT, 2021.
Savani, Y., Trockman, A., Feng, Z., Schwarzschild, A., Robey, A., Finzi, M., and Kolter, J. Z. Antidistillation sampling. arXiv preprint M.Xix:2504.131400 2025.
Nenov, R., Haider, D., and Balazs, P. (Almost) Smooth Sailing: Towards numerical stability of neural networks through differentiable regularization of the condition number. In ICML Differentiable Almost Everything Work- shop, 2024.
Schramowski, P., Brack, M., Deiseroth, B., and Kersting, Safe latent diffusion: Mitigating inappropriate degenera- tion in diffusion models. In Proc. CVPR, 2023.
Sekhari, A., Acharya, J., Kamath, G., and Suresh, A. T. Re- member what you want forget: Algorithms for machine unlearning. In Proc. NeurIPS, 2021.
Nesterov, Y. Lectures on convex optimization, volume 137. Springer, 2018.
Tamirisa, R., Bharathi, B., Phan, L., Zhou, A., Gatti, A., Suresh, T., Lin, M., Wang. J., Wang, R., Arel, R., et al. Tamper-resistant safeguards for open-weight llms. arXiv preprint arXiv:2408.00761 2024.
Nguyen, Q. P., Low, B. H., and Jaillet, Variational bayesian unlearning. Proc. NeurIPS, 2020.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, nigh-performance deep learning library. In Proc. NeurIPS, 2019.
Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., and Li, L.-. YFCC100M: The new data in multimedia research. Communications of the ACM, 2016.
Petersen, K. B., Pedersen, M. S., et al. The matrix cookbook. Technical University of Denmark, 7(15):510, 2008
Van der Sluis, A. Condition numbers and equilibration of matrices. Numerische Mathematik, 1969.
Qu, 7.,, Gao, W., Hinder, O., Ye, Y., and Zhou, Z. Optimal diagonal preconditioning. Operations Research, 2024.
Wightman, R. Pytorch image models. https: /github. com/ rwi www.gh.taaa.pptttccc-iiieee www.::/ooooooo.mmmmmmmmmeeeeeeee 2019.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In Proc. ICMI., 2021.
Wu, G., Hashemi, M., and Srinivasa, C. Puma: Performance unchanged model augmentation for training data removal. Proc. AAAI, 2022.

---

Model Immunization from Condition Number Perspective
Xu, X., Yue, X., Liu, Y., Ye, Q. Hu, H., and Du, M. Unlearn- ing isn't 't deletion: Investigating reversibility of machine unlearning in Ilms. arXiv preprint arX1v:2505.16631, 2025.
Zhan, Q. Fang, R., Bindu, R., Gupta, A., Hashimoto, T., and Kang, D. Removing rlhf protections in gpt-4 via fine-tuning, In Proc. NAACL, 2024.
Zhang, G., Wang, K., Xu, X., Wang. 7., and Shi, H. Forget- me-not: Learning to forget in text-to-image diffusion models. In Proc. CVPR, 2024.
Zheng, A. Y. and Yeh, R. A. Imma: Immunizing text-
to-image models against malicious adaptation. Proc. ECCV, 2024.
Zheng, A. Y. and Yeh, R. A. Multi-concept model immu- nization through differentiable model merging. Proc. AAAI, 2025.
Zheng, A. Y., Yang. C.-A., and Yeh, R. A. Learning to ob- struct shot image classification over restricted classes. In Proc. ECCV, 2024.
Zhuang, F., Qi Z., Duan, K., D., Zhu, Y. Zhu, H., Xiong, H., and He, Q A comprehensive survey on transfer learning. Proceedings of the IEEE, 2020.
12

---

Model Immunization from Condition Number Perspective
Appendix
The appendix is organized follows:
In Sec. A, we provide the complete statements of the properties of Ree for minimizing the condition number.
Sec. B, we provide the complete proof for the Theorems stated the main paper.
Sec. C, we provide additional experiment details. The code will be open-sourced upon the acceptance of this paper.
A. Properties of the Condition Number Minimizing Regularizer
Theorem A.1 (Properties of k-minimizing regularizer (S), Theorem 2.1, 3.1, in Nenov et al. (2024)).
(I) [Nonnegativity] A S E RRP XPe, Rean(()) 2 0. S + 0, Rean (S) 0 if and only if S has full rank and ss() 1.
(2) [Upper Bound] k(S) < ( Rmn(5), ie., r(S) is an upper bound of (o())((()) as long as min is bounded away from O.
(3) [Differentiability]] of 01 > 0. for any i 1, i.e. o is unique, then Ruell (S) is differentiable and its gradient is given by s Rooel (S) 11 S.
(4) |Monotonic Decrease] If o is unique, update with V RS) such that S' S 1/1 V S for x(S)-1 (1-))=(5)+)' then (s))) RS)).
M11 <
B. Proof of Propositions and Theorems
B.1. Proof of Proposition 3.2.
Proposition 3.2. The singular values of the Hessian matrix in Eq. (9) are given
D
Wi (L,..,,,,D"
(10)
Here, do,; and ug,; correspond to the i-th singular value and vector of Next, 7/ and qj correspond to the j-th singular value and vector of the covariance K.
Proof. Substitute the SVD of 0 and the eigendecomposition of K into 0J K0:
K0 (T
Simplify the expression:
8 K# = Vo(EUU QT UVE,VV
Define M = E QT, so that:
8 ---K-VAL VAMM V
The elements of M are:
M Mi.j] = - (3)
where J' 's for i E [d] are the singular values of 8, 73ys 's for e [d] are the diagonal entries of I, and (u),. .4)) measures the alignment between the i-th column of Up and the j-th column of Q.
13

---

Model Immunization from Condition Number Perspective
We observe the following decomposition of . in matrices O and D:
M
0
v Σ (o
ODD
) where O is an orthonormal matrix, i... O = I, and D = diag(d),..., da) with d; )s is
diagonal matrix, As result, diagonal entries of 2 are:
d
(o0, (o0,((u),44)))²² ==1
Thus, MM = (OD) = OD and the eigenvalues of 0 K are the diagonal entries of D given by:
de =
(0 (u) i= d. d.
j=1
B.2. Proof of Theorem 4.1
B.2.1. PROOF OF THEOREM 4.1 (1)
Theorem 4.1. (l) For any S RP..Xe, Rmn (S) 2 0, and Rin (S) 0 if and only if (S) co.
Denote RS)(()) = (( [|STI2).
Proof. By definition, Rin(S)-
then we have Ru (S) =
|||[ (
(S) and
(S) = 2 o Σ ot
Σ
0o - of)
2k 1=1
< 0,
since Vie [k], o = S o.. As result, (S) > and Rm (8) = [R] 0, 0.c. Rin (S) is nen-negative.
14

---

Model Immunization from Condition Number Perspective
Also, by definition, 01 K (S) ok. Therefore.
2
Run (S)
2/==1 of
(onin)
2
k o + k-1
(ogg
2
(0
k
2
1 of (oggn) }
h
2k
(n(S)2 1) (a
If (SS) 0o, (1n(S) 5
2h (x))))=
0 for o > 0, which yields (S) = 0 given that RRn(S) 2 0.
Similarly, we have
2
Ru11 (S)
Li=1 of
(dgin)
2
2
k-
o
(o
2
E
2
k-
of (agin)
k
2k
Ā-I
(M(5) 1) (ogin)
If Ran (S) = 0, we have (S) 2
(ogo) + 1 = co which yields x (S) co.
B.2.2. PROOF OF THEOREM 4.1 (2)
- [1S]]? To prove Theorem 4.1 (2), we start by analyzing Rin (S) = ((ogi))-
with the following lemma.
Lemma B.I. For (S) = 1/2 ((0320)² - x +11117).
x(S) _et-rn "
(19)
That is, (S) is an upper bound of log
i.e., - -o)(()(S).
((5)
Proof. Similar to the proof of Theorem 3.2 in (Nenov et al., 2024).
1
2R (S)
(agin)
|
k
k
nin 2
Σ o
T: i=1
nin)
1
(k: 1)o + (o 2)
k.
k) 1) (login)² of
15

---

Model Immunization from Condition Number Perspective
In the meantime,
2 log
logg (x{5))
((S))
log
(g 2)
log (oi)
(0² -005 2)
$ 07
1 of
(l55m)² of
in which the inequality follows from the Mean Value Theorem. As a result,
1
S 200 (00
x(S)
I e 20 + 2k
x o (5)
Theorem 4.1. (2) [og(n(()) S ( Run (S), ie., (S) upper bounds [og[x([]] when d'g reasonably away from
00.
Proof. Taking the logarithm of Lemma 1. we have
k: log (S)) S
oi 2R (S).
-
Negating both sides,
k
log (x (5)) 2
01
R1nn1 (S).
k --1
Finally, taking the reciprocal,
of
5
log (x (S))
k
R/m
(N (S))
-
of
k
(ogim)²
2
1
||
2
2h
< 5 RiR (S)
B.2.3. PROOF OF THEOREM 4.1 (3)
To analyze the differentiability of (S)(S) |S| 3 (03)))
start by analyzing the differentiability of (S)
(lggm² I... which needs the following lemma prerequisite.
Lemma B.2 (Theorem 3.1 in (Lewis, 1995) without Convexity). If a function :: E. is absolutely symmetric, that is, y x E Rp and any y as a permutation of x. f(x) fly), then f is differentiable matrix S R.. xv2 if and only if is differentiable at = (S). In this case, for the singular value decomposition S = V
V (f UDiag (V(
16

---

Model Immunization from Condition Number Perspective
Proof. For the forward direction, by Corollary 2.5 in (Lewis, 1995), for S U'iaago)VV
UDiag(u)VV) 1 E (f(())
By Theorem 25.1 in (Rockafellar, 1970), since f o a is differentiable at matrix S E XP2 we know that its subgradient (f o ar) (S) is a singleton, meaning that UDiag (4) V+) is unique, and consequently, ja E 0f(ar is unique. As a result, (r) is also singleton, which, again by Corollary 2.5 (Lewis, 1995), indicates that is differentiable at a. The reverse direction holds true following similar argument.
Lemma B.3. For S UDi in which o = [01, ,ok]" such that d's = 00 02
S i.e. is differentiable and for thy, Us as the h column vector U,
ok < for any < k, (S) (10g))2 -
[11 -
V,
T 1 S. (S)
nin 4k K:
(20)
Proof. For x E R denote
R/m (x)) min
2 2 kE[k]
1
R. (2))
Σ o
2k:
With - = (ogim))
|S F. we first analyze (a Proposition 4.9 in (Mordukhovich, 2018), we have for x E R*
By the subdifferential of piecewise minimum given by
26
1
Og R R1, (x))
On
è arg min JE(M
2
z,e;
i arg min jE(A]
2 2
x i arg min jE[l]
in which e; is the 5th vector from the k-dimensional standard basis. Therefore,
O Ri,, c (rielli arg min jj jElk
Since for any < k, a < oi i.e., the minimum non-zero singular value ogin is unique, we know that the subdifferential
(a:e; i arg minje 0j] = {a } is singleton. Therefore, by Theorem 25.1 in (Rockafellar, 1970), we know R41,1 is differentiable with respect to and V (o) o Regarding = or (S) as a function of S in which or (.) represents taking the singular values of a matrix, we have by Corollary 2.5 in (Lewis, 1995)
0s (gg)))
Os (R (S)
(Diaag(H) p dg (1)
Given that R/n,1 is differentiable and apparently also absolutely symmetric with respect a, by Lemma B.2, we know (ogin)² is also differentiable and
(o nin 2) = UDiag V. (
V
UDiaag nin ( V
og "
17

---

Model Immunization from Condition Number Perspective
In addition, we have
ds
1 2k:
|||
0s
1 2h
Σ σ
(5)}
UDiag(()
a O R R1
by Corollary 2.5 in (Lewis, 1995). R/m1,2 is apparently differentiable with VR 22 = ta.. Therefore, again by Lemma B.2,
||
UDiag VR
2k
UDiag (as) V
k
1
S.
k
By the linearity of gradients,
VR) = -V 4/2 2
(7(1)
2 (ogin)
||S||
2k
=os
min
us Uk
S
which completes the proof.
Theorem 4.1. (3) Ifa = ok < for any' k, then (S) is differentiable and g
GLULD 1 I )(
Proof. Since Rm)(S)- SS
we have
(0
-
25 |
2 (o
0R (S)
1
||| 1 (ogin)
2h
[ (0
24 |||1
23 || 1 (0
2R (S)
|| (ogin)2) 2 2k
By Lemma B.3, we know that if ogin = o < o, for any i < k Rín (S) is differentiable and VR = - 2 S. Consequently, (S) is differentiable and
n n 2 - S
o
VRn (S)
|| 1 (0gin)
26
B.2.4. PROOF OF THEOREM 4.1 (4)
Theorem 4.1. (4) If niin is unique, update with Rmn((( such that S' --n - s for 0 m2 <
&
zx S 2- 1/S (ogin)?)= then (S') > K (S).
18

---

Model Immunization from Condition Number Perspective
Proof. Given that S' = S 1/2 (S) and that (S) =
(ornso) - 5) for
du || }²)
Rn
(S) = (ok k ]
S' = 5 - 12 (S)
12
S
k. S
R1 (S)2
722
+ RRn (S)2
S
R111 (S)2 Tkusok
1/2
1/2 R/111 (S)2
(5)2 kk
RR (S)²
KR (S)2
7/2
o,u00 + 11 R: 111 (S)
1/2
(1)3
1/2
0 uo
UDiag(oss) VT.
where ass = [[1+
Tha (Sy
Tk kh-1,
(1+
Te k (8)
Tee R
(8y
is the vector formed by
the singular values of S' but not necessarily in the decreasing order.
Now we argue that (1 (5))
σ remains to be the maximum singular value while (1 + (S) (11
T2
the minimum. Since 0% < o, for any i < k, i.e., o Tg is unique, we must have 3 < 1 such that ox B0k-11 Also, given that 7/2 <
k k-1 26 || 1 (S)) we have 1 + KR.
7/ R
(S) (s > 0. Therefore,
1 + RR (S)2
122
1/2 R/111 (S)2
1 1+
Tp
1/2
ER (8)
R (8)
1R (S)2
Ok
kR (S)7
1/a
1/2
Ri. (S)=
RR' (S)²2
1
1 +
1/2
0k
KR 112 (S)
1/2
RRA (S))
0k
1/2
1
RR (S)²
B Th
1 1/2
RR/m (S)²
0k-1-
(1
Since 01 2 (2) 2 ... 2 0k-1
> Th and 1 +
82
0, we know that of =
m2
and
KR
(S)
k (5)
aginn = (1 (1+ + nR. m2
oh- Finally,
(S)²
Ri
0 nax
K (S")
S'
I
FR
Okk
KR (ST
K (S)
1 KR (S)
122
(8)
K (S).
19

---

Model Immunization from Condition Number Perspective
B.3. Proof of Theorem 4.2
Theorem 4.2. For H (0) A K0, its maximum and minimum singular values or and o are unique, then
(1) RoRzel1 ( (0)) = 2K0 011110
Drr 1 (00.
(2) VaRm (H (0)) 2K 2K0 Aa -200 (O*KO)
(xe l K0lli--1 ===
Proof. Given that H (0) T K0 for K = X X, we know H (0) is symmetric and positive semidefinite. Therefore, for compact SVD H (0) = Diag (o) V we have U V.
(1) When the maximum singular value 01 of H is unique, we know from Theorem A.I (3) that Ra011 (H (0)) is differentiable with respect to H, and V WH (H (0)) = ui D H = o1 Du H.
Given the form H (0) 0 K0, we have dH (d0) T KO + T K (d0). Furthermore,
(dRz (H (0)) = (V H Rell (0)), dH)
= Tr
Dia H)
(dA) KO + TT K (d8)
Tr
(d0)
K0
+
H
0
(dd)
Dio
Dia
Tr K0 Tannio
H Dan
1
T
(do)
+ Tr
Taaaio (d0)
1
0 K
- Din
in which (,, p denotes the Frobenius inner product, and that last equality follows from the cyclic property of trace. As result, following the derivatives of traces as in Eq. (100) and Eq. (104) in Petersen et al. (2008),
OR (0))
V RoRelll H (0)
00
KO
01U111 Din
HH T
+
(0r0101
-1_111 "Din
1
H1
T
0
K
(vi1T)) T-HHH HT) KTO (olaie)
1 H
K0
ai
Din
Dan
2K0
Din H
(2) When the minimum singular value og of H is unique, we know from Theorem 4.1 (3) that Run (H (0) is differentiable with respect to H, and V Rim (H (0)
$ h
Following similar arguments as in (1), we have oof)"
2K6 -H)
VoRm (H (0)) = -
B.4. Proof Theorem 4.3
Theorem 4.3. For the trainable feature extractor 0. feature covariance Hp (0) = 0T Ko0 of the primary task and Hş (0) the
@ Kg of the immunization task with rank (Hp) kp, rank (Ha) kg and compact SVD Hp (0) UpDiag(d:)V,
Hu (0) U,Diag(au) ( VAT for de [02,1." ... Jp og [on,1,"" "u,tu)
(1) if oax is unique, i.e., H, Tp.1 > op.2, update such that 0' 0 - 1pp K V VoR (Hp (0)) for 0 < mp <
min
then (0' (KTTO)) < (0T Kp0).
[1 Dip
oz
2 73
20

---

Model Immunization from Condition Number Perspective
(2) if Hy is unique, i.e., min Oa O. update 0 such that 0' 00 1/a VVRRi Hg (0)) for m
2kn 0 K (otin) 2) then (8' Ku(f) KnØ).
1-20 /km
Proof. (1) By Theorem 4.2 (1), we know VoRweI1 (Hp(() = 2Kp0 (op,
1 Ho). Since = 0
Dia
Tpp K RoRoon1 [Hp (0)), we have
of Kp0' = (0- K VoRwon (Hp (0))) Kp (0 Kp 'I VoRuol (Hr
K K,0 (FP.1U,,1u71 - Din
1
Hp )
Kp 0 2TpKp Kpl (0..Uep.1pp1 - Dia
Hp ))
0 2np8 (0p,189,10.1
Din Hp) )) Kp
( (2-2 (o..19.11o)
2mp0
Hp
Dan
1 Hp
0 Kp8 2010
o Kp0 2npd" K20 op..Up,, Up,11
Hp
Dim
Dan
+ A1/2 00,110p,107711
1 Dan
HHpp
Hp
0 Kp0
Op,,10,,UP,1.
Hp
Dan
Hp 2np (0p.IUp.Iup_1
Hp
H₂ - 21p (0p.10p.10..
Dan -Hp
Din
T
1
+ 41p 0p,110,110.1
Hp
Hp
Op..UUp.UUp11
Hp
Dan
Dim
Since Hp (0) = T0 KpO for K₁ Xp Xp is symmetric and positive semidefinite, we know for Hp (0) U,Diag (op) Vp it holds that U₂
Vy. Furthermore,
hp pp. Up Up
0p.10p,1
T
H₂ Up,
D
Din
1=1
p. up,1
Up
Din
Din 1=2
UpDiag (ōp) VØ
VeDiag (ōp) Ve
for (ōr) = [11- Dis )
0p,1, Dea 0p...
Des kp Therefore, plugging this and the SVD of Hp back in,
00 Kp0' = -VDDiag (Op) V₂ -2mp (V,Diag (āp) V3 VyDiag (op) Vp
- -2np (V,Diag (o+) VOT) V:Diag (ō+) VT
+ (VaDiag (ō+) VØT) VoDiag (o=) V, VęDiag (ōp) VØ
VoDiag (op) V. 2mp V,Diag (op) Diag (ap) Vp 2mp ViDiag (σp) Diag (Øp) Vp
+ 4n VADiag (ōp) Diag (op) Diag (öp) VØ
= ViDiag (op) VP 2np V,Diag (ōp 3 op) VP - 21p VyDiag (or 0 Vp
+ Anp ViDia (ō, og σr) VeT
VęDiag (op Ampo c oo 4mpo σ 1 0 V.
= VoDiag (at) V.
σ.,1 47p
02. + 4np (1- Dio
op1 if
in which of To.."" ob
for of.
D
de-
Tp,
+ D.
Ane
oū. +
4n
Dr.
op.
if
notes element-wise product and the second equality holds by the fact that Vo orthonormal, i.e., V, V
21

---

Model Immunization from Condition Number Perspective Since H, is unique, know that 3 a such that 0p,
p. Therefore,
41pp
41pp
0. 0p2
0,2
+
P.2
Din
= 1
41pp
41lp 0,2
0,2
p,2
Dia
D
1
4tke
Di
Jp,2
4n2 Din
0.2
0..
α
With DpP < result,
00,,10r.--02.2 2 23
we have 1 +
Di
pp2 +
D
1np 2
<
417p
(1-
Din
1
op,1 +
+11/ (1 D.n)
oF.1
As a
Ante
4ng
1
Di
Tp,2 +
D2,
< <l<a,
1 41pp 1
0a,1 41 (1 - D/o
that is,
1
Ane D
p22
And D
P.2
1 4np (1-
1
op,1
+ 41p
1-
1
2
op.1
a
Dio
Dio
Plugging this result back in,
1
Ani
Die
d.p
+
Ang
DE
p.2
p,2
Op.1
a
1
1
2
1 417p 1
Ta,1 + 41p
1
0p.1
0p.1
Dia
Dia
0p,1-
In addition, ot.2 op. + Ane o2 + for i 3.... hp by definition. Therefore, of, remains to be the maximum singular value of 0 Kpl', and of b the minimum. Finally,
Ank 22. 2 σp. + ,
4n D
o for = ..... kp since dp.2 Oe,i
D. Dí
K (00T Kg0))
Tp.1 4np 1
D
1
0ā.1
+ 4mp 1
Dio
11
2
op1
Op,
+
41p
Den
op
+ 13% op
4n 3
0..1
4np
Dim
2
+ 41 (1
Di
3
Op.
0p.1
kg.ko
(0 Kp0)
where the second inequality holds when T/
<
which indicates that -4pr (1- oū +
07.1
4np (1 D.o of < 0.
22

---

Model Immunization from Condition Number Perspective
Denote Rin (Hg)
HElle then by Theorem 4.2 (2), we know VoRın (Hg (0)
2kg
2kg on (a Ma Mil Ha) Since 6' K VoRin Hy (8)), we have Rin (Ha)?
8¹ Kyt' = =00 -f VEEe-VVRRR VoRin (H (0)) Ka (A
Ka
VoRmn Ha (0)))
T
27m
2
1
S K#0
HB
0 K2O
00 Kn0 on Ua Uni
Ha
Rin (Hx)²
kin
R/a1 (Ha)²
ki
4m
1
+ R/111 (Hn)
OHH Ke UH
Hu
O KRO
UH Un
H
kis
kin
2MJa
1
2mm
1
HHs
OH,A, Un Valba
HH
Hu
Hu
THH Ua Vi,ke
HH
R/11 (Ha)²
Ki
R/111
(Hn)
kin
4n
OHHhy Az VRL
Ha
HB
OH Un.An Va,ka
T
H8
R/111 (Hg)²
kig
Riu
Since Hp (0) H KgO for Kn X₂1 Xa is also symmetric and positive semidefinite, we know for Hg (0) Vn. Following similar arguments as in (1),
UşDiag (ox) ViT it holds that UH
hE-1
u Un, + (1
1
kn ku UR
Hu
TR Ut, Vg,
ka
kp
ViDiag (ōx)
VB
On for Diag (ōn) [[ Ou.,
1 0z
Since V1 is orthonormal, i.e., V/ Va I,
ki
00 K2O'
ViDiag (0x) VH
277h
(VaDiag (ō+) ViT) VæDiag (ax) V4
R/mn
(H₂)
27/
VaDiag (oa) VAT)
ViDiag (ap) Vp
R/4 (H₂)²
4np
VVaDiag (ōa) VaT)
ViDiag (Ox) VB ViDiag (Øx) Vi
R/111
(Hz)*
4m
4m
VæDiag
On 1 on +
o
o
dB
Vi
R/m (Ha)
R/m1 (Ha)
VæDiag (o+) VAT
for o'4
o
Øh.
4n Ra
(H)
Hii +
by
Rin Hn)
And
H.,
if i < kis
OB
(H,)
4np
(
Au
4n
(n)
(1
)
o
if kig
and @ denotes element-wise product.
Since min is unique, we know that 3 E (0, 1) such that oB,An = 38 A Hu
Then we have
4rp
4n
1
JB.kk
R'11
(H₂)²
1 -
kin
H.
km
R'11
(Hz)²
kin
3
41/a
4n
1
2
R/111 (H₂)²
kig
TH.
+
R/m11
(H₂)²
hx
Hi
JH.
41jB
4ng
2
+
o Kz
BoB
R4111 (Ha)²
ky
R'm (H₂)²
hx
417a
4 2
R.
4176
4n R.kn
Sm H.
BoRkku-1
Ru (Ha)²
k R/
(Hn)*
R/a (Hn)*
R/1 (Ha)*
Ru (Ha))
23

---

Model Immunization from Condition Number Perspective
Letting 0 fm <
Ri (Ha)²
|10T
K |
2 (o)2
2 we have
2
1-200 2 /ha
1- 1-20
20
2km
41/
2 2
8n R, < 0.
R.
(21)
R/111 (Hx)²
R/11
(Ha)
R (Ha))
Also,
Ane
1
4ng
(1
1
o
(1-
2n
(1 (1- 1
2
0 for any
Rin Hn)
R/ (Ha)
R/ (Ha)²
Ku
The > 0. Given that Øg > Tz.kk and Eq. (21),
B <1
+
R/ Hn)
41h
HH
kg R/ (H.)"
4np
H.Au
41h
4ng
hy R (Hn)
Az
Big
Ri
(n))
+
4ng
(Hn)
H,
Ang ky Rin Hn))
B,Ka 11
2
+ R/n (H.)
4m
Hu)
Rin Hn))
4mm
H. +
4n R/i1 (Ha)) y
H.Au
Rr (Hr)) H,
8m
indicating
1 +
Ru H2)
Arn
OHAA +
RE
(Ha)
H.
Rin (Ha) OH,Kg + Rin (Ha)"
4ng
H,
Ru Hz)
Sm
o H.hz
3 < 1+
Rn
OH.
k R Ha)
4ng
H, "H Therefore,
ongo H.
+ 4ng
8mb RgR (Ha)*
db =
Bo,
Rg
(Ha)
R (Hg)
Rím (Hx)²
R4m (Ha)*
41p
4ng
+
+
OH,
Rg (Hg)
R
(Hg)²
OHA
In addition, dh O't of for = 1,-.. by since OH S Thi for i = 2, ... Ky 1 by definition. That is to say, o remains to be the minimum singular value of Kg0', and % the maximum. Finally,
4m Rg H2)) H. 1 i H,) H, S on. Ry (H.) H, R (H H
41l
4m
4
Ki TT KgO")
o/H,1
T H,
1 +
4th
OH,1 +
4 k (Hn)
T oa.1
Ka R/
(H.))
(1
Ry SRR (Ha)"
4nh
OH +
Rua (Hn)*
And
H,
2
Rh. H.))
4nhy
OR,Au +
Ri,, (H)) R.
And
02
02 R H. H,Kgz
Oz
4rp
4n
Ra (Hn)
OH,1
R
(Hn)
TH.1
4n
1 Re (Ha)" OH.
R (Ha)*
H.Ay
OH.
Tx.1
Oh.kk
= K (0 K20)
where the first inequality holds by Eq. (21) and the second by σH,1 >> OB.
24

---

Model Immunization from Condition Number Perspective
C. Detailed Experiment Setup
C.1. Datasets
Stanford Cars (Krause et al., 2013) contains 16,185 images of 196 car models and focuses on fine-grained image classification. Country211 (Radford et al., 2021) dataset used for country classification based on satellite images, comprising 211 country-level labels, each with 150 training images. This is subset of the YFCC100M dataset (Thomee et al., 2016) providing user-generated photos and videos, used for domain adaptation evaluation.
C.2. Immunization training details
We summarize the hyper-parameters of training for model immunization in Tab. 4. We choose Ap and X by balancing the gradient norm of Rual1 and R111- Specifically, we obtain the scale of Xe and X first and search over multiples of {1, 5). For linear models, we search over the set of {0.0005, 0.001, 0.005, 0.01} and report the best result. For ImageNet we followed the default learning rate = 1 x 10-5. The number of epochs is based on early stopping using RIR and the test accuracy. All experiments are conducted using float64 precision to ensure numerical stability and reduce potential inaccuracies in computations.
Table Hyperparameters for immunization training.
Dataset
Model
Ap
A
Epochs
C
HousePrice
Linear
0.005
100
1 x 10'
1000
Mean squared error
MNIST
Linear
0.001
1
5 x 107
30
Binary Cross-entropy (CE)
ImageNet vs. Stanford Cars
ResNet18
1 10
5 10-5
2 106
3
Labol-smoothing CE
ImageNet vs. Country211
ResNet18
1 x 10
1 10-4
2 106
3
Label-smoothing CE
ImageNet Stanford Cars
VIT
1 x 10-5
3 10-6
3 10a
2
Label-smoothing CE
ImageNet vs. Country211
VIT
1 10-5
10-6
1 10
2
Label-smoothing CE
Details of immunizing linear models. For the regression task, the linear feature extractor 0 E R 799799 is a randomly initialized dummy linear layer, as discussed in Sec. 4.4. We handle missing values in the tabular data by filling NaNs with O. Categorical features are converted into numerical values using LabelEncoder Finally, the features and labels are normalized using their respective mean and standard deviation. To create Dp and Da, we split the House prices dataset (Montoya & DataCanary, 2016) by the feature MSZoning. Specifically, all entries where MSZoning 'RL' assigned to Da. while the remaining entries form Dp.
For the binary classification task on MNIST, the linear feature extractor 0 E 784x784 is also a randomly initialized dummy linear layer, and we construct training dataset by selecting two specific target digits. The dataset is created using a custom BinaryDataset class, which filters the original MNIST dataset to include only the chosen digits and assigns new labels: one digit mapped to label and the other to label 1. To ensure balance in the dataset, we limit the number of samples for each digit to the smaller count between the two. For optimization, we use Adam (Kingma, 2014) with 3 (.9, 0.999) and
1 10-8 instead of the basic gradient descent in Alg. 1. For the linear model, computed the Hessian inverse by solving regularized least-squares system, where the Hessian is in the shape of IR Di, D Here Din 79 for the regression task and D,, = 784 for the image classification task.
Details of immunizing non-linear models. The pre-trained ResNet18 and ViT are loaded from Pytorch Image Mod- els (Wightman, 2019) with the model name resnet and vit.base.pat ch1 ccllpp.224 We also create the dataset with the built-in function create dataset from Wightman (2019). The feature embedding sizes for ResNet18 18 and ViT are 512 and 768, respectively. To facilitate balanced training when dataset sizes differ, we implement Comb ineeiieddee reddderer which pairs batches from two data loaders. The longer dataset dictates training duration, while the shorter dataset cycles continuously using itertools. The number of epochs reported in Tab. 4 corresponds to the epochs of Da, i.e., the shorter loader.
For optimization, we use SGD with Nesterov momentum to optimize Eq. (11), setting an initial learning rate of 1 x 10-5 with momentum 0.9. The trainable feature extractor parameters are optimized with zero weight decay, while the classifier parameters use a weight decay of 2 x 10-5
25

---

Model Immunization from Condition Number Perspective
C.3. Pseudo-code of the dummy layer
We provide the Pseudo-code for implementing the dummy layer in Fig. 4 below. The DummyLinear layer extends torch. nn .n...nninn Linear and incorporates optional preconditioning mechanism in the backward pass using the inverse feature covariance matrix. The LinearFunct inn class defines the forward and backward computations, where the forward pass applies a standard linear transformation XW + b and stores the input, weight, and bias for gradient computation. In the backward pass, the input gradient is computed normally, while the weight gradient is modified based on whether preconditioning is enabled (use precond=True). If enabled, the weight gradient is adjusted by solving a regularized least-squares system using the inverse of the feature covariance matrix X X + el, improving numerical stability.
class LinearFunction: tetaticmethod
def [esward(ctx, input, weight, bias, lembda_reg, use_precond):
Save input tensers for backward
stx.save__aar war__farrroaaaaeeee (input, waight, blas)
lambda reg lambda_reg etx.use_precond use_precond
Compute output imput..m(weighht..m
biss is not None:
output una.ueennee))) expand_asssseeeeeeeeeeeeed pout
return output
@staticmethod
backward (ctx, grad_ourput]]
Retrieve saved tensors
input, weight, biag atx.arved_tensor lambda_reg
lambda_ree
use_precond ctx. use_ppreddd
Initialize gradients
grad_input grad wrag_w grad_bias
cex.reeda www.rredddnuuut.__eeeee input_grad[0]] [0]
qrad_input emad_output.n (weight)
etx.reeds, etx.reeds_input_pee :rad|]]g
base_ gane__rad__eight grad_outpot.t. ()..ttt......mmn.m.))mm (input)
if use sse_precond:
xtx (inputt))
lambda_eye sarbda_reg torch.eye (Xt.aaaaeee) device-XtX.dev
XtX_reg
lambda_eye
veight = linelg. solve (XtX_reg, besiggrad_weight)
else:
grad_weight a basa_grad_weiggh
if biss is not None and ctx.needs_iiee needs_input_geed[[]]
qrad_blas - grad_outpu..sunt
return irguiinguz, weight, bias, None, None
class DummyLinear (nn. Linear):
def forward(self, input, lambda_reg, use_precend) Dynamically decide whether to use the covariance Inversion as the precondi foner
LinsarPunction. apply (input, self. weighteight, self.bias, lambda_reg, use_precond)
Figure 4. Dummy layer with selective inverse feature covariance matrix in backward function.
26


Reviews:
[{"review_id": "JB54ukVutc", "reviewer": "Reviewer_c1aR", "summary": "This paper proposed a framework for studying model immunization, i.e., the task to make fine-tuning on harmful datasets harder. The authors proposed that for linear models, the difficulty of fine-tuning can be characterized by the condition number of the Hessian matrix. Based on this theory, the authors proposed two regularizers to increasing the condition number of harmful dataset while keeping the condition number stable for the pre-training task. The authors further proved that both regularizers are differentiable, and the optimization goal can be achieved through gradient-based algorithm. Empirical result on linear models supports the theory and shows better performance than other baselines. The authors also empirically evaluated the method on two non-linear models ResNet18 and ViT. The results are also promising and better than baselines.\n\n## update after rebuttal\n\nThank you for the further clarifications from rebuttal. They have addressed all my concerns. I personally agree immunization is an interesting and important direction to explore more.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive and constructive review. We answer individual questions below.\n\n> Q1. Justification for task and dataset selection\n\nOur experiments consist of two settings: (a) a linear model setup, where the setting matches our proposed theoretical framework, and (b) a deep-net setup, where we experimented with non-linear models despite the theoretical gap.\n\n**For linear models**: For the regression task, we choose the House Price dataset as it is a widely used tabular dataset, e.g., in intro ML courses. For the classification task, we choose MNIST as it is the most basic image classification dataset. Additionally, linear models are effective on these datasets.\n\n**For deep-nets**: The transfer learning setting follows from [A] (See Line 352). The chosen Stanford Cars dataset and Country211 dataset are simply the first two datasets presented in [A]'s Figure. 4 (the official ICML version), where they demonstrate linear probing to work well. We now provide experimental results on the third dataset from their Figure. 4 (Food-101) as $D_{\\tt H}$ in the table below. Here the feature extractor backbone is ResNet18. We observe that the proposed method is also effective in Food-101. \n\n- [A] Radford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" ICML 2021.\n\n| $D_{\\tt H}$ | Eq. 17 (i) $\\uparrow$ | Eq. 17 (ii) $\\downarrow$ | $\\tt RIR_{\\theta_0}$ $\\uparrow$| $D_{\\tt P}$ Test Acc. $\\uparrow$ |\n| -------- | -------- | -------- |-------- |-------- |\n| Food-101   |  1.712 |  0.571   | 3.045 |  63.74% |\n\nWe will clarify the motivation of these setups in the experiment section.\n\n\n\n> Q2. Explanation of the experiment setup for ResNet18 and ViT\n\nWe choose to update only the last two convolutional blocks of ResNet18 and the final transformer block of the Vision Transformer (ViT) following common practices in transfer learning. For this experimental setup, we are starting the immunization with a model pre-trained on ImageNet. Typically, only the final layers are updated for efficiency reasons. We will clarify this choice.\n\nWe now provide the results of immunizing the entire ResNet18 backbone on the Stanford Cars dataset below. When training the entire backbone, we observe a similar result, but with a slightly lower immunization effect on $D_{\\tt H}$ and test accuracy on $D_{\\tt P}$. Note that the running time for the full model is more than twice that of only updating the last blocks.\n\n| Trainable module | Eq. 17 (i) $\\uparrow$ | Eq. 17 (ii) $\\downarrow$ | $\\tt RIR_{\\theta_0}$ $\\uparrow$| $D_{\\tt P}$ Test Acc. $\\uparrow$ |\n| -------- | -------- | -------- |-------- |-------- |\n| Entire ResNet18    | 2.102   | 0.672    | 3.127  | 61.48%   |\n| Last two blocks    | 2.386    |0.699  |  3.467   | 62.36% |\n\n\n\n> Q3. The challenges for modeling immunization for non-linear models and generalizing the framework to non-linear models\n\nOn the theoretical front, characterizing the Hessian in arbitrary non-linear models remains challenging. In particular, the Hessian of the linear model admits a tractable form for which we can analytically relate its condition number to the singular values of the task-specific data matrix and the shared weight matrix. As a result, our theoretical guarantees on gradient updates with respect to the feature extractor $\\theta$, which relies heavily on rigorous matrix analysis, are yet to be generalized to non-linear models. **The proposed regularizations, however, are applicable to bounding the condition number of general matrices, including the Hessian of non-linear models**.\n\nHence, we have tested the empirical performance of the immunization framework on various non-linear models including ResNet18 and ViT with linear probing. As demonstrated in Sec. 5.2, the results validate the practical effectiveness.", "claims_and_evidence": "* Claim: immunization to harmful fine-tuning, at least for linear models, can be charactered using the condition number of the Hessian matrix.\n* Evidence: theoretically, condition number is a known indicator for the difficulty of gradient-based optimization; empirically, the experiment results, including the proposed method and $R_{ill}$-based approach also support the claim.\n\n* Claim: (linear) model immunization can be modeled as an optimization problem with the objective shown in equation (11), and can be solved with algorithm 1.\n* Evidence: (1) minimizing $R_{ill}$ will increase the condition number of harmful dataset, this regularizer is proven to be optimizable using gradient descent with guaranteed increase in condition number. (2) the regularizer $R_{well}$ is proven by Nenov et al. to be optimizable using gradient descent. (3) both regularizers are proven to be differentiable w.r.t. model parameters $\\theta$, and has closed-form gradients. (4) an implementation of algorithm 1 in PyTorch and successful evaluation.\n\n* Claim: solving equation (11) would make the model hard to fine-tune on harmful datasets while maintain a good pre-training task performance.\n* Evidence: empirical experimental results on linear regression and image classificationl\n\n* Claim: solving equation (11) also shows promising results on non-linear models.\n* Evidence: empirical experimental results on ResNet18 and ViT.", "methods_and_evaluation_criteria": "The evaluation criteria, relative immunization ratio (RIR), make sense. The paper also provided condition numbers for both harmful dataset and pre-training dataset.", "theoretical_claims": "Seems correct, have not checked carefully.", "experimental_designs_or_analyses": "The experiments use three baselines for comparison. The selection of these baselines are reasonable.\n\nThe experiments use a few exemplary tasks for linear and non-linear models. While the tasks and datasets are all common and widely used, the selection of these tasks and corresponding dataset is not fully justified. Therefore, it's possible that the proposed method would not work as well on other tasks and datasets, especially for non-linear models.\n\nFor ResNet18, the last two convolutional blocks are updated; for ViT, the final transformer block is updated. Not sure why this setup.", "relation_to_broader_scientific_literature": "Model immunization is a promising approach to make open-weight models safer against malicious fine-tuning. This paper advances the state-of-the-art in this field.", "essential_references_not_discussed": "Have not found.", "other_strengths_and_weaknesses": "Maybe it would be obvious for experts, but I would hope to see some explanations for the challenges for modeling immunization for non-linear models.", "other_comments_or_suggestions": "No", "questions_for_authors": "* What are the challenges for modeling immunization for non-linear models?\n* Can the proposed framework generalize to non-linear models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DCI7HDWitK", "reviewer": "Reviewer_CAgr", "summary": "The paper reframes the immunization task, i.e., make models robust against finetuning on specific task, (Contribution 1: Section 3) from a novel perspective using condition number. \n\nThrough this insight, the authors propose a regularization method at the pretraining stage that makes finetuning for specific tasks more difficult (Contribution 2: Section 4.1, 4.2). They theoretically prove that using this regularization increases the condition number for tasks they aim to learn and decreases it for tasks they want to immunize against in linear model setting (Contribution 3: Section 4.3). To demonstrate the effectiveness of this regularization, they empirically showed improved relative immunization ratio for linear regression and image classification tasks on both linear models (Contribution 4-1: Section 5.1) and deep neural networks (Contribution 4-2: Section 5.2).", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful review and respond to questions below.\n\n> Q5. Generative task\n\nWe consider the immunization against linear probing (Sec. 3). For generative tasks, linear probing is not commonly used for transfer learning. Instead, other techniques, e.g., LoRA, are more common. Nevertheless, we now report on immunization against linear probing for a generative model. \n\nWe experiment with CVAE on the MNIST dataset. Let $D_{\\tt P}$ be digit 0 images and $D_{\\tt H}$ be digit 1 images. The CVAE uses MLP for both the encoder and decoder. The last layer of the decoder is used for linear probing, and all other layers are treated as the feature extractor. The loss $\\mathcal{L}$ is the negative ELBO. \n\nAs in the paper, we report RIR to evaluate immunization. To evaluate generation quality, we report the accuracy of a digit classifier (with 99.3% acc. on the MNIST test set). \n\nIn **Tab R1**, our approach successfully immunizes the model against $D_{\\tt H}$ without hurting the performance on $D_{\\tt P}$, as indicated by a large RIR and acc. of 100% on the generated images. \n\nIn **Tab R2**, we report linear probing on $D_{\\tt H}$ across different epochs on models w/ and w/o immunization. We observe that the acc. of the model w/o immunization gradually learns to generate $D_{\\tt H}$ (digit 1), while the immunized model struggles to learn $D_{\\tt H}$ (digit 1).\n\n**Tab R1.** Results of immunizing CVAE on MNIST.\n|| Metrics|\n|-|-|\n|Eq. 17 (i)|1128.4 |\n|Eq. 17 (ii)| 0.016| \n|RIR|68971.8|\n|Classification acc. on $D_{\\tt P}$  |1.0 |\n\n**Tab R2.** Acc. of generation on $D_{\\tt H}$ w/ and w/o immunization.\n|Linear probing epoch|5|10|15|20|25|\n|-|-|-|-|-|-|\n|w/o Immu. |0.30|0.37|0.47|0.54|0.69|\n|w/ Immu. |0.01|0.00|0.02|0.01|0.01|\n\n> Q6. Test acc. on $D_{\\tt H}$ for deep nets v.s. fine-tuning epochs\n\nThe acc. on $D_{\\tt P}$ is fixed as reported in Tab.3 in our paper. In Tab. R3 & R4, we further report the linear probed (fine-tuned) results on different feature extractors and provide the test acc. on $D_{\\tt H}$ during the first 20 epochs of linear probing (fine-tuning). Here, $D_{\\tt H}$ refers to the Stanford Cars dataset. The acc. are reported every 5 epochs. As shown, our method exhibits the slowest convergence rate with on both ResNet18 and ViT; indicated by the lowest acc. compared with baselines. We will include a plot visualizing these results in the paper.\n\n**Tab R3.** Test acc. on $D_{\\tt H}$ with ResNet18 as the backbone.\n|Method/Fine-tuning epoch|5|10|15|20|\n|-|-|-|-|-|\n|Pre-trained model|13.6|18.2|21.0|23.8|\n|$\\mathcal{R}_{\\tt ill}$ Only | 12.8 | 16.6 | 19.9 | 23.8 |\n|IMMA|16.8|20.9|23.7|26.4|\n|Opt $\\kappa$|12.9|17.8|19.9|23.3|\n|$\\bf Ours$|9.5|15.5|18.0|21.4|\n\n**Tab R4.** Test acc. on $D_{\\tt H}$ with ViT as the backbone.\n|Method/Fine-tuning epoch|5|10|15|20|\n|-|-|-|-|-|\n|Pre-trained model|30.7|42.2|51.4|60.3|\n|$\\mathcal{R}_{\\tt ill}$ Only|8.8|20.9|23.3|39.0|\n|IMMA|23.31|35.7|47.4|58.7|\n|Opt $\\kappa$|11.8|20.1|27.5|42.1|\n|$\\bf Ours$|7.9|14.6|24.5|34.0|\n\n> Q7. Any other papers use RIR for evaluation?\n\nTo the best of our knowledge, we are the first to study immunization from a condition number perspective. Therefore, we are not aware of other works using RIR as an evaluation metric.\n\n> Q8. Should the dummy layer be applied to every weight parameter in deep nets?\n\nUnder the linear probing setting, the dummy layer only needs to be inserted at the last layer of the feature extractor, and the rest of the layers can be trained normally. Intuitively, we are treating a deep-net to be extracting \"learnable feature\" $\\mathbf{x}$ followed by a linear feature extractor $\\theta_L$, that is, we view a deep-net $f_{[\\theta_1, \\dots \\theta_L]} = \\mathbf{x}(\\theta_1,\\dots, \\theta_{L-1})^\\top\\theta_{L}$. Note, our theoretical result is limited to linear models and does not fully justify such constructions. We believe this is an important future direction, and we believe the linear model's results provide a promising first step.\n\n> Q9. References on adversarial prompting \n\nThanks! We will review the suggested [1, 2] along with other related works on concept erasing methods to provide a more comprehensive view of the field.\n\n> Q10. Suggestion on proof sketch\n\nWe will adjust the length/placement of the proof sketches. This would also help to create space for the additionally suggested experiments for the main text.\n\n> Q11. Numerical stability of Reg. terms\n\nIndeed, $\\mathbf{S}$ being a diagonal matrix composed of $\\sigma_\\min$'s is the only case for gradient explosion. We would note, though, that the premise of our theorems, e.g., Eq. (13), is the minimum singular value $\\sigma_\\min$ being unique, which would prevent this case in theoretical derivation. In practice, even though the Hessian could be low-rank, we observed that $\\sigma_\\max$ is usually much larger than $\\sigma_\\min$, in many cases to the extent of several orders of magnitude. Therefore, we did not empirically observe issues with gradient explosion.", "claims_and_evidence": "Theoretical claims and evidence: The main framework of condition number and its theoretical connection to the immunization task is very novel and clear.\n\nExperimental evidence: This will be discussed in detail in the methods and evaluation criteria section.", "methods_and_evaluation_criteria": "As the paper itself acknowledges (In section 4.4), its biggest weakness is the experimental setting. I would like to see two major improvements from the rebuttal.\n\nFirst, there are no results for generative models. As mentioned in the introduction regarding text-to-image models, immunization is a more important issue for generation than classification. While the theoretical contribution of this paper is sufficiently commendable, it's difficult to strongly recommend it due to the lack of coverage on generative models.\n\nFor the immunization on generative model, I would be satisfied only on the simple dataset like MNIST. For example, it would be good to compare convergence speed after training an unconditional or conditional diffusion model on data excluding the digit 7 and then finetuning on 7. But if you have a preferred setting, feel free to use that.\n\nSecond, the paper heavily relies on the RIR metric. While I agree that immunization can be approximated by RIR, due to the limitations of 2nd order approximation of loss landscapes in neural networks (unlike linear models), it's difficult to claim that improvements in RIR directly translate to improvements in resistance on actual finetuning. For deep neural networks, I would like to see **test accuracy** on both D_H and D_p simultaneously. Drawing a Pareto curve w.r.t finetuning epochs or hyperparameter and showing superior results than baseline methods would be ideal. Currently, immunization on D_H is only reported through RIR, which I think doesn't accurately show how effective this method actually is for DNNs.\n\n**The second problem appears quite critical to me**. I find it difficult to improve my score to \"accept\" if this experiment is not conducted. (Minor Q) Are there any papers dealing with immunization in non-linear settings that use RIR as the main evaluation metric? If so, please share them and I will take that into consideration. \n\n---\n\nIf both issues are **perfectly** addressed, I think this paper has **oral-level** contribution.", "theoretical_claims": "I was not able to verify all proofs provided in the appendix. The theorem results seem reasonable.", "experimental_designs_or_analyses": "As stated in Methods And Evaluation Criteria, not addressing generative models and using only RIR as a performance metric for immunization are major issues.", "relation_to_broader_scientific_literature": "As far as I know, this research is the first to apply condition number to a field other than optimization contexts related to convergence speed. Condition number seems to have potential applications in areas beyond safety (the goal of this research), such as model interpretability and other fields. I would like to give high credit for being the first research to apply condition number to a different field.", "essential_references_not_discussed": "Adversarial prompt to generate unsafe images by T2I models could be one of problem that can be solved from an immunization perspective. Please consider add this kind of technique. \n\n[1]: Circumventing Concept Erasure Methods For Text-to-Image Generative Models, https://arxiv.org/abs/2308.01508\n[2]: Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?, https://arxiv.org/abs/2310.10012", "other_strengths_and_weaknesses": "The writing is clear, the motivation is also very good. I think the only weaknesses of this paper is the experiment setting.", "other_comments_or_suggestions": "(This is only a suggestion, so I would appreciate it if the authors could consider them and incorporate them at their favor.)\n\nS1: Is the proof sketch of theorem 4.1 really necessary for the main text? I believe that even if the paper is aimed at a theoretical point, this part should be relegated to the appendix and more experimental results should be presented.", "questions_for_authors": "Q1: I'm curious about numerical stability of regularization term due to the 1/x form. What are the conditions for gradient explosion? Looking at eq (13), it seems like explosion occurs when S becomes a diagonal matrix composed of sigma^min, is that correct? I'm curious if this is the only case where explosion occurs. If so, since Hessian matrices typically have low rank structure, numerical stability could be further justified.\n\nThis question is just to improve the theoretical contribution of this paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "odyDQ8snd9", "reviewer": "Reviewer_uVHQ", "summary": "This paper proposes to achieve model immunization, that is, pretraining a model that is hard to fine-tune on some harmful tasks while preserving the performance of other tasks, by maximizing the condition number of the corresponding harmful fine-tuning objective so that the convergence is slow and numerically unstable. Specifically, they propose a differentiable regularization that is proved to increase the condition number of the matrix regularized and further extend such property to the model immunization setting for the linear model. Experiments are conducted for both linear models as well as nonlinear neural networks with various architectures and the proposed method is shown to effectively increase the condition number on the harmful task objective while preserving that of the primary objective.\n\n---\n\n## update after rebuttal\n\nAfter reading the authors' rebuttal, I feel comfortable recommending **strong accept** for this paper.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and encouraging feedback. We are glad the reviewer found the theoretical formulation sound, the proposed regularizers technically solid, and the empirical results convincing. We address the specific concerns and questions below.\n\n> Q12. Manipulating the condition number of objective functions seems computationally expensive. Could the authors justify how the proposed method could be generalized to an even larger scale?\n\nThe computational cost depends on two main factors: the number of samples and the complexity of computing singular values. For the first, in large-scale problems, the Hessian of the full dataset can be approximated using only a minibatch. For the second, as the reviewer points out, the proposed method requires only the maximum or minimum singular value of the regularized matrix, which (particularly in high-dimensional settings) can be efficiently computed using techniques such as the Lanczos algorithm [1,2] or randomized SVD [3,4]. These approaches reduce the computational complexity from $\\mathcal{O}(d^3)$ for a full SVD of a $d \\times d$ matrix to $\\mathcal{O}_k(d^2)$, with the rank-dependent factor absorbed into $\\mathcal{O}_k$. This reduction is beneficial given the typically low-rank structure of Hessian matrices.\n\n[1] Cullum and Willoughby. Lanczos algorithms for large symmetric eigenvalue computations: Vol. I: Theory. Society for Industrial and Applied Mathematics, 2002.\n[2] Golub and Van Loan. Matrix computations. JHU press, 2013.\n[3] Halko et. al. \"Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions.\" SIAM review, 2011.\n[4] Tropp. \"Randomized algorithms for matrix computations.\" 2020.\n\n> Q13. How would the model perform on tasks other than the primary task and the harmful task?\n\nThanks for pointing out this interesting direction. We now additionally report the ratio between condition numbers with and without immunization, i.e., Eq. 15 (i) but for all digits, for a model with $D_{\\tt P}$ = digit 0 and $D_{\\tt H}$ = digit 1. \n\nFor digits other than $D_{\\tt P}$ and $D_{\\tt H}$, we observe that the ratio remains close to 1, indicating that immunization does not affect the condition number of the features on other tasks.\n\nFor a theoretical perspective, following the reasoning in Sec. 3.1, the performance on other tasks is intuitively related to the correlation with $D_{\\tt H}$, i.e., the relative angle between the singular vectors. \n\n\n|             | 0 ($D_{\\tt P}$) | 1 ($D_{\\tt H}$) | 2    | 3    | 4    | 5    | 6    | 7    | 8    | 9    |\n|--------------|----------------|-----------------|------|------|------|------|------|------|------|------|\n| condition number ratio | 0.0451         | 9.8571          | 1.7440 | 1.2147 | 0.9582 | 1.0049 | 0.9785 | 1.3007 | 2.2467 | 1.1453 |", "claims_and_evidence": "The proposed regularizer is supported with theoretical proof to show that it monotonically increases the condition number, and is numerically verified that when applied for model immunization, the condition number of the harmful task indeed increases. The inferior convergence of such task with large condition number is also verified in experiments.", "methods_and_evaluation_criteria": "The condition number is a well-known factor of convergence speed and stability in both the classic optimization and deep learning literature. It’s reasonable to motivate and evaluate model immunization from the condition number perspective.", "theoretical_claims": "I have checked the proof of Theorem 4.2, 4.3, and briefly 4.4. The proof seems sound.", "experimental_designs_or_analyses": "The experiments mostly align with the theory and is extended to nonlinear neural networks. The proposed evaluation metric relative immunization ratio is intuitive and reasonable.", "relation_to_broader_scientific_literature": "The paper proposed a new method to achieve model immunization, a recent proposed concept in the broader context of AI safety, and serves as an alternative to IMMA, a bilevel optimization method proposed together with the concept of model immunization, while achieving better performance in terms of preserving the performance of the primary task.", "essential_references_not_discussed": "None to the best of the reviewer’s knowledge.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The proposed framework of achieving model immunization by manipulating the condition number of the objective function corresponding to different tasks is very novel.\n2. The proposed regularizers that increase or decrease the condition number are theoretically well-supported and technically solid.\n3. The numerical experiments demonstrate superior performance compared to baseline methods for model immunization.\n\nWeaknesses:\n\n1. Manipulating the condition number of objective functions seems computationally expensive. Even though the proposed regularizers are differentiable alternatives to the condition number, they still involve the maximum or minimum singular value of the regularized matrix. Could the authors justify how the proposed method could be generalized to an even larger scale?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "The proposed model immunization framework involves a primary task and a harmful task. How would the model perform on tasks other than the primary task and the harmful task?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a novel and theoretically grounded framework for model immunization through condition number manipulation. The authors propose differentiable regularizers that make fine-tuning on harmful tasks more difficult while preserving performance on benign tasks. The paper makes strong theoretical contributions with provable guarantees in linear models, and further demonstrates empirical effectiveness on both linear and deep nonlinear models (ResNet18, ViT, and CVAE).

Pros:
* the use of condition numbers to formalize and optimize for model immunization is principled and innovative.
* the theoretical analysis is rigorous and well-supported, particularly for linear models.
* the authors addressed all reviewer concerns during the rebuttal, including adding generative model results and showing fine-tuning behavior over epochs.
* experimental results show promising immunization effects without degrading primary task performance.
* reviewers appreciated the clarity of the writing, solid empirical support, and potential for broader impact in model safety and unlearning research.

Cons:
* the primary limitation is the theoretical framework's reliance on linear models, with extensions to deep models being empirical only.
* the evaluation metric (RIR) is task-appropriate but some reviewers suggested it should be supplemented with more direct measures of immunization in deep models.

Overall, the paper introduces a promising, technically sound approach to model immunization, with theoretical depth and convincing experiments. Despite limitations in generalizing theory to nonlinear models, the extensive rebuttal and new experiments strengthen its contribution. I recommend acceptance.
Decision: Accept (oral)