Exploring and Leveraging Class Vectors for Classifier Editing
Authors: Jaeik Kim
Jaeyoung Do
Venue: NeurIPS 2025 poster ()
Keywords: Classifier Editing
Vision Transformer
Class Vector
Primary area: applications
TL;DR: We propose the novel concept of class vectors for efficiently and effectively editing modern classifiers.

Abstract: Image classifiers play a critical role in detecting diseases in medical imaging and identifying anomalies in manufacturing processes. However, their predefined behaviors after extensive training make post hoc model editing difficult, especially when it comes to forgetting specific classes or adapting to distribution shifts. Existing classifier editing methods either focus narrowly on correcting errors or incur extensive retraining costs, creating a bottleneck for flexible editing. Moreover, such editing has seen limited investigation in image classification. To overcome these challenges, we introduce class vectors, which capture class-specific representation adjustments during fine-tuning. Whereas task vectors encode task-level changes in weight space, class vectors disentangle each class’s adaptation in the latent space. We show that class vectors capture each class’s semantic shift and that classifier editing can be achieved either by steering latent features along these vectors or by mapping them into weight space to update the decision boundaries. We also demonstrate that the inherent linearity and orthogonality of class vectors support efficient, flexible, and high-level concept editing via simple class arithmetic. Finally, we validate their utility in applications such as unlearning, environmental adaptation, adversarial defense, and adversarial trigger optimization.

Exploring and Leveraging Class Vectors for Classifier Editing
Jaeik Kim Jaeyoung Dot 1.2,
AIDAS Laboratory, IPAII & EECE Seoul National University i indicates corresponding author {jake630, jaeyoung do)@snu. kc.
Abstract
Image classifiers play critical role in detecting diseases in medical imaging and identifying anomalies in manufacturing processes. However, their predefined be- haviors after extensive training make post hoc model editing difficult, especially when it comes to forgetting specific classes or adapting to distribution shifts. Exist- ing classifier editing methods either focus narrowly on correcting errors or incur extensive retraining costs, creating a bottleneck for flexible editing. Moreover, such editing has seen limited investigation image classification. To overcome these challenges, we introduce Class Vectors, which capture class-speciff representation adjustments during fine-tuning. Whereas task vectors encode task-level changes in weight space, Class Vectors disentangle each class's adaptation in the latent space. We show that Class Vectors capture each class's semantic shift and that classifier editing can be achieved either by steering latent features along these vectors or by mapping them into weight space to update the decision boundaries. We also demonstrate that the inherent linearity and orthogonality of Class Vectors support efficient, flcxible, and high-level concept editing via simple class arithmetic. Finally, validate their utility applications such as unlearning, environmental adaptation, adversarial defense, and adversarial trigger optimization.
Introduction
Classifiers have long been fundamental in Computer Vision (CV), applied in diverse fields from medical imaging (35] to anomaly detection [84) With the rise of Vision Transformers (ViTs) [3] 44.
their classification capabilities have significantly improved, leading to the widespread availability of fully fine-tuned models across various tasks. As a result, open platforms such as HuggingFace now offer extensive collections classifiers, enabling plug-and-play usage for diverse applications (66). However, even within the same task, users may have specific requirements for certain deterministic rules. For example, in disease diagnosis, some users may prioritize minimizing errors for specific conditions they handle, others may require a classifier that performs reliably in their own distributional context, such as a snowy environment. Thus, one-size-fits-ll classifier is impractical for meeting diverse user needs within the current model supply chain. This highlights the importance of classifier editing, which modifies class-specifee knowledge post hoc while preserving unrelated prior knowledge (72).
minor misclassification
have serious consequences. Alternatively,
Despite its importance, classifier editing remains challenging because deeply optimized models encode rigid behaviors shaped by their training distributions. For example, data scarcity for vehicles in snowy scenes often teaches the model to adopt the shortcut vehicle + snow + snowplow, causing it to mislabel buses in snow snowplows [60] 266. Moreover, efficiently modifying classifiers with minimal data in real-worke scenarios remains an open problem. Recent classifiers (e.g., ViTs), for instance, require significantly more training compared to traditional CNNs (64 68).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

making knowledge modifications with few samples more difficult [39 36 and increasing the risk of introducing new biases [6. As a result, existing classifier editing methods are computationally intensive [80] and often rely on auxiliary information such object mask, requiring representations to be modified one by one for each image [6)]. These challenges, amplified by the sparse information
density of visual data [23], require defining "where-to-ede and remains underexplored in vision
models, whose scope is largely restricted to correcting image-wise misclassifications [62 80].
To address these challenges, we revisit recent image classifiers through the lens of model's adaptation to specific classes during training by introducing the novel concept of Class Vectors. Class Vectors capture per-class representation shifts during fine-tuning by computing the dif-
Fine tuned
Class Wectors
Ex] Unleaming
Class Veelt
Transition
Error Correcting
ference between the centroid representations of
pretrained and fine-tuned models. Inspired by task vectors [29]. which represent weight up- dates for tasks during fine-tuning, Class Vee- tors aim to disentangle class-specific behavior from task-wide adaptations. Although task vec- tors are effective for task-leved applications such as model alignment [2 66, 43 [5] and detox- ification (29 83
Pretrained
Editting Vector
(a)
(b)
(c)
Figure 1: Class Vector and its applications. Class Vector captures centroid representation adap- tation in the latent space. (b) Editing vector with high-level concepts using arithmetic operations on Class Vectors. (c) Editing vectors can undo predic- tive behaviors by reversing the adaptation direction, or transition the classifier logic to correct errors.
(a)
they inherently capture task- level modifications, limiting their applicability for fine-grained classifier editing. In contrast, Class Vectors operate at the class level and can be applied either by directly steering latent rep- resentations via a training-free approach or by mapping them back into weight space for model editing. This class-level modification alleviates existing editing constraints by replacing predictive rules across an entire class rather than adjusting the model on an image-by-image basis.
Our findings reveal that linear trajectories exist along which the model adapts to specific classes during fine-tune toning forming the basis of Class Vectors. This behavior remains barrier-free despite the complexity of high-dimensional representation learning and the nonlinear characteristics of modern classifiers [57] 556,, supported by Cross-Task Linearity (CTL) [7]. We then explore two key properties of Class Vectors that enable their effective use in classifier editing: (1) linearity and (2) independence. During inter-class interpolation, predictions and logits transition smoothly along linear paths. Furthermore, we demonstrate that modifying a target class's representations does not influence other classes, confirming that Class Vectors act independently. This behavior is supported by the Neural Collapse [55): during fine-tuning, class- specific feature shifts become guasi-ortoogonal. enabling targeted adjustments with minimal interference. These fundamental properties of Class Vectors enable precise editing of class-specific predictive rules through simple arithmetic operations such as addition, subtraction, and scaling (Fig. D. offering several advantages as follows:
High efficiency: Enables edits without retraining via latent steering, or for specific tasks can be trained in under 1.5 seconds using fewer than 1.5K parameters and single sample.
High-level interaction: Facilitates intuitive high-level concept editing, also allowing non- expert users perform edits without neural network expertise.
Flexibility: Provides precise control over the degree and nature of edits by adjusting the scaling coefficients of Class Vectors to align with user intentions.
We present extensive experiments demonstrating the effectiveness of Class Vectors in real-world applications such model unlearning, adapting to unfamiliar environments, preventing typographic attacks, and optimizing triggers for backdoor attacks.
2 Related Work
Adaptation vectors. Empirical studies of neural network loss landscapes show that fine-tuning proceeds along convex, well-aligned directions in weight space [18 73 Building on this, task vectors [29], the weight delta from pretrained model to its fine-tuned version, have been used in
classifiers 76 78), large language models [77 81), and LoRA adapters 10), enabling multi-task

---

learning [2 79.. detoxification [6]], and style mixing 74]. Meanwhile, in large language models (LLMs), in-context vectors steer model oulputs inference time by encoding task- or example-level instructions as additive offsets in latent space (e.g., for controllable text generation 43
25
47
Such adaptation vectors typically perform model editing at the global, wask level. contrast, our Class Vectors isolate adaptations at the per-class level in latent feature space, providing localized, class-specific control with minimal interference to other classes. Unlike task vectors, which impose global weight shifts, Class Vectors capture intrinsic, persistent representation shifts that can be mapped to weights. They differ also from in-context vectors-transienn, label-agnostic offsets-and from concept activation vectors (CAVs) [33, which describe rather than edit concepts.
Characterizing Neural Networks. Early work [8] demonstrated that the loss landscape along the straight- line path from random initialization to fully trained model is nearly convex, suggesting that training could follow a linear trajectory. Linear Mode Connectivity (LMC) 288 15
49 then
showed that independently trained models on the same task maintain almost constant loss under linear
revealed that scaling these vectors yields semantically meaningful performance changes. Layerwise Linear Feature Connectivity (LLFC) [86] extended this phenomenon to the feature space, proving that at every layer the feature maps of an interpolated model align proportionally with the linear blend of the feature maps of the original models that show LMC. More recently, Cross-Tase Linearity (CTL) 87] found that models fine-tuned on different tasks still exhibit approximate linear behavior in their features under weight interpolation, and Neural Collapse (NC) described how penultimate-layy prototypes [55]. In this work, we show that pretrained-tor to--fine-tttt fine-tuned model pairs also satisfy CTL with feature alignment, and we use NC to establish the independence of Class Vectors.
weight interpolation, and the concept of task vectors 29 53
features converge to equidistant class
3 Foundations for Class Vectors
Given n data {21, ,n C Yask with corresponding k labels fci, (c..,,..., cc C y, image classifier is defined MM 0 Rd)= Xask + y, comprising encoder [(, 0c E Rd)): Xusk + Z and classification head g(., 0h E [d) 2 + Y. Here, classifier is represented as M g f. where o denotes function composition. Let ffT E R& represent the pretrained weight and ff E Rd be the fine-tuned weight of the classifier encoder for a specific task. We first define the Class Vector.
Definition 3.1 (Class Vector). For a class e, let S = (81,..., }191} C S denote the set of its samples. The Class Vector Ke E R. is the difference between the expected last-layer representations of the fine-tuned and the pretrained encoders (i.e., penultimate layer of models):
Reg = Ee
- (s,66())
The centroid representation of fine-tuned classifier zi for a class e can be formulated as z + Kig. where *pre denotes pretrained centroid representation and S Kask-
>c "pre
3.1 Formal Justification
We aim to demonstrate that the class-specific changes induced by fine-tuning are captured by single latent vector Kg := z -zpre- so that merely scaling Ke interpolates a smooth path of class-c behavior. To justify this claim, we build on two well-documented phenomena.
(i) Task-level weight linearity. Prior work 22 shows that the task vector 7 0f 0p. captures linearly meaningful direction in weight space: moving the weights along t, f [z; 6 + Xr). causes predictable performance shifts as X varies [8]
(ii) Cross-Task Linearity (CTL) 87 When fine-tuned checkpoints 0,, 03 originate from the
same ppee. weight interpolation is almost equivalent to latent interpolation for every input æ:
f(r; + (1 a)0,)
fz; 0.) (1 - aa f(;;
CTL thus bridges weight-space linearity to latent-space linearity. In Theorem 3.1 beyond any pair of fine-tuned weights, we show that it is even tighter on the segment connecting the pretrained model to its fine-tuned checkpoint, and we confirm that this pretrain-to- fo-neetune interpolation traces a semantically meaningful path in latent space.
3

---

500 N2.
Nar 200
10
(b)
Figure 2: (a) Line-search between Spre and zi explores linearly evolving representation. (b) Linear interpolation between cross- cross-Class Vectors with ViT-B/32 shows smooth transition between classes.
Theorem 3.1 (CTL between pretrained and fine-tuned weights). Suppose the function f IRP +R and two fine-tuned weights 0; and f satisfy CTL [7]. Let pre be the pre-trained weights. Define
pre, = pree + +1 (1-a)0)) ((p))
8 = If(a0: + (1- ((- af
If 1110,-60p- - Ppeel < |10: 0 |l. then 6pre.i < deg; ; the segment from pro to shows strictly smaller CTL,
deviation, hence is more linear, than the segment between two fine-tuned solutions 0: 0-.
The proof is provided Appendix $ SC.I Using Theorem 3.1
we can apply CTL to pair of ppe and 0n. Applying CTL to the set of inputs To E D. for single class and averaging over Te yields:
fre Ppe + aT) = f(zc; Opee) + aree
De.
Thus scaling the fixed vector Kr in latent space reproduces the effect of moving pre along T for class-c samples, the class- specifce adaptation. Fig. (Top) visualizes this effect ViT-B/32 across six downstream tasks: as a increases from to 1, class-c accuracy (normalized by the fully fine-tuned score) rises smoothly and concavely, confirming the linear path predicted by the theory. We observe similar behavior in both an MLP and ResNet-18 18 [22] with two other tasks (CIFÁRIO, CIFAR 100), indicating that Class Vectors arise independently of network architecture finetuning specifics (Fig. 2a (Middle), Fig. (Bottom)), Experimental evidence for the inequality ||0: pre |00 8, and training details for all models are in Appendix Fig. and DD3
Take-away. The adaptation required for single class can be approximated by scaling single latent vector Kei class-wise representation learning often reduces to simple vector arithmetic.
3.2 Class Vector-based Editing
To edit classifiers, we first construct an editing vector Zedit E I the latent space by linearly using the Class Vectors 4. Prior work has shown that task vectors (in the weight space) and in-context vectors (in the latent space) can steer model behavior; our approach supports both injection modes.
Latent-space injection. Given = ()) we shift the representation by Redit and obtain 9 g(r + Rodit, 0²). To avoid collateral edits in other classes (i.e., to ensure localization of the edit), gate the shift with 8 1 (sim(r) > 7.). where sim is given by the cosine similarity z and denotes thresholds for gating (Algorithm.[]. Please note that in most cases, z is known when constructing Zedit ($4). and latent space injection does not require additional training.
Weight-space mapping. Latent-space manipulation of the classifier cannot fundamentally alter the deterministic rules encoded in the model's 's weights, leaving decision boundaries unchanged [48] 30). also imposes additional gating computations on the editor and requires maintaining 2f. Following previous editing approaches [80] 60, 46) that embed edits directly into the model weights, introduce a method for permanently embedding editing vectors into the model parameters. To mapping editing vectors the weight space, learn Øedir "m= RRd, such that
0 Of Hsdin (Fodit)

---

Algorithm Latent-space injection
Algorithm Weight-space mapping
1: Inputs: encoder [(;0%): ): head g(:;0");
1: Input: encoder f(;;; 8"), data Nack Aedit- collecting number N, class c, epochs T
Sedri" class centroid =i: threshold
Output: logits y
2: Freeze all but editable block C
3: r r fa; ff.)
3: for i 1..... . and each (ar, E Xiak do
4: Collect N class-c references To. (f(xz) y= cf
4: sim(r)= - Iil 1
5: Set Timrger mean(r.) + Zedin if Furgg None
5: 3 = 1 (r) > y
6: r= f(x) : rly = c] - Fingger
+ r + Zedi 7: Ú + (rati: 8*)
7: Update L with cecl
end for
8: return y
9: return 6cdh
Addition
300
0.2
Urean ACC 1761
20.0
NOTTL AVg ACC (76)
0.4
29.7
Target
Mon-target Class
99.5
1.00 079 050 4.2 od3 638 on 169
043 2-0
065 S4i 3-0 B+99-0
(a)
(b)
(c)
(d)
Figure 3: Independence of Class Vectors in MNIST. (a) Scaling the target class representation using Zedit a Ne, (b) Adding non-target Class Vectors the target class based on the combination count.
(c) Modifying the target class to each destination class (Zodit = Kdes. ). with the averaged task
accuracy. (d) Shifting all representations from C; + C++1 simultancously with transition success rate.
minimising [z Pedn (Zedr) - zedill 1 To make this more practical, assuming a linear mapping f (a common assumption [81] 17), the optimization reduces to P
Edit
argmin [xx (aan)) (f(xz, 0 + 1
Similar to latent space steering, to ensure that the edit only affects the intended class, first collect reference samples from that class and compute their latent representations. We then add the Class Vector Ka each of these reference embeddings to form fixed target representations, and train the few encoder layers to map the original embeddings onto these shifted targets /Algorithm.22 ( Theorem 3.2 shows that the mapping of Class Vectors from R. into the model's 's weight space Rd admits infinitely many solutions, implying that it remains effective across diverse mapping configurations.
Theorem 3.2 (Existence of a Mapping). Let Pedit R + R& be any mapping that sends latent Class Vectors weight perturbations applied in the encoder's final layer or a small subset of layers. Under the assumption that these edits are sufficiently small and confined to that small subsel of layers of an overparameterized encoder (e.g., ViT) with de >> m, there exist infinitely many distinct dedit-
See Appendix C..2 for a detailed proof. Note that Theorem 3.2 guarantees the existence of valid mapping for overparameterized encoders, provided that the edit is restricted to local subset of layers. This theoretical result suggests that such mappings may be inherently robust to the specific training procedure used for the encoder. The training setup for mapping are provided in Appendix $D.3.2
3.3 Properties for Effective Classifier Editing
We now explore the properties of Class Vectors. Throughout our experiments, we employ the classes listed in Tab. 7 and validate our findings via the weight-space mapping approach.
Linearity. The linearity between two classes a fine-tuned model is crucial for editing, as barriers or divergence along the path may cause zedit to fail, leading to unpredictable behavior. We interpolate between two fine-tuned classes e₁ and 02 by Zedit model's adaptation from C₁ to c2. As shown in Fig. 2b predictions and logits change smoothly:
-a -a + a Kegg This effectively shifts the
5

---

Table 1: Comparison of class unlearning with baselines, including the mean and std accuracies.
Method
MNIST
EuroSAT
GTSRB
RESISCAS
DTD
ACCy (
ACC, (t)
ACCy (U) ACC, (1)
ACC, (1)
ACC, (1)
ACCy (1)
ACC, (1)
ACCy (U)
ACC, (f)
Pretrained
53.4
51.71:42
66.5
53.44225
8188
41.7
76.81:885
66.21004
36.01:358
44,91:08
Fine -uned
99.91:0.1
99.8+0.0
99.9:022
99.8:t0.0
99.6=0.0
99.24:00
99.3
96.8+:000
73.5+169
82.3+04
Retrained
0.140011
76.4000
0.0+000
85.7=00
41,8=25.
57.5+00
33.94206
75.0+a0
14,5
55.5+00
NegGrad
0.0::00
43.41:003
0.0=0.0
11.621.
0.04000
15.6225.6
0.04:00
2.5:06
0.01000
13.621677
Random Voctor
99.9110.1
99.81:00
99.910.1
80.9:126.1
99.6=0.5
98.21:077
100:100
79.41:19.1
72.01318
51111111
Class Vector 0.0:000
99.71:00
0.0:00
99.51:03
0.0::0.0
98.6220.0
28.2=26.1
94,627a
13.5216.5
78.1
Class Vector'
0.0+000
96.2+
0.0+0..
99.7+000
0.0+00.
93.4
10.0=10.9
90.7+332
15.2
72.9+0.1
samples switch cleanly from e₁ to C2 at the midpoint, with no detours to other classes. This results show that Class Vectors permit precise linear edits of the classifier.
Independence To modify class towards 02 without effect other classes, we require f(x', f(z', + of Sedit) )) for all x' $ C1, while f f 0% + ó(Zedit)) = f( 0%) Neural Collapse (NC) phenomenon [55 states that, near the end of training, (i) all penultimate-layer features belonging to the same class tightly collapse to a class mean, and (ii) these class means themselves form a simplex Equiangular Tight Frame (ETF) centred at the global mean. Building on NC structure, we show that a class-specifie update vector K. exerts negligible influence on the embeddings of every other class.
Theorem 3.3 (Independence of Class Vectors). Suppose (i) the pretrained class embeddings col- lapse to common mean gnT that is zpre x gpre (ūi) after fine-tuning the embeddings follow centre-shifted ETF form = = + th with E, uc = 0; and (iii) the global drift || - _tre|| is negligible compared to the class-specific update |uel. Then, for any two distinct classes c. + c'
cos(Kes cost Kee
O,
i.e. the Class Vector Ke is approximately orthogonal to the fine-tuned embedding of every other class.
detailed proof and empirical evidence that strongly support these conditions for ViT are presented empirically evaluate the independence of Class Vectors. It shows that Class Vectors preserve the accuracy of non-target classes and ensure independent edits across classes, even when multiple classes are edited simultancously.
in Appendix CC In Fig. 3
4 Editing Classifiers
We now introduce editing applications with Class Vectors. For all applications, we first design Zedit then steer the models in latent spaces map it to the weight space to alter their predictive behavior.
4.1 Experimental Setups
To evaluate Class Vectors for classifier editing, we extract pretrained and fine-tuned class centroids from three widely adopted CLIP encoders, ViT-B/16, ViT-B/32, and ViT-L/14 (58), and form Class Vectors as their differences. In and $4.4 Class Vectors are derived from initialized and pretrained encoders, as they predict ImageNet classes, the pretraining dataset for these classifiers. We denote latent-space steering by Class Vector and weight-space mapping by Class Vectori, using default similarity threshold 7f = 0.5. As baselines, we include Retrained (80] 60). which retrains only the target class using cross-entropy loss (or excludes it entirely in the unlcarning setting), and Random Vector, which is initialized to match the magnitude Zedit and mapped to the weight space to test for non-semantic effects. Among all considered methods, only the Class Vector method enables latent steering without requiring any additional training.
We note that Task Vector 22 is not included as a baseline in our main experiments, since it operates at the task-wide level and unsuitable for evaluating class-wise editing. For completeness, we additionally provide comparative results between task vectors and Class Vectors in the class unlearning setting Tab. 17 Additional task-specific baselines are described in their respective sections, and full experimental details in Appendix $D

---

4.2 Class Unlearning
Since ViT classifiers are exposed to a wide range of data during training, they naturally encode information across all classes. Practically, class unlearning is intended to modify classifier decision boundaries to prevent classification into specific categories, while minimizing unintended changes to non-target classes, often motivated by privacy or security concerns [52). Existing class unlearning methods typically involve retraining the model or performing multi-stage post hoc edits, which are computationally expensive and risk inadvertently erasing features shared across classes (7]
To evaluate Class Vectors on class unlearning, we set Zedit X Re with À -1.5 effectively modify the model to unlearn adapted predictive rules for class c. We utilize the ViT-B/16 model for our experiments (See Appendix L results on other ViTs). For an additional baseline method, we retrain models with gradient ascent to the target class (i.e., NegGrad) following previous For mapping, we utilize the single reference samples the subset of each task's s test set and evaluate on the remaining data, training only the final layer's layer normalization of the encoder. The first five labels in each task are used as the target (forget) class in each experiment. As shown in Tab. [ Class Vectors demonstrate the most effective editing strategy for unlearning the target class (ACC,) while preserving the performance of the non-target classes (ACC,). In contrast, random vectors have minimal effects, indicating that Class Vectors point to meaningful directions in the latent space. Additionally, retraining struggles with limited data and trainable parameters.
studies [3 40
4.3 Adapting to New Environment
Snowplow- Tonk
Snowplow- Traffic Light
Imbalanced training scenes often lead classifiers to re- duced performance in specific contexts [65]. Thus, adapt- ing open-hub classifiers to suit individual users' environ- ments in an efficient manner is necessary in practice. Fol- lowing previous study [60], we examine a scenario where the model struggles to classify objects in snowy environ- ments (Fig. 4). This occurs when the representation of snowy objects fails to accurately capture the object's 's fea- tures due to the ambiguous influence of snow.
Snowmoble- Firetruk
Snowmoble- Racer
Class Vectors can effectively address this through high- level concept-based arithmetic operations. Specifically, our goal is to eliminate the snow features from the image representation. Therefore, the objective of Zedit is:
Figure 4: Adapting the classifier to a snowy environment. Red text marks the misclassifications made by the original model, while blue text shows the correct predictions after classifier editing.
Cobject g 8h) = g(2 + zean), ())
(1)
where Aedit is designed to eliminate the representation acti- vated by the model when snow is input, ensuring that the model focuses solely on the object.
We consider practical scenarios when only limited external samples are accessible. Namely, the editor can obtain a few images of the target object from external sources. We set Redit as X tes KKe ) Here, +c denotes the representation adaptation of the model with snowy images. At a high level, this retains only the model adaptation related to snow. Editor can steer the model or map Redit to satisfy Eq. [ with < O used to suppress snow features. We evaluate on Snowy ImageNet [60] (7 classes, 20 images each), using 5 reference samples per class for mapping and testing the remainder for mapping experiments. As an additional baseline, DirMatch [60] trains models to align images to target-class representations using external samples individually. We train only the final MLP and layer-norm in the transformer block, reporting mean + standard deviation accuracy across classes. With À - 1.0 and 4 external samples (Fig. [10), Class Vectors deliver a 10-20% improvement over the pre-edit classifier (Tab. 2a), underscoring their high-level interactions and effectiveness.
4.4 Defending Against Typography Attacks
Vision-language pretrained models like CLIP have exhibited vulnerabilities to typography attacks, where the text in an image leads to misclassification of the model [16]. Thus, mitigating typographic attack risk crucial in safety-critical domains, such as medical imaging, before deploying classifiers.
Similar to 44.3 given an object with text written on it, the representation is expected to become
7

---

Table 2: Results on classifier editing with Class Vectors in two scenarios: (a) adapting to new distribution (snowy environments) and (b) defending against typographic attacks.
Method
Average (1)
Method
Average
VII-B/16
VIT-B/32
VIT-L/14
VIT-B/16
VIT-B/32
VIT-L/14
Pretrained
55.2+-24.
23.4
60.2+:22.5
Protrained (Attack)
Pretrnined (Clean)
48.91338.2
75.0+3382
76.74:33.1
100+00
18.9
100
Retrained
55.8++48.4
55.84448.5
75.3+:15.5
Retrained
80.013442
66.71:47.1
98.822.5
Random Vector
26.21:23.0
16.2:122.7
49.7:126.11
Random Vector
41.14:30.7
74.44438.6
33,3++2144
DirMatch
72.04-22.8
73.94223.5
74.64-16.4
DirMatch
97.7
91.1+8.3
13.8+130
Class Voctor
69.7122.6
72.2:338
71.3132
Class Vector
88.9
98, 9::25
93.3+7777
Vector
72.7=214
76,2421.2
18.3+169
Class Vector?
98.9
99.0+
93.3226.7
(a)
(b)
Claan
Tysogruphy Attakk
Inviible Small Panch Oeenn
30
Puyl
30
30
(a)
(b)
Figure 5: Visual examples for the scenarios in
4.4 and $.5
(a) examples of typographic attacks that
cause the model to misclassify inputs as iPod. (b) optimized backdoor triggers on traffic-sign images.
obbecct after classifier editing, allowing the deterministic rules of the model to focus solely on the object for accurate recognition. Thus, our goal aligns with that described in Eq. L Practically, the editor can easily generate small set of augmented samples by directly adding text objects or performing data augmentation. Using them, the Class Vectors o for text-affected images and Kiojject for clean images can be derived by feeding each image sets into the model. Finally, we define Zodit = - - Kusbjcrt). where Zedit removes adaptations from text-object images at a high-level, isolating the model's s clean object representations.
Following a previous study (60). we utilize web-sourced image sets with 6 classes from ImageNet that include both clean and text with objects. Each image is augmented into 15 images per class (Fig.Ba) For mapping. we use total of 4 reference images for training and evaluate on the remaining images. We use the same baselines as $4.3 setting À -155. For DirMatch, text-augmented images are trained to directly align with clean image representations. Results Tab. deddonsteeeee that the Class Vector effectively defends against typographic attacks, achieving average accuracies on par with or exceeding clean-image performance across models.
4.5 Adversarial Trigger Optimizations
Class Vectors can also be utilized for backdoor attacks, which alter a model's 's logic using adversarial triggers such small patches (20) or imperceptible noise (85), typically optimized by training the
classifier to misclassify triggered images [82]. However, primary limitations of these approaches
are the requirement for large amounts of triggered samples and full access to the training process to adjust the model's weights. We consider a scenario where an editor (i.e., attacker) aims to mislead classifier into misclassifying specific classes, without altering the model's weights. The attacker knows the classifier's architecture but lacks access to the user's model or training process. With knowledge of the architecture, the attacker can acquire model with the same design from open hubs. By embedding the intended representation into a trigger patch or an invisible trigger using the same model architecture, the attacker cause the classifier to misclassify any object or scene where the pattern appears, whether on the object itself or attached to the camera lens.
To effectively embed malicious representations into the trigger, the attacker aims to optimize the initialized trigger Tirigger's 's representation into Zedit A (Ney Key) such that when Tirigger is attached to an image, the classifier misclassifies ej c2. Unlike previous sections, Zedit is mapped to the pixel
8

---

space by training the pixels in Fuigger (Algorithm
Fiiggger
argmin
Il (Frigger:
0n) -Zedit)
(2)
Fuisser
Note that, because the backdoor triggers are optimized, latent-space steering cannot be applied in this scenario. Class Vectors evaluated across real-world tasks, including GTSRB, RESISC45, and SVHN. For mapping, 30, 10, and 200 samples are selected from each test set, with performance assessed on the remaining data. We additionally include BadNet (20] for baselines, where weights are manipulated with unlearnable triggered images to classify into the destination class.
We measure Attack Success Rate (ASR) on trig- gered images and Clean Accuracy (CA) on clean images, averaged across tasks. For DirMatch, the triggers are optimized directly toward the destination representation of target class. The scaling coefficient À for Zedit is set to 1.5 for small patches, using 0.8% of total pixels and 1.0 for invisible noise by default. All experiments VIT-B/32 (see Appendix {E results other ViTs). Tab. 3] shows that Class Vectors achieve high ASR, achieving high effectiveness without modifying model weights. As shown in Fig. 5b triggers are optimized be either very small or stealthy, making them imperceptible.
Table 3: Results backdoor attacks with opti- mized triggers.
Method
Small
Invisible
ASR (?)
CA
ASR (f)
CA (f)
Pretrained
19.64426.2
43,8492
22.24-299
43,8499
Finetumed
0.0:a.11
95.247.11
95.24771
BadNet
100:00.0
10.0+50
100400
9.4474
Random Vector
0.1:0.11
95.247711
39.41:55.8
95.2417.
DirMatch
96.5:1:47
95.2447.1
96.8-14.55
95.2
Vector
99.8+228
95.24%7.1
Table 4: Results of class unlearning MNIST using ResNet18, ResNet50, and ConyNeXT-Tiny.
ResNet50
ComNNeXT-Ting
Method
ACC, (1)
ACC, (1)
ACCy (4)
ACC, (f)
ACC, (U)
ACC, (1)
Retmined
99.8
99.5
89.9
99.6
78.2
99.4
NegGrad
14.2
97.2
95.0
11.2
Random Vector
99.7
99.4
99.5
99.4
99.5
99.3
Class Vector
2.2
97.0
11.5
84.2
95.3
Class Vector!
99.4
99.1
99.1
Table 5: Class unlearning with BERT-Base
AG-NEWS
DBPedia-14
20-Newsgroups
Method
ACCy (1)
ACC, (1)
ACCy (4)
ACC, (t)
ACC,
ACC, (1)
Retrained
71.9
89.3
99.0
66.5
NegGrad
0.0
48.9
0.0
93.8
0.0
47.0
Random Vector
93.8
94.3
98.6
99.1
62.9
67.9
Class Vector
0.0
93.2
0.0
96.9
57.9
Class Vector
3.2
94.4
0.0
99.1
63,8
4.6 In-Depth Analysis
Model Generality across Architectures. To examine the architectural generality Class Vectors, we extend our analysis beyond ViT encoders to convolutional and language models, including ResNet18, ResNet50, ConyNeXT-Tiny [45), and BERT-Base [12]. For each model, Class Vectors derived from the pretrained and fine-tuned representations and applied to the class unlearning setting, targeting the first class the dataset. As shown Tab, and Tab. 5 Class Vector maintains strong forgetting performance while preserving non-target accuracy across all architectures. Notably, both the latent-space variant (Class Vector) and its weight-space mapping (Class Vectort) exhibit consistent gains compared to gradient-base based random baselines, confirming that Class Vectors capture transferable, semantically meaningful directions independent network type.
Impact of threshold in latent space steering. In the latent-space steering method, Class Vector injection applied only to inputs whose cosine similarity to the target feature exceeds the threshold y. To examine the impact of y, we perform systematic sweep accompanied by class Maas-unnearniig

---

0.8
Avg. Accuracy (80
Accuracy (8g
0.6
(1)
60
(T)
40
Typo.
0.0
Snowy
-15
(a)
(b)
Figure 6: In-depth analysis: (a) Effect of the cosine-similarity threshold ry on class-unleamingg clean accuracy; (b) Effect of the scaling coefficient À on controllable editing. Horizontal dashed lines denote pretrained model performance.
80
Accuracy (%6]
85
Accuracy
80
75
70
Retrained Class
Class Vector
65
Class Vectort
60
Class Vector!
25
35
50
55
60
25
30
50
55
Sample ratio (%)
Sample ratio (%)
(a)
(b)
Figure 7: Effect of sample ratio on editing performance. Each curve shows how model accuracy varies as the proportion of available samples increases for (a) Snowy ImageNet and (b) Typo Attack.
evaluations to assess both editing efficacy and collateral impact. In Fig. 6a editing remains effective for 7 (0.0, 0.5). and, even at y 0, we observe that edits retain independence, owing to the independence of Class Vectors (Theorem 3.3).
Impact of scaling coefficient. The scaling coefficient À determines both the strength and direction of an edit. To make editing intuitive for non-experts, À should yield predictable, controllable outcomes. For instance, in defense against typographic attacks, learned "iPod" adaptation, while in other cases milder edit suffices. We sweep À in the snowy- environment and typography-attack (Fig. (bb) scenarios, observing clear trends: negative values erase snow features and restore correct predictions, whereas positive values amplify them and degrade performance. This demonstrates that can be tuned reliably based on high-level editing goals.
negative aggressively removes the
Impact of sample size. Retraining-basee methods improve with more samples, but such abundance is rare in real deployments. To test scalability, varied the sample theio-the portion of available target data-in Snowy ImageNet and Typo Attack (Fig. 7a 7b). With less than 30% of data, both Class Vector and Class Vectort outperform retraining, which only catches up after 35% This highlights Class Vector's s strong data efficiency, leveraging latent semantic directions instead of full parameter optimization, and maintaining competitiveness even under low-data regimes.
5 Conclusion and Future Work
We have introduced Class Vectors, which capture class-specific representation adaptations during training. Open-hub models can leverage Class Vectors to modify predictive rules for task-specific personalization. Our analysis of their linearity and independence, supported by extensive experiments, highlights their potential for efficient and interpretable classifier editing across diverse applications. While our work primarily focuses on image classification, we anticipate future extensions of Class Vectors natural language processing (NIP) and generative models, including Large Language Models (LLMs) and image generative models.
10

---

Acknowledgements
This work was supported in part by National Research Foundation of Korea (NRF) grant (RS-2025- 00560762), and Institute of Information & communications Technology Planning & Evaluation (ITPP) grant (RS-2025-022637)
RS-2021-ID 1343). This research was also conducted as part of the Sovereign AI Foundation Model Project (Data Track, 2025-AI Data-wi43), organized by the Ministry of Science and ICT (MSIT) and supported by the National Information Society Agency (NIA). J. Do is with ASRI, Seoul National University.
55-2022--25442335
2015-222--S--- ------S-22-400-0000000
References
[1] A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, al. A general language assistant as laboratory for alignment. arXiv preprint 12X0v:21/2000861 2021.
[2] O. Avrahami, O. Fried, and D. Lischinski. Blended latent diffusion. ACM transactions graphics (TOG), 421 2023.
[3] S. Balasubramanian, S. Basu, and S. Feizi. Decomposing and interpreting image representations via text in vits beyond clip. arXiv preprint arXiv:2406.01583 2024.
[4] D. Bau, S. Liu, T. Wang. J.-Y. Zhu, and A. Torralba. Rewriting a deep generative model. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pages 351-369. Springer, 2020.
[5] L. Beyer, J. Hénaff, A. Kolesnikov, X. Zhai, and A. v. Oord. Are we done with imagenet? arXiv preprint 1xXiv:2006.07153: 2020.
[6] R. Bhardwaj, D. D. Anh, and S. Poria. Language models are homer simpson! safety re-alignment of fine-tuned language models through task arithmetic. arXiv preprint arXiv:2402.11746, 2024.
[7] W. Chang, T. Zhu, H. Xu, W. Liu, and W. Zhou. Class machine unlearning for complex data via concepts inference and data poisoning. arXiv preprint arXiv:2405. 15662, 2024.
[8] Z. Chen, Y. Bei, and C. Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):772-78) 2020.
[9] G. Cheng, J. Han, and X. Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10): 1865-1888, 2017.
[10] R. Chitale, A. Vaidya, A. Kane, and A. Ghotkar. Task arithmetic with lora for continual learning. arXiv preprint arXiv:23/ 1.00442,, 2023.
[11] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proc. of Computer Vision and Pattern Recognition (CVPR), pages 3606-3613, 2014.
[12] Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter the association for computational linguistics: human language technolo- gies, volume 1 (long and short papers), pages
2019.
[13] A. Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. Proc. of Int'l Conf. on Learning Representations (ICLR). 2021.
[14] L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, B. Tran, and A. Madry. Adversarial robustness prior for learned representations. arXiv preprint arXiv: /00tv 00945, 2019.
[15] Frankle, G. K. Dziugaite, Roy, and M. Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Proc. of Int'l Conf. on Machine Learning (ICML), pages 3259-3269, PMLR, 2020.
[16] G. Goh, N. Cammarata, C. Voss, Carter, M. Petrov, L. Schubert, Radford, and C. Olah. Multimodal neurons in artificial neural networks. Distill, 6(3)=e00, 2021.

---

[17] J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 141..5.412.6672.222 2014.
[18] I. J. Goodfellow, O. Vinyals, and A. M. Saxe. Qualitatively characterizing neural network optimization problems. Proc. of Int'l Conf. on Learning Representations (ICLR), 2015.
[19] Y. Goyal, A. Feder, U. Shalit, and B. Kim. Explaining classifiers with causal concept effect (cace). arXiv preprint arXiv: Aekiv:/000 /907.0077000 160,66622000000 2019.
[20] T. Gu, B. Davitt,, and S. Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv: 700.. 0000000633,, 2017.
[21] H. A. A. K. Hammoud, U. Michieli, F. Pizzati, P. Torr, A. Bibi, B. Ghanem, and M. Ozay. Model merging and safety alignment: One bad model spoils the bunch. arXiv preprint arXiv:2406. 14563, 2024.
[22] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proc. of Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[23] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proc. of Computer Vision and Pattern Recognition (CVPR), pages 16000- 16009, 2022.
[24] Helber, Bischke, A. Dengel, and D. Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):211-2226, 2019.
[25] R. Hendel, M. Geva, and A. Globerson. In-context learning creates task vectors. arXiv preprint arX 15916, 2023.
[26] J. Hinns and D. Martens. Exposing image classifier shortcuts with counterfactual frequency (cof) tables. arXiv preprint arXiv:2405. 15661, 2024.
[27] C. Huang, P. Ye, T. Chen, T. He. X. Yue, and W. Ouyang. Emr-merging: Tuning-free high- performance model merging. arXiv preprint arXiv:2405. 17461, 2024.
[28] G. Ilharco, M. Wortsman, S. Y. Gadre, S. Song. H. Hajishirzi, S. Korbblith, A. Farhadi, and L. Schmidt. Patching open-vocabulary models by interpolating weights. Proc. of Neural Information Processing Systems (NeurIPS), 35:29262-2993 2022.
[29] G. Ilharco, M. T. Ribeiro, M. Wortsman, S. Gururangan, L. Schmidt, H. Hajishirzi, and A. Farhadi. Editing models with task arithmetic. Proc. of Int'l Conf. on Learning Representations (ICLR), 2023.
[30] H. Karimi, T. Demr, and J. TTng- Characterizing the decision boundary of deep neural networks. IEEE International Conference on Machine Learning and Applications (ICMLA), 2019.
[31] T. Karras, M. Aittala, Laine, E. Härkönen, J. Hellsten, Lehtinen, and T. Aila. Alias-free generative adversarial networks. Proc. of Neural Information Processing Systems (NeurIPS), 34:852-863. 2021.
[32] A. Kasirzadeh and I. Gabriel. In conversation with artificial intelligence: aligning language models with human values. Philosophy & Technology, 36(2):27, 2023.
[33] B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In Proc. of Int'l Conf. on Machine Learning (ICML), pages 6678-2677. PMLR, 2018.
[34] B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In Proc. of Int'l Conf. on Machine Learning (ICML), pages
PMLR, 2018.
[35] H. E. Kim, A. Cosa-Linan, N. Santhanam, M. Jannesari, M. E. Maros, and T. Ganslandt. Transfer learning for medical image classification: a literature review. BMC medical imaging, 22( (),,,, 2022.
12

---

[36] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan,
J. Quan, T. Ramalho, A. Trabska-Barwinskie al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3222 2017.
[37] Kodge, G. Saha, and Roy. Deep unlearning: Fast and efficient gradient- freee class forgetting. Trans. on Machine Learning Research (TMLR), 2024.
[38] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny (mages.(2009), 2009.
[39] A. Kumar, T. Ma, and P. Liang. Understanding self-training for gradual domain adaptation. In International conference machine learning, pages 5468-5479 PMLR, 2020.
[40] M. Kurmanji, P. Triantafillou, J. Hayes, and F. Triantafillou. Towards unbounded machine unlearning. Proc. of Neural Information Processing Systems (NeurIPS), 36, 2024.
[41] Y. LeCun, C. Cortes, and C. Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann. http://aaw.eccu...o////ee/// cowpe///////xx/iiissttt. 2, 2010.
[42] E. Liu. Leveraging intermediate neural collapse with simplex etfs for efficient deep neural networks. arXiv preprint arXiv:24/2.00884 2024.
[43] S. Liu, H. Ye, I.. Xing, and J. Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering. arXiv preprint rrXiv:2311.06663 2023.
[44] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proc. of Int'l Conf. on Computer Vision (ICCV), pages 10012-10022, 2021.
[45] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A convnet for the 2020s. In Proc. of Computer Vision and Pattern Recognition (CVPR), pages 11976-11986 2022.
[46] K. Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and editing factual associations in gpt. Proc. of Neural Information Processing Systems (NeurIPS), 35:17359-17372, 2022.
[47] J. Merullo, C. Eickhoff, and E. Pavlick. Language models implement simple word2vec-style vector arithmetic. arXiv preprint arX1v:23355.661.00 2023.
[48] D. Mickisch, F. Assion, F. Greßner, W. Günther, and M. Motta. Understanding the decision boundary of deep neural networks: An empirical study. arXiv preprint arXiv:2002.01810, 2020.
[49] S. I. Mirzadeh, M. Farajtabar, D. Gorur, R. Pascanu, and H. Ghasemzadeh. Linear mode con- nectivity in multitask and continual learning. Proc. of Int'l I Conf. on Learning Representations (ICLR), 2021.
[50] E. Mitchell, C. Lin, A. Bosselut, C. D. Manning, and C. Finn. Memory-based model editing at scale. In International Conference Machine Learning, pages 15817-15831. PMLR, 2022.
[51] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, A. Y. Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 4. Granada, 2011.
[52] T. T. Nguyen, T. T. Huynh, Z. Ren, P. L. Nguyen, A. W.-C Liew, H. Yin, and Q. V. H. Nguyen. A survey of machine unlearning, arXiv preprint arXiv:2209.02299 2022.
[53] G. Ortiz-Jimenez, A. Favero, and Frossard. Task arithmetic the tangent space: Improved editing of pre-trained models. Proc. of Neural Information Processing Systems (NeurIPS), 36, 2024.
[54] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 011-77300227444 2022.
13

---

[55] V. Papyan, X. Han, and D. L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652-24663, 2020.
[56] N. Park and S. Kim. How do vision transformers work? arXiv preprint arXiv:2202.06709 2022.
[57] R. K. Rachman, D. R. I. M. Setiadi, A. Susanto, K. Nugroho, and H. M. M. Islam. Enhanced vision transformer and transfer learning approach to improve rice disease recognition. Journal of Computing Theories and Applications, 1(4):446-400, 2024.
[58] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-878 1188-773. PMLR, 2021.
[59] M. T. Ribeiro and S. Lundberg. Adaptive testing and debugging of nlp models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 3253-3267, 2022.
[60] S. Santurkar, D. Tsipras, M. Elango, D. Bau, A. Torralba, and A. Madry. Editing classifier by rewriting its prediction rules. Advances Neural Information Processing Systems, 34: 23359-23373, 2021.
[61] D. Shirafuji, M. Takenaka, and S. Taguchi. Bias vector: Mitigating biases in language models with task arithmetic approach. arXiv preprint arXiv:24/2.11679 2024.
[62] M. Sotoudeh and A. V. Thakur. Provable repair of deep neural networks. In Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, pages 588 603, 2021.
[63] Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. The german traffic sign recognition benchmark: a multi-class classification competition. In The 201 international joint conference on neural networks, pages 1453-1460. IEBE, 2011.
[64] A. Steiner, A. Kolesnikov, X. Zhai, R. Wightman, J. Uszkoreit, and L. Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. Trans. on Machine Learning Research (TMLR), 2021.
[65] J. Tanha, Y. Abdi, N. Samadi, N. Razzaghi, and M. Asadpour. Boosting methods for multi-class imbalanced data classification: an experimental review. Journal of Big data, 7:1-47, 2020.
[66] M. Taraghi, G. Dorcelus, A. Foundjem. F. Tambon, and F. Khomh. Deep learning model reuse in the huggingface community: Challenges, benefit and trends. arXiv preprint arXiv:2401.13177 2024.
[67] A. Torralba and A. A. Efros. Unbiased look at dataset bias. In CVPR 2011, pages 1522-1528. IEEE, 2011.
[68] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data-efficient image transformers & distillation through attention. In Proc. of Int'l Conf. on Machine Learning (ICML), pages 10347-10357. PMLR, 2021.
[69] T. P. Van, T. M. Nguyen, N. N. Tran, H. V. Nguyen, L. B. Doan, H. Q. Dao, and T. T. Minh. Interpreting the latent space of generative adversarial networks using supervised Icarning. In 2020 International Conference on Advanced Computing and Applications (ACOMP), pages 49 54.54. IEEE, 2020.
[70] B. Wallace, M. Dang, R. Rafailov, L.. Zhou, A. Lou, S. Purushwalkam, S. Ermon, C. Xiong, S. Joty, and N. Naik. Diffusion model alignment using direct preference optimization. In Proc. of Computer Vision and Pattern Recognition (CVPR), pages 8228-8238. 8238, 2024.
[71] C. Wang and Golland. Interpolating between images with diffusion models. Proc. of Int'l Conf. on Machine Learning (ICML), 2023.
14

---

[72] S. Wang, Y. Zhu, H. Liu, Z. Zheng, C. Chen, and J. Li. Knowledge editing for large language models: A survey, ACM Computing Surveys, 57(3):1-37, 2024.
[73] M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Komblithh, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Interna- tional conference machine learning, pages 23965-23998. PMLR, 2022.
[74] X. Wu, S. Huang, and F. Wei. Mixture of lora experts. Proc. of Int'l Conf. Learning Representations (ICLR), 2024.
[75] J. Xu and H. Liu. Quantifying the variability collapse of neural networks. In Proc. of Int'l Conf. on Machine Learning (ICML), pages 38535-38550. PMLR, 2023.
[76] Yadav, D. Tam, Choshen, C. A. Raffel, and M. Bansal. Ties-merging: Resolving interfer- ence when merging models. Proc. of Neural Information Processing Systems (NeurIPS), 36, 2023.
[77] Yadav, T. Vu, J. Lai, A. Chronopoulou, M. Faruqui, M. Bansal, and T. Munkhdalai. What matters for model merging at scale? arXiv preprint arXiv:2410.03617 2024.
[78] E. Yang, Z. Wang, L. Shen, S. Liu, G. Guo, X. Wang, and D. Tao. Adamerging: Adaptive model merging for multi-task learning, arXiv preprint arXiv:2310.02575. 2023.
[79] E. Yang, L. Shen, Z. Wang, G. Guo, X. Chen, X. Wang, and D. Tao. Representation surgery for multi-tast model merging. arXiv preprint arXiv:2402.02705 2024.
[80] Y. Yang, L.-K. Huang. S. Chen, K. Ma, and Y. Wei. Learning where to edit vision transformers. Proc. of Neural Information Processing Systems (NeurIPS), 2024.
[81] L. Yu, B. Yu, H. Yu, F. Huang, and Y. Li. Language models super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning, 2024.
[82] Y. Yuan, R. Kong, S. Xie, Y. Li, and Y. Liu. Patchbackdoor: Backdoor attack against deep neural networks without model modification. In Proceedings of the 11 ACM International Conference Multimedia, pages
2023.
[83] Zhang, J. Liu, He, et al. Composing parameter- -fffecient modules with arithmetic operation. Proc. of Neural Information Processing Systems (NeurIPS), 36:12589-12600, 2023.
[84] W. Zhang, Q Yang, and Y. Geng. A survey of anomaly detection methods in networks. In 2009 International Symposium on Computer Network and Multimedia Technology, pages 1-3, IEEE, 2009.
[85] N. Zhong, Z. Qian, and X. Zhang. Imperceptible backdoor attack: From input space feature representation. arXiv preprint arXiv:2205.03 /90, 2022.
[86] Z. Zhou, Y. Yang, X. Yang, J. Yan, and W. Hu. Going beyond linear mode connectivity: The layerwise linear feature connectivity. Proc. of Neural Information Processing Systems (NeurIPS), 36:60853-60877, 2023.
[87] Z. Zhou, Z. Chen, Y. Chen, B. Zhang, and J. Yan. On the emergence of cross-task lincarity in the soinatning-iietuuning paradigm. arXiv preprint arXiv:2402. 03660, 2024.
15

---

Supplementary Material
Contents
Broader Discussion
17
Extended Related Work
17
A.2
Limitations
17
A.3
3
Ethic Statement
17
4
Licenses
17
Algorithms
18
Supplementary Theoretical Analysis
19
Theoretical Justification of Class Vectors
19
Existence of a Mapping Function
20
Class Vectors Preserve Inter-Class Orthogonality
21
D Experimental Details
21
D.I Class Configurations
21
D22 Task Details
22
D.3 Training Details
23
Additional Experiments
25
Mapping Sensitivities
26
Computational Analysis
26
Supplementary Figures
27
16

---

A Broader Discussion
A.1 Extended Related Work
Model interventions Model intervention aims to adapt trained models to new knowledge for specific user needs. Retraining an entire model is time-consuming and data-intensive, driving the development of more efficient methods that use limited data. These include model alignment [
54
70 32
debugging [59], and editing (29 60 50 4 Model intervention Computer Vision
(CV) remains limited. Previous work propose editing generative adversarial networks (GANs) by identifying and modifying specific locations 44, while this approach is extended classifiers for debugging errors by mapping new rules to existing ones [60]. However, these methods require editing locations or additional data. More recent saaa--lerning-baseeee approach [0) address this but remain computationally demanding.
Latent representations Latent space interpolation in generative models aims to blend the styles of different images by navigating their latent vectors. This approach has been widely explored GANs [69 31 and diffusion models 771 2]. while its application to classifiers remain largely unexplored. Meanwhile, recent efforts to explain deep neural networks have focused on linking models' internal processes to high-level concepts, such as analyzing cumaanunndrstandable features or testing their influence predictions (i.e., decomposability) [8] [4 [4 3 [19]. Building these
ideas, we investigate representation adaptation in the latent space, revealing properties and enabling effective, high-level editing.
A.2 Limitations
While Class Vectors enable efficient and interpretable classifier editing across variety of tasks, our method exhibits several limitations. First, the approach assumes that class representations are well- structured and approximately linearly separable the latent space. This assumption is empirically supported by CTL and Neural Collapse, but may not hold in scenarios with high intra-class variance, noisy labels, or long-tailed distributions. Second, the method currently focuses on single-label classification tasks where each example is associated with single semantic class. Extending Class Vector-based editing to multi-label or hierarchical classification, where class boundaries are less distinct and often overlapping. remains an open challenge. In addition, the latent- steering method relies on access to the centroid representation of each class, which may not be readily available in privacy-constrained black-box settings. Similarly, the weight-space mapping method requires updating small subset of layers with a few reference samples, which still assumes partial access to model internals. This limits the applicability of our method in fully closed-source environments. Despite these limitations, our work lays the foundation for structured classifier editing and invites further research into expanding its scope to more complex, unconstrained settings.
A.3 Ethic Statement
This work does not involve research with human participants, sensitive data, or personally identifiable information. All experiments were conducted using publicly available pretrained models and open- source datasets. We include limited web-sourced or user-generated imagery (e.g., for typographic attack evaluation), and such images used solely for non-commercial, academic research purposes under fair use or Creative Commons-complian terms. While our method enables editing of classifiers for beneficial purposes such as unlearning and robustness, we recognize that similar techniques may be repurposed for malicious intent, such as backdoor trigger optimization. To mitigate such risks, we release code and dataset under non-commercial license (CC-BY-NC-SA 4.0) and emphasize that practical deployment of editing techniques should be preceded by careful threat modeling and access control.
A.4 Licenses
We plan to release our code under the Apache 2.0 license.
17

---

B Algorithms
We present algorithms that leverage Class Vectors by mapping them into non-latent spaces. For latent space steering, refer to Algorithm Specifically, we introduce two approaches: (1) standard mapping an encoder, and (2) pixel-space mapping for adversarial trigger optimization.
Algorithm 3 Pscudocode for optimizing classifier encoder with Class Vector
Require: Classifier encoder (f(:, 0*) and trainable layers), Dataset Kuskk Editing vector Fedit Number of epochs T. Target class Learning rate r)
Ensure: Edited encoder weight Oedi such that f(r E Naski 0*) =f + Zedit
1: Freeze all layers of (0 except the final trainable layers
2: Ftarget t- None
3: Initialize list R +
[to store class-c representations)
4: for epoch
T do
5: for each mini-batch (X, in X' do (Encoder representation!
6: r
f(X, +
7: if Farget = None then
8:
Collect f- -rre+rrVVV c Append r. to if enough class-c reps in R. then
(Representations for target class)
9:
10:
f Me)n(R) (Average representation when reference sample number is satis-
fied]
11:
Imaget F Zedit
(Add editing vector)
12:
end if
13:
Continue to next mini-batch if Turget = None end if
14:
15:
Filter + r|Y= =
[Only align class-c representations
16:
Compute alignment loss: f= Furget |
17:
Backpropagation with
18:
end for
19:
end for
20:
return M (Bdited encoder weight Oddi)
Algorithm Pseudocode for optimizing adversarial trigger with Class Vector
Require: Classifier encoder f(-, 0*) Trainable trigger Tirigger. Dataset Yask, Editing vector Redit
Number of epochs T, Target class Learning rate 1)
Ensure: Edited encoder weight 0 such that (x + Tiriggers 0%) = x" Zedit
Ftareet < None
2: Initialize list R +
[to store class-c representations)
for epoch
do
4: for each mini-batch (X, in X' do (Encoder representation)
r +- f(X + Furigger: 8C)
6: if Larget = None then
Collect r. + -rrY = c]. Append r, R if enough class-c reps in R then
(Representations for target class)
8:
F mean(R) (Average representation when reference sample number satis- fied)
10:
Tiaget + r + Zedit
(Add editing vector)
end if
12:
Continue to next mini-batch if Traget = None end if
14:
Filter i 4 =
(Only align class-c representations)
Compute alignment loss: f f= Furgel
16:
Backpropagation with
end for
18:
end for return Firigger
(Edited trigger Farigger)
18

---

(a) ViT-B/16
(b) VIT-B/32
(c) VIT-L/14
Figure 8: Empirical validation of Theorem 3.1: CTL deviation increases with parameter distance.
Blue markers denote interpolation between pretrained and fine-tuned weights (6pro ++ 0;); red markers denote interpolation between different fine-tuned weights (0, 0z).
C
Supplementary Theoretical Analysis
In this section, we provide theoretical analyses, including proofs for the theorems, as well empirical evidence supporting them.
C.1 Theoretical Justification of Class Vectors
Here, prove Theorem 3.1 and provide empirical results supporting the assumptions illustrated in the figure.
- Theorem 3.1 (CTL between pretrained and fine-tuned weights). Suppose the function f RP + R is
three-times differentiable on open convex set θ C I2, and that its Hessian is spectrally bounded at every Og E O: Amin S f(o)) I Amax 87
Let Opr be the pre-trained weights and 0z, 0z
two fine-tuned weights that satisfy CTL. Define
6pre. = ff pre + - (1-a)),) - -11 (f==))(
81, [f(a0: + (1- a)0,) ((1-))(()))+
If 0 prel < |10, 0, |l. then 6pre, ō.g; the segment from pr to 0g shows strictly smaller CTL
deviation, hence is more linear, than the segment between two fine-tuned solutions 0 + 0j.
Proof. From Theorem 5.1 Zhou et al. [87. for any pair 6a A, O, f: - differentiable on open convex domain θ, and its Hessian satisfies the spectral bound
E is three-times
Amin < | S Amaxs v0 O,
then the CTL deviation is bounded by:
80_ 9, (a(0 + +1-a)0,) - [0.f(..)) a)] < a(1 a.) >maxx 10 0bl1 + E,
where the remainder term E O(I1a0, + - a)0,, - 0ol13) vanishes as the interpolation point approaches 0g
Now fix ppe:: 0, 0g E O, and assume:
|10 ppeell < 110,- 0,
Apply the bound to each deviation:
a(1 -
Spre,
Amax 1|0 Oproll Ei,
2
825 < a(1 - a))
Amaa 110 0, 2 E2,
2
for small remainder terms E1, Eg of order O(ll -13).
19

---

Because |10; - Pprell ll < ||0; Byll, 1, it follows that:
Opro. ويg up to cubic-order error.
Therefore, the CTL deviation along the interpolation path from Ppro to 02 strictly smaller than that between 0 and fg in the small-distance regime.
L
Fig- Billustrates the empirical validation of Theorem 3.1 which states that the CTL deviation between two parameter vectors is upper- bounded by a quadratic function of their Euclidean distance. CTL deviations are computed using synthetic quadratic loss function as surrogate for the true task loss. In particular, the deviation tends to be smaller when interpolating between pretrained model and fine-tuned model pfe <+> 0.) than between two independently fine-tuned models (0, ++ fy).
C.2 Existence of Mapping Function
The mapping function Pedit effectively exists in practical editing scenarios, such as small weight modifications within overparameterized encoder. We demonstrate that KL-divergence brrggence-bbeedd mapping also performs effectively (Tab. (9).. Furthermore, experiments across various learning rate configurations (Tab. [8] show that the mapping is robust to different optimization setups.
Theorem 3.2 (Existence of a Mapping). Let Pedit R IR& be any mapping that sends latent editing vectors to weight perturbations applied in the encoder's final layer small subset of layers. Under the assumption that these edits are sufficiently small and confined to that small subset of layers of an overparameterized encoder (e.g.. ViT) with de > there exist infinitely many distinct Oodit.
Proof. For any input v, the editable-paramerre perturbation w sufficiently small, then
f(r;0 0 + w) fxx;0) + (0) + o([]w]],
80 a first-order Taylor approximation around 0 is valid. Let J == Jr(0) E RRmxxd denote the Jacobian of the encoder output with respect the editable parameters, evaluated 0. Since edits restricted to an overparameterised subset of layers with de > m, we assume rank(J) m (full row rank).
We seek perturbation w E Rd that realises a target change & E RT the encoder output, i.c.
Jw
Because J has full row rank, the matrix JJJ Rmxm invertible, and the Peorose pseudoinverse
;t J (JJ") Bd
satisfies JJJ = Im- Hence particular solution of (A) is
wo(=) Jtz.
Let N := ker = {o E dR Jo = =0. Its dimension is de - m > 0, so the full solution set of (A) is the affine subspace
S(z) = J z ker J = { 2 + + U kor J}. Since A is nontrivial, infinitely many distinet w satisfy Jw Now take any linear map N e RR xm whose columns lie ker (JN 0), and define
R:= jt N.
Then JR - JJ Im so for every 2 E Rmm
J(Rz) z.
The mapping
dedit (z) Rz
thus provides valid weight-space perturbation realising the desired edit. Varying N (or equivalently adding any vector in ker / to R2) yields infinitely many distinct mappings Pedit Im -> Rd that
all satisfy J dedit (2)
Finally, since ker J is a linear subspace, for any 5 > 0 we may rescale target z (or equivalently R) to ensure []oedit < E, so the perturbation remains sufficiently small while achieving the desired first-order effect. Therefore, under the stated conditions, there exist infinitely many sufficiently small weight-space mappings dedit that satisfy J dedit (z) for all E m
D
20

---

C.3 Class Vectors Preserve Inter-Class Orthogonality
The independence of Class Vectors from other classes is a key property for effective localized editing. Based on Neural Collapse (NC) [55,, we provide theoretical evidence supporting this claim.
Theorem 3.3 (Independence of Class Vectors). Suppose (i) the pretrained class embeddings col- lapse common mean gpre, that is z gpre; (ii) after fine-tuning the embeddings follow centre-shifted ETF form z μs uh with Ee Ue 0; and (ii)) the global-shift l|I g is negligible compared to the class-specifcc update || [ell-- Then, for any two distinct classes c. # d,
cos(Ner>
O,
Lee the Class Vector Re is appraximately orthogonal to the fine-tuned embedding of every other class.
Proof. Define the editing vector Ne = =, With Assumption (i) we write
With
2pre = gure + Ce:
where ||eell < ||uell-
Hence
Re (1s + ue) (apre + ec) = (1 =pte) + uc Cc.
For distinct class d' + the inner product becomes
(Ke. 2 (1= grre, p + ue + (Wey ue) (Eer f + ue)).
Assumption (ii) implies Ee the = 0 and that {ue} form an equiangular tight frame, so (Me, Mer) Mell and (ns uey) = 0 after choosing orthogonal to the span of the Assumption (iii) states k-1
||-- _Pre|] << ||uell (μ gere f + wee ) negligible compared with ||ue]]? Finally, |/eell < ||uel] makes the last term negligible. Collecting these bounds,
(Ke:
ft Zel
k
1
which is small when the number of classes is moderate to large. Since ||Hie|| > ||uel| and || R v Mull
Mer || O(lleel): the cosine similarity satisfies
(Kec 2
cos(Ret Ker =
0,
|
establishing that each Class Vector K2 is approximately orthogonal to every other fine-tuned embed- ding =0 for c'.
We note that the Neural Collapse (NC) phenomenon emerges across diverse classifier architectures, data distributions, and training paradigms (55 42 75 Here, we validate the underlying assumptions on VIT-B/32. In Fig. 9 observe that the cosine similarity between pretrained class centroids, gpre nearly 1. This indicates that the pretrained class embeddings collapse to a common mean, thereby validating Assumption (i). Furthermore, shown in Tab, 6. the cosine similarities among class centroids closely match the theoretical value of -1/(k: 1). strongly supporting the hypothesis that the centroids form equiangular tight frame (ETF) strueture (Assumption (ii)). Furthermore, we show that the global shift in the mean representation across all classes is significantly smaller than class-wise updates, indicating that the editing process remains highly localized (Assumption (iii)). Fig. further shows that the fine-tuned class representations are quasi-orthogonal, providing additional evidence for inter-class independence.
D Experimental Details
D.1 Class Configurations
Tab. 7 shows the class configurations used to evaluate class-vector properties. For each task, we consistently select the first two classes.
21

---

Table Verification of Assumptions and 3 across five tasks. Cosine similarity is computed across re- ce-eeteeed class embeddings (Assumption 2). The last column reports the deviation of global drift relative to class-specific update norms (Assumption 3).
Task
Cos.Sim. (Ass. 2)
Theoretical ETF Cos.Sim.
Global-Shift /Cls-update (Ass. 3)
DTD
0.02
0..2
0.00
EuroSAT
-0.10
-0.11
-0.12
GTSRB
-0.02
-0.02
-0.02
MNIST
-0.10
-0.11
-0.02
RESISC45
0..2
0.02
0.07
Pretrained Fine-tuned
Abs. Avg. Similarity
0.8
0.4
0.0
MNIST
EuroSAT
GTSRB
RESISC45
DTD
Figure 9: Comparison of cosine similarities across class representations within the pretrained and fine-tuned classifiers.
Table 7: Task overviews and target classes for 3.3
Task
# classes
Target classes
MNIST
10
0,1
EuroS AT
10
Annual crop. Herbaceous Vegetation 20kph speed limit, 30kph speed limit I Airplane, Airport Banded, Blotchy
GTSRB
43
RESISC45
45
DTD
D.2 Task Details
The fundamental properties linearity and independence, and applications of Class Vectors in the context of unlearning are evaluated in $3.3 and $4.2 using six widely adopted image classification tasks, MNIST [11, EuroSAT (24), SVHN [51. GTSRB (63, RESISC45 [ and DTD [I]. Addi- tionally, we empirically justify the Class Vectors in both MLP and ResNet- 18 architectures, using CIFAR-10 and CIFAR-100 datasets. All images are rescaled to image size 224 224. The details for each task are follows:
MNIST 41: A handwritten digit classification task with 60,000 training images and 10,000 test images, categorized into 10 classes from 0 to 9.
EuroSAT 24 : Land use and cover satellite image classification task, containing 13 spectral bands and 10 classes, with 16,000 training images and 5000 test images, 27, 27,000 images in total with validation set.
SVHN 51: A real-workee digit classification benchmark task containing total of 630,000 images with 2,700 test data of house number plates.
GTSRB (33 Traffic sign classification task with categories, containing 39,209 training data and 12,630 test data under varied lighting and complex backgrounds.
22

---

A benchmark for remote sensing scene classification task with 31,500 images across 45 distinct scene types.
RESISC45 19:
DTD I1
Image texture classification task with 47 categories, containing collection of 5,640 texture images, sourced from diverse real-world settings.
CIFARIO 38 The dataset is 10-class classification task consisting of 60,000 images, with 6,000 images per class. contains 50,000 training images and 10,000 test images.
CIFAR100 38 : The dataset is subset of the Tiny Images dataset and consists of 60,000
color images. The 100 classes are organized into 20 superclasses, with each class containing 600 images.
For evaluating Class Vectors in adapting to snowy environments ($4.3) and defending against typography attacks ($4.4) we use snowy ImageNet and clean and text-attached object images from previous study [60]. For the snowy environment adaptation scenario, we collect clean images for all classes from ImageNet, as shown in Fig. T0 For the defense against typography attacks, we augment images from all classes with: 1) digital text attachment, 2) rotation, random cropping, and color jittering (Fig. 1T.. The details for each task are as follows:
Snowy ImageNet: A collection of images from ImageNet with snowy environments, consists of 7 classes with 20 images per class, totaling 140 images.
Images for typography aftack: It consists of web-scraped images with classes of indoor objects and text-attached images for each. There are 6 images of each object. We augment each sample into 13 images per class, thus making total of 90 images, including both original clean and adversarial images.
D.3 Training Details
D.3.1
Class Vector Justification
In Fig. 2b we justify Class Vectors using three architectures--ii-BRB an MLP, and ResNet- 18 and train each as follows. The Vision Transformer (ViT-B/32) is fine-tuned per task for ap- proximately 22 epochs with learning rate of le-5 and a batch size of 128. For both the MLP and ResNet- we train MNIST and CIFAR-10 with learning rate of le-3, batch size 128 for 10 epochs, and on CIFAR-100 100 with a learning rate of le-4, batch size 512 for 300 epochs.
D.3.2 Mapping Editing Vectors
We utilize fully fine-tuned weights for the five tasks mentioned above from open-source repositories of Task Arithmetic 29 We use pretrained ViTs to classify ImageNet classes in $4.3 and $4.4 as they are trained on ImageNet. For these tasks, we initialize the encoder and create the representation for all classes from the model to generate Class Vectors. All our experiments first design Sedit then map to the weight space (i.e., Oedi) achieve classifier editing. The following are detailed training settings for mapping editing vectors. Refer to Algorithm 3 and Algorithm| 4 for details on the reference sample. All our experiments are conducted on a single NVIDIA A100 GPU.
Exploring properties of Class Vectors 3 exxloors the fundamental properties of Class Vectors: linearity and independence. To evaluate the linearity of Class Vectors, we train Zedit for 15 epochs, using target classes within 1% of the test set, with a learning rate of 1.5e-2 and a single reference sample. For all experiments verifying independence, set, as described above, with a learning rate of 5e-2 and one reference sample.
train the model for 15 epochs on the 1% test
Class unlearning Tab. shows the hyperparameters for class unlearning. We find the best hyper- parameter by grid searching [3e-2, 4e-2, 5e-2, 6e-2]. The same hyperparameters with MNIST are applied to SVHN in the cross-task class unlearning scenario. The following are the hyperparameters we adopt to evaluate class unlearning:
23

---

Adapting to new environment and defense against typography attacks In Tab. 11 hyperparameters for training models in and $4.5 Note that we post-train the MLPs and layer normalization in final encoder layer.
we present
Adversarial trigger optimization We present hyperparameters for optimizing adversarial triggers (small patches and invisible noise) in T2 and T3b Transparency-o in Tab denotes the process blurring the noise Wrrigger to make triggered image stealthy. Specifically, we attack the model with triggered image, Tataak (1 a) FFrriiinal + ar -Frigger. Additionally, for training BadNet (20]. we post-train the fine-tuned classifier to misclassify trigger-attached images using cross-entropy loss, with learning rate of le-4, while keeping other settings unchanged.
Table Hyperparameters for class unlearning with ViT-B/32.
Hyperparameters
MNIST
EuroSAT
GTSRB
RESISC45
DTD
Epochs
15
Sample size (%)
1
5
10
Learning
4c-2
5e-2
4e-2
3e-2
5e-2
Scaling coefficient
-1.5
Reference sample (image)
Table 9: Hyperparameters for class unlearning with ViT-B/16.
Hyperparameters
MNIST
EuroSAT
GTSRB
RESISC45
DTD
Epochs
15
Sample size (%)
1
5
10
Learning rate
4e-2
5e-2
6e-2
3e-2
6e-2
Scaling coefficient
-1.5
Reference sample (image)
Table 10: Hyperparameters for class unlearning with ViT-L/14.
Hyperparameters
MNIST
EuroSAT
GTSRB
RESISC45
DTD
Epochs
Sample size (%)
1
5
10
Learning rate
4e-2
3e-2
6e-2
3e-2
6e-2
Scaling coeflicient
-1.5
Reference sample (image)
Table 11: Hyperparameters for adapting to new environment and defense against typography attacks.
Hyperparameters
Snowy
Snowy (2)
Typography attack
VIT-B/16
ViT-B/32
VIT-L/14
Epachs
Sample size (images)
Learning rate
le.4
Scaling coefficient
Reference sample (image)
Table 12: Hyperparameters for optimizing small patches across classifiers.
Hyperparameters
SVHN
GTSRB
RESISC45
VIT-B/16
VIT-B/16
ViT-B/32
VIT-L/14
VIT-R/16
VIT-B/32
VIT-L/14
Epochs
100
Sample size (%)
Learning rate
5
5
10
Scaling
Reference sample
Patch Size
20
24

---

Table 13: Hyperparameters for optimizing invisible noise across classifiers.
Hyperparameters
SVHN
GTSRB
RESISC45
VIT-B/16
VIT-B/32
VIT-L/14
VIT-B/16
VIT-B/32
VIT-L/14
VIT-R/16
VIT-B/32
VIT-/114
Epochs
Sample size (%) Learning rate
10
300
200
600
Scaling coefficient
1.0
Reference sample
Patch Size
224 224
Transpareccy-a a
2e-4
E
Additional Experiments
In this section, we present additional experiments on ViT-B/32 and ViT-L/14 for class unlearning, as well as results for backdoor attacks using adversarial triggers on the ViT-B/16 and ViT-L/14. As shown in Tab and Tab Class Vectors consistently outperform baselines. Meanwhile, random vectors highlight that editing with Class Vectors provides an intuitive way to remove model adaptations. Additionally, gradient ascent (i.e., NegGrad) still struggles to maintain accuracy, while retraining remains ineffective due to data insufficiency. Tab. T6 also demonstrates Class Vector's stable performance in backdoor attack scenario, effectively surasssing other baselines across two different types of adversarial triggers.
Table 14: Comparison of class unlearning with baselines on ViT-B/32.
Method
MNIST
FaroSAT
GTSRB
RESISCAS
DTD
ACC, (4)
ACC, (1) ACCy (1) ACC, (1) ACC, (U) ACC, (t) ACC, (4)
ACC, (1) ACC/ ())
ACC, (t)
Pretrained
54.61:33
47.3833
57.842144
44.54-24
43.34:0.11
32.2416
71:2444
60.1
35:3332
44.64077
Fine tuned
99.8+
99.75000
99.441
99.8:00
99.5
98.7200
98.9=0.1
96.1
70.5
79.6404
Retrained
0.0:0:00
63.7#:2.2
0.0:100
79.64:15
1.242.4
48.6:10.7
27.7=:2.5
71.8405
17.04316
$4.6106
NegGrad
0.0+00.0
33.7
0.0+000
16.696.11
0.0=00.0
31.74:27.6
0.0+:00
6.1476
0.0+00
9.0+117
Random Voctor
99.8.1:.11
99.61.0.0
99.9110.1
99.41:022
99.5
97.21155
97.9=35
43.31:25.1
63.513306
42.41205
Class Vector
0.0000
99.6220.1
0.0:0.00
92.02888
0.0:0..0
94.62559
7.1
65.42:19
15.0
66.32112
Table 15: Comparison of class unlearning with baselines ViT-L/14.
MNIST
EuroSAT
GTSRB
RESISC45
Method
ACC, (1) ACC, (?)
ACC, (4) ACC, (?)
ACC, (1) ACC, (1) ACC, (1) ACC, (1)
ACC; (1) ACC, (t)
Pretrained
86.72881
75.2109
58.12287
63.01:66
85,42142
49.0
68.51188
714105
35.01335
558-07
Finc-tuned
99.8::0.0
99.71:00
99.9:022
99.71:00
99.62007
99.21:00
99.61.077
97.3100
76.5111777
843:23
Retrinedd
42.8::37.0
88.8110
8.71145
90.11113
59.6=310
64.2211
36.11267
80.2104
1651130
62.729
NegGrad
0.0::000
13.7
0.0400
11.2410
0,0:00
21.22199
0.00000
10.5+:22
0.0=0.0
9.7-100
Random Vector
99.8.00.1
99.74:00
100.0:000
94.2189
99.5.007
99.1
99.51.07
94.81330
59.0:146.
53.74238
Class Vector'
3.7
97,943.
0.04000
92.748.7
9.7
94.825.0
0.0:000
91.2433
23.543349
70.5
Table 16: Results backdoor attacks with optimized triggers for ViT models.
VIT-B/16
VIT-L/14
Method
Small Patch
Invisible
Small Patch
Invisible
ASR (t)
CA (1)
ASR (1)
CA
ASR (f)
CA
ASR (1)
CA
Pretrained
32.7=22.0
53.9499.
28.5+-22.7
$3.9-19.5
14.9+2204
60.148866
22.24-219
60.14886
Finctuned
0.3:1:022
98.0:10.9
0.3100
98.0:1099
0.1:10.11
98.2:1088
0.1::0.1
98.2:008
BadNet
90,9=12.8
29.641:3.9
91.32123
27.413009
70.22-41.1
10.0.5..0
1001:0
9.3274
Random Vector
0.1:::011
98.0:109
0.2:03
08.0:10.9
0.0:10.0
98.2:1088
1.2.1..7
08.2:008
DirMatch
99.8=27.9
98.0-409
97.1422.3
98.0-1099
69.3
98.244088
95,3-1-500
98.240.8
Class Vector'
100::0.0
98.0.10.9
97,542.5
98.0.20.
100::00
98.2:10.8
97.5
98.22208
25

---

Table 17: Task Vector vs. Class Vector in class unlearning. Each cell reports (ACC, 4. ACC,)
MNIST
EuroSAT
GTSRB
Model
Task Vector
Class Vector
Task Vector
Class Vector
Task Vector
Class Vector
VIT-B/32
20.0,
0.0, 99.6
20.0, 10.1
0.0, 92.0
20.0, 0.3
0.0, 94.6
VIT-B/16
20.0, 8.8
0.0, 99.7
20.0, 9.7
0.0, 99.5
20.0, 0.5
0.0, 98.6
VIT-L/14
1.1
3.7,
0.0.
020.
9.7,94.8
F Mapping Sensitivities
Table 18: Impact of learning rate (LR) on Class Vector editing.
LR
MNIST
EuroSAT ACC,
GTSRB
ACC2
ACC,
ACC,
ACC,
ACC'
20-5
0.0
99.7
0.0
99.7
0.0
98.3
5e-B
0.0
99.7
0.0
99.8
0.0
98.6
0.0
0.0
0.0
2e-4
0.0
99.4
0.0
99.7
0.0
96.2
Table 19: Performance of Class Vector mapping with KLD loss across datasets.
MNIST
EuroSAT
GTSRB
RESISC4S
DTD
ACC,
ACC,
ACC,
ACC,
ACC,
ACC,
ACC, J
ACC,
ACCy +
ACC,
KLD
0.0
99.7
0.0
0.0
98.7
0.0
94.5
15.0
79.3
MSE
0.0
96.2
0.0
99.7
0.0
93.4
10.0
90.7
15.2
72.9
Sensitivity to learning rate. Tab. 18 examines the effect of learning rate class-vector editing across MNIST, EuroSAT, and GTSRB. We report Forget Accuracy (L) and Retain Accuracy (f) for each setting. At low moderate rates above 94%, peaking at nearly 99.8%. However, at 5e-4, retention on GTSRB plummets to 60.3%, indicating that excessively large learning rate destabilizes the edit.
forgetting remains at 0% while retention stays
Impact of loss function. Theorem 3.2 demonstrates that mapping solutions can be obtained from infinitely many distinct configurations. Tab. 19 compares KLD vs. MSE (default) as the mapping loss across MNIST, AT, GTSRB, RESISC45, and DTD. Even with KLD loss, forgetting remains perfect (0%) and retention exceeds 94.5% on all but the most challenging texture dataset (DTD, 79.3%).
G Computational Analysis
Table 20: Analysis on computational complexity of classifier editing with Class Vectors.
Application
Task
# of parameters
Time (s)
Unlearning
MNIST
L.5K
2.5
Adapting new environment
4.7M
1.2
Defending against typography attacks
4.7M
10.4
Small patch trigger optimization Invisible noise trigger optimization
RESISC45
0.4K
238
RESISC45
1.5M
38.1
We evaluate the computational complexity of classifier editing using Class Vectors mapping by measuring the number of trainable parameters and the time required for model editing with ViT-B/32. As shown Tab. 20 Class Vectors enable efficient classifier editing across diverse applications, requiring minimum of 1.5K trainable parameters just 1.2 seconds. This aligns with the philosophy of model intervention, which aims to adjust models quickly using small number of samples while achieving effective editing performance.
26

---

H Supplementary Figures
Figure 10: (Top) Snowy objects for adaptation to a snowy environment. (Middle) and (Bottom) Web-crawled clean object images for each class to isolate snow representation in to eliminate snow representation in the second scenario.
FOO
iPod
Figure 11: Augmented images across classes for typography attacks.
Ciaan
10
5
JO
2
Small Patch
5
10
10
2
Invisible
2
Figure 12: Trigger- images in RESISC45 and SVHN.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have included paper's contributions and scope in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have included the limitations the conclusion part and in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
28

---

Justification:
For each theorem, we provide its proof and an empirical validation of its assumptions in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Detailed information for reproducing the experimental results provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
29

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We are going to release our code either during the review period or shortly thereafter.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Coaee//dddeSebmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide experimental details in Appendix
D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Throughout our experiments, we consistently report error bars representing standard deviations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
30

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the resources information in Appendix $D.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have adhered to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss this in Appendix A.3
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
31

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We discuss this in Appendix A.3
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We follow the license and usage guidelines provided by creators of the assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
32

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide the license, limitations and training details in Appendix.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
33

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
34


Reviews:
[{"review_id": "jruMir6pKI", "reviewer": "Reviewer_KFN6", "summary": "This paper proposes class vectors for editing image classifiers efficiently. A class vector represents how a specific class's latent features change during fine-tuning. Unlike task vectors, class vectors allow per-class fine grained edits. This paper also found that class vectors follow linear trajectories in latent space (Cross-Task Linearity). Besides the insights above, this paper also explores the application of class vectors on several application domains like class unlearning, environmental adaptation, adversarial defense and etc. And the method is fast, flexible, and works well with very few samples. It shows strong results across several benchmarks like MNIST and ImageNet.", "questions": "Please refer to weakness part for the questions:\n\n1. Could you please add more baselines for each applications?\n\n2. Any experiments to show the benefits for using class vector instead of task vector?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is logically organized and clearly written, making it easy to follow the proposed ideas and methodology.\n\n2. Theoretical analysis is well-grounded, and the empirical results are convincing, demonstrating the effectiveness of class vectors across multiple settings.\n\n3. The experimental validation is extensive, covering a wide range of applications such as class unlearning, environmental adaptation, typographic attack defense, and backdoor trigger optimization.\n\nWeakness:\n\n1. The machine unlearning experiments lack strong baselines. Although simple comparisons are acceptable for exploring feasibility, including more competitive or state-of-the-art baselines like salun [1] would provide a better benchmark for evaluating performance.\n\n> [1] Fan, Chongyu, et al. \"Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation.\" arXiv preprint arXiv:231\n\n2. The paper is inspired by task vectors but does not include a direct comparison between task vectors and class vectors. Since both approaches aim to edit model behavior, especially in applications like unlearning, it would be helpful to include empirical comparisons or theoretical discussion to clarify their respective advantages and limitations. At least, it is better to make it clear for the comparison between two methods\n\n3. Minor issue: the Cross-Task Linearity equation on page 3 appears to contain an error and should be double-checked for correctness.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns; however, I believe one issue still requires attention: the application of the class vector for unlearning is only suitable for class-wise unlearning, which I feel limits the scope. In addition, the innovation from task vector to class vector does not seem particularly significant. Therefore, I am keeping my original score as borderline accept.", "rebuttal": "Dear Reviewer KFN6, we appreciate your valuable questions and comments. Below, we address the main concerns.\n\n### **(1) Comparing class vector to SOTA baselines in unlearning**\n\nThank you for raising this concern. In response to your review, we conducted additional experiments comparing our class vector approach with several recent state-of-the-art machine unlearning methods [1, 2, 3].\n\nTo robustly evaluate the effectiveness of forgetting a target class, we adopted the CIFAR-100 dataset, following the original setup, and applied all methods to ViT-B-16. We chose CIFAR-100 over smaller-class datasets (e.g., MNIST), as models tend to perform well across all methods when the number of classes is small, making it difficult to distinguish their forgetting performance. Our evaluation includes two settings: 1) class unlearning with 10 target samples and 2) class unlearning with 100 target samples. The results are summarized in the table below. We report both forgetting and retaining accuracies as a pair.\n\n|  | 10 Samples | 100 Samples |\n| --- | --- | --- |\n| Fine-tuned | 87.0, 76.5 | 87.0, 76.5 |\n| SALUN [1] | 9.0, 56.1 | 12.0, 66.8 |\n| SSD [2] | 0.0, 55.2 | 0.0, 70.1 |\n| SVD [3] | 17.0, 73.4 | 4.3, 74.8 |\n| **Class Vector+** | **0.0, 69.3** | **0.0, 69.3** |\n\nAs shown in the table, class vector outperforms SOTA baselines in the 10-sample setting, achieving complete forgetting while retaining strong performance (69.3%). This highlights its effectiveness in limited-data scenarios, where other methods such as SALUN [1], SSD [2], and SVD [3] exhibit either suboptimal forgetting or lower retention, underscoring the precision of class-level editing with class vectors in data-sparse regimes. In the 100-sample setting, class vector still performs well, though SSD and SVD surpass it on some metrics. This suggests that while certain baselines scale better with more data, as they claim, class vector remains an efficient and practical choice when compute or time is constrained.\n\nOverall, these results offer important takeaways: there is a trade-off between methods depending on the number of target samples. Class vector is especially suitable for low-resource or rapid-editing scenarios, making it a complementary and effective tool alongside other approaches depending on deployment needs.\n\nWe will add the corresponding experiments and additional comparisons with state-of-the-art adversarial attacks (Section 4.5) for completeness.\n\n### **(2) Comparison of class vector with task vector**\n\nThank you for highlighting this valuable point. As you noted, our method is inspired by task vectors, and our theoretical foundation is derived from the same principles (CTL, Theorem 3.1). However, as described in Section 2, task vectors modify model behavior at a global, task-wide level, whereas class vectors enable fine-grained, class-specific editing in both latent and weight space. This distinction allows for more precise control over class-specific behavior, in contrast to task vectors that operate at the broader task level.\n\nDuring the rebuttal period, we further evaluated task vectors as a baseline in the class unlearning setting. The relevant results are provided in our response to **Reviewer QfPd, item (2)**. As expected, task vectors led to significantly lower retaining accuracy, as they tend to erase predictive rules across all classes rather than selectively targeting a specific one. These results underscore the advantage of our method in scenarios requiring localized, class-level editing.  \n\nWhile class vectors excel at precise control, they may be less effective when editing entire classes simultaneously, as they are inherently designed for class-specific modifications. Therefore, when broader behavioral changes across entire classes are required, we recommend using task vectors instead.\n\nIn the revised version, we will include these results in the Appendix, clarify this distinction in the related work section, and strengthen the discussion of the behavioral differences between task vectors and class vectors in weight space, following Theorem 3.1.\n\n### **(3) Mistakes in CTL equation**\n\nThank you for pointing out the mistake in the CTL equations on page 3. We had inadvertently reversed the direction of \\alpha in the interpolation. The correct version is provided below.\n\n$f(x \\, \\alpha \\theta_i + (1 - \\alpha)\\theta_j) \\approx \\alpha f(x \\, \\theta_i) + (1 - \\alpha) f(x \\, \\theta_j)$\n\nWe will make the necessary correction in the revised version of the paper to reflect this.\n\n---\n\n[1] Fan et al. \"Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation\", ICLR 2024\n\n[2] Foster et al. “Fast Machine Unlearning Without Retraining Through Selective Synaptic Dampening”, AAAI 2024\n\n[3] Kodge et al. “Deep Unlearning: Fast and Efficient Gradient-free Class Forgetting”, TMLR 2024"}, {"review_id": "ATqKm0Su29", "reviewer": "Reviewer_QfPd", "summary": "The paper introduces class vectors, latent difference vectors between a class’s pretrained and fine-tuned centroids, and shows they enable fast, targeted edits to vision encoders via latent steering or a tiny weight-mapping adapter. Experiments on multiple ViT-CLIP models demonstrate consistent gains over retraining and recent editing baselines, highlighting CVs’ practicality but also revealing limitations in generality and baseline coverage.", "questions": "1. Have you tried CV editing on a traditional CNN or a BERT-style text encoder?\n2. The current study modifies one class at a time. What happens when several mutually interacting classes (e.g., “cat”, “lion”, “tiger”) are edited together?", "limitations": "See weaknesses", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The motivation of the paper is natural, which moves task-level vectors to class vectors to capture each class’s semantic shift.\n2. The proposed method is theoretically sound, with connecting class vectors to Cross-task linearity and proofs, and empirical experiments in appendix.\n3. The ablation studies and analyses are thorough and insightful.\nWeaknesses\n1. Building on prior work like Task Vectors, In-Context Vectors, and CTL, the authors should clarify how Class Vectors differ from concept activation and representation surgery methods.\n2. Authors should conduct experiments to compare the task-vectors with class-vectors.\n3. Typo in algorithm title, “classfier”.", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer QfPd, thank you for your helpful questions and comments. We respond to the main concerns as follows.\n\n### **(1) Differences from Prior Works**\n\nWe appreciate the feedback and would like to provide clarification. Below, we outline the specific differences between Concept Activation Vectors [1, 2] and Representation Surgery [3, 4].\n\n> **vs. Concept Activation Vector (CAV)**\n\nClass vector and the Concept Activation Vector (CAV) are fundamentally different, as our vector is a prescriptive tool for model editing, whereas a CAV is a descriptive tool for model interpretability. Specifically, class vector is computed directly as the centroid of an output class’s activations to define a precise direction for model editing. In contrast, a CAV is derived by training an auxiliary linear classifier to identify the direction of an arbitrary, human-defined concept (e.g., “stripes”) that need not correspond to any of the model's output classes. Therefore, our method provides a vector to *modify* a model's knowledge of its own classes, while CAVs provide a vector to *quantify* a model's reliance on abstract concepts.\n\n> **vs. Representation Surgery**\n\nOur method is also distinct from Representation Surgery, primarily in application and mechanism. Class vector is a static vector used for single-model editing, applied during inference to steer representations of a specific class. Conversely, Representation Surgery is a trainable, lightweight module designed to solve a multi-model merging problem. It is trained post-hoc to correct the 'representation bias' that occurs when combining weights from several specialist models. Thus, our approach is an intervention to selectively modify predictive rules from one model, while Representation Surgery is a repair mechanism to faithfully preserve and integrate information when combining multiple models.\n\n### **(2) Comparison of class vector with task vector**\n\nWe sincerely thank you for raising this comment. As noted in Section 2, task vectors modify the model’s behavior at a global, task-wide level, which can lead to suboptimal performance on non-target classes. \n\nTo empirically demonstrate this, we conducted additional experiments in the class unlearning setting. The resulting pairs of forgetting and retaining accuracies show that task vectors tend to yield significantly lower retaining accuracy, indicating that they are less suitable for class-specific editing. In contrast, class vectors allow for effective and disentangled correction of the model’s behavior specific to the target class, without interfering with other classes. The corresponding results are presented below.\n\n| Model | Dataset | Task Vector | Class Vector |\n| --- | --- | --- | --- |\n| **ViT-B-32** | MNIST | 20.0, 8.8 | 0.0, 99.6 |\n|  | EuroSAT | 20.0, 10.1 | 0.0, 92.0 |\n|  | GTSRB | 20.0, 0.3 | 0.0, 94.6 |\n| **ViT-B-16** | MNIST | 20.0, 8.8 | 0.0, 99.7 |\n|  | EuroSAT | 20.0, 9.7 | 0.0, 99.5 |\n|  | GTSRB | 20.0, 0.5 | 0.0, 98.6 |\n| **ViT-L-14** | MNIST | 0.0, 1.1 | 3.7, 97.9 |\n|  | EuroSAT | 0.0, 10.7 | 0.0, 92.7 |\n|  | GTSRB | 0.0, 1.0 | 9.7, 94.8 |\n\nIn the revised version, we will include these results in the Appendix, along with additional experiments on adversarial trigger optimization, which were omitted due to time constraints.\n\n### **(3) Additional experiment on CNNs and BERT encoder**\n\nThank you for giving us the opportunity to demonstrate the architecture-agnostic nature of class vectors. As supported by our theoretical analysis, class vectors can be applied regardless of architecture, provided that both a pretrained and fine-tuned model are available (Theorem 3.1), and that class-wise representations converge to a simplex structure (Theorem 3.3).\n\nFollowing your suggestion, we conducted additional class unlearning experiments on ResNet18, ResNet50 [5], and ConvNeXT-Tiny [6] for image classification, as well as a BERT-Base encoder [7] for text classification. All methods, except for Class Vector+ (latent steering), were trained for 3 epochs using 1% of the validation set. **T**he results for forgetting and retaining accuracies are summarized below.\n\n> **ResNet18**\n\n| Dataset | **Retrained** | **NegGrad** | **Random Vector** | **Class Vector** | **Class Vector+** |\n| --- | --- | --- | --- | --- | --- |\n| MNIST | 99.8, 99.5 | 14.2, 97.2 | 99.7, 99.4 | 2.2, 97.0 | 0.0, 99.4 |\n| EuroSAT | 89.3, 97.9 | 19.5, 67.0 | 98.3, 97.9 | 19.8, 72.3 | 0.1, 97.9 |\n| GTSRB | 99.9, 99.9 | 22.4, 91.8 | 99.9, 99.9 | 3.7, 96.1 | 0.0, 99.8 |\n\n> **ResNet50**\n\n| Dataset | **Retrained** | **NegGrad** | **Random Vector** | **Class Vector** | **Class Vector+** |\n| --- | --- | --- | --- | --- | --- |\n| MNIST | 89.9, 99.6 | 1.0, 95.0 | 99.5, 99.4 | 11.5, 84.2 | 0.0, 99.1 |\n| EuroSAT | 79.8, 99.5 | 15.8, 67.9 | 99.7, 99.5 | 10.4, 86.6 | 0.2, 98.8 |\n| GTSRB | 69.7, 99.9 | 3.7, 91.3 | 99.4, 99.9 | 8.2, 98.2 | 0.0, 99.9 |\n\n> **ConvNeXT-Tiny**\n\n| Dataset | **Retrained** | **NegGrad** | **Random Vector** | **Class Vector** | **Class Vector+** |\n| --- | --- | --- | --- | --- | --- |\n| MNIST | 78.2, 99.4 | 0.0, 11.2 | 99.5, 99.3 | 0.0, 95.3 | 0.0, 99.1 |\n| EuroSAT | 77.3, 98.4 | 0.0, 10.6 | 99.4, 98.5 | 0.0, 78.1 | 0.0, 98.8 |\n| GTSRB | 100, 100 | 74.7, 99.9 | 100, 99.5 | 7.8, 96.8 | 0.0, 97.4 |\n\n> **BERT-Base**\n\n| Dataset | **Retrained** | **NegGrad** | **Random Vector** | **Class Vector** | **Class Vector+** |\n| --- | --- | --- | --- | --- | --- |\n| AG-NEWS | 71.9, 89.3 | 0.0, 48.9 | 93.8, 94.3 | 0.0, 93.2 | 3.2, 94.4 |\n| DBPedia-14 | 98.4, 99.0 | 0.0, 93.8 | 98.6, 99.1 | 0.0, 96.9 | 0.0, 99.1 |\n| 20-Newsgroups | 59.8, 66.5 | 0.0, 47.0 | 62.9, 67.9 | 0.0, 57.9 | 0.0, 63.8 |\n\nOverall, the class vector methods outperform baseline approaches and demonstrate effective editing across architectures, highlighting both the effectiveness and robustness of class vectors across a variety of model architectures.\n\nWe will include all results in the Appendix in the revised version.\n\n### **(4) Further experiment on multi-class editing**\n\nThank you for the insightful comment. As you pointed out, although our method edits one class at a time, real-world scenarios may require editing multiple classes simultaneously. In response, we conducted an additional experiment on MNIST involving mutually interacting classes.\n\nWe selected classes 0, 2, and 6 as target classes, as they are known to lie on a shared learned manifold and exhibit linear interpolations between each other [8]. To jointly edit them, we constructed a composite editing vector as follows:\n\n$z_{\\text{edit}} = -1.5 * (\\lambda_0 \\kappa_0 + \\lambda_2 \\kappa_2 + \\lambda_6 \\kappa_6$)\n\nwhere \\lambda_i are weighting hyperparameters.\n\nBelow, we report four settings: 1) the average performance of single-class editing, 2) a naive average of class vectors with equal weights ($\\lambda_i$ = -1/3), 3) all $\\lambda_i$= -0.6 following Task Arithmetic [9], and 4) an optimal combination of $\\lambda$ values obtained via grid search.\n\n| Method | Accuracies |\n| --- | --- |\n| Single class (Averaged) | 0.0, 96.7 |\n| $\\lambda_{0,2,6} = 1/3$ | 99.6, 99.6 |\n| $\\lambda_{0,2,6} = 0.6$ | 5.7, 98.8 |\n| $\\lambda_i = 1.0, 0.6,0.8$ | 1.3, 99.2 |\n\nWe find that when editing multiple classes simultaneously, the choice of weighting hyperparameters becomes critical. Interestingly, while we originally used $\\alpha = -1.5$ for independent single-class editing, the optimal performance in the multi-class setting was achieved with lower effective $\\alpha$ values (i.e., $-1.5 \\times \\lambda_i$) as shown in Row 4. This mirrors observations in the model merging literature [10, 11], where interference between tasks can degrade performance. We hypothesize that similar interference may arise between class vectors during joint editing. \n\nInvestigating such inter-class interactions in multi-class editing remains an open direction for future research. The corresponding experiments will be included in the Appendix.\n\n### **(5) Typo**\n\nThank you for pointing this out. We will correct it accordingly in the revised manuscript.\n\n---\n\n[1] Kim et al. “Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)”, ICML 2018\n\n[2] Nicolson et al. “Explaining Explainability: Recommendations for Effective Use of Concept Activation Vectors”, TMLR 2025\n\n[3] Yang et al. “Representation Surgery for Multi-Task Model Merging”, ICML 2024\n\n[4] Yang et al. “SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgery”, Arxiv 2024\n\n[5] He et al. “Deep Residual Learning for Image Recognition”, CVPR 2016\n\n[6] Liu et al. “A ConvNet for the 2020s”, CVPR 2022\n\n[7] Devlin et al. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, ACL 2019\n\n[8] Kingma et al. “Auto-Encoding Variational Bayes”, ICLR 2014\n\n[9] Ilharco et al. “Editing Models with Task Arithmetic”, ICLR 2023\n\n[10] Yadav et al. “TIES-Merging: Resolving Interference When Merging Models”, NeurIPS 2023\n\n[11] Huang et al. “EMR-Merging: Tuning-Free High-Performance Model Merging”, NeurIPS 2025"}, {"review_id": "xxllRoASeX", "reviewer": "Reviewer_tBPC", "summary": "This work introduces a novel approach to model editing using class vectors, which manipulate hidden representations, as opposed to task vectors that modify model weights. The authors provide theoretical backing for their method, citing task-level weight linearity and CTL. They present two main techniques: training-free Latent-space injection and Weight-space mapping, the latter of which fine-tunes the model for better alignment with the identified class vectors. The effectiveness of these methods is demonstrated through experiments using three sizes of ViTs, showcasing the utility of class vectors in various applications, including class unlearning, environmental adaptation, defense against typographic attacks, and adversarial trigger optimization.", "questions": "1. Can the authors evaluate how task vectors compare to their method? \n\n2. Have the authors considered getting class vectors without using a finetuned model to make this method even more efficient? \n\n3. Can authors provide more points in Figure 6c as e.g., for 10, 5,0, and 100 examples to have a clearer picture of the scalability of the proposed method", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "#Strengths \n\nS1. The paper's primary strength lies in its strong theoretical justification for the proposed method. By grounding the methodology, the authors provide a compelling argument for why it should work. This goes beyond simple empirical success and provides confidence that the results are not just an artifact of the chosen datasets. \n\nS2. The method offers a highly practical and efficient framework for classifier modification. The most important part is performing those edits without additional training with latent-space injection and with weight-space mapping method that requires a small fine-tuning. \n \nS3. The paper demonstrates the effectiveness of class vectors across a diverse and practical set of classifier editing tasks. The applications, from unlearning and environmental adaptation to defending against typographic attacks, are well-motivated and showcase the flexibility of the proposed framework. The results, particularly for unlearning and adaptation, are strong, showing significant improvements over baselines with remarkable efficiency. \n\n#Weaknesses \n\nW1. The central weakness is the lack of a direct comparison to task vector arithmetic. Since the cost of fine-tuning is a prerequisite for both methods, the paper must empirically demonstrate why class vectors are superior for class-specific editing. The authors should include task vectors as a baseline in the class-unlearning experiments (e.g., Table 1) to validate their claim of improved granularity. Without this comparison, the novelty and practical advantage of the proposed method are not fully substantiated. \n\nW2. The methodology requires access to both a pretrained model and a corresponding fine-tuned version to compute the class vectors. This could limit its applicability in scenarios where only the final, fine-tuned model is available, which is common in the context of using models from open-source hubs. \n\nW3. The paper's analysis of how performance scales with sample size is limited. The only direct ablation study (Figure 6c) is confined to a very small data regime (1-5 samples) for a single task. While this demonstrates the method's strength in data-scarce scenarios, it fails to provide a clear picture of the trade-offs when more data is available. A more thorough analysis showing how the method compares to standard fine-tuning at larger sample sizes (e.g., 10, 50, 100) is needed to understand its scalability and when it might be preferable to a full retraining.", "was_revised": true, "final_justification": "During the rebuttal authors have resolved my initial questions. \nAlso, it seems like the authors have provided great number of experiments for questions of other reviewers.\n\nOverall, I think the paper should be accepted: it is well-motivated and gives good results on wide range of tasks. \nHowever, I am not able to judge how impactful this paper will be (whether it will get high impact on at least one sub-area of AI or moderate-to-high impact on more than one area of AI,", "rebuttal": "Dear Reviewer tBPC, we sincerely thank your for their constructive feedback. Below, we provide our responses to the primary points raised.\n\n### **(1) Comparison of class vector with task vector**\n\nThank you for highlighting this valuable point. As described in Section 2, task vectors modify model behavior at a global, task-wide level, whereas class vectors operate at a finer granularity, enabling targeted editing in both latent and weight space. This makes class vectors more suitable for scenarios requiring class-specific intervention, such as the applications presented in Section 4. In contrast, task vectors indiscriminately erase all predictive rules associated with a task, making them unsuitable when only a single class should be modified.\n\nDuring the rebuttal period, we further evaluated task vectors as a baseline for class unlearning. Please refer to our response to **Reviewer QfPd, item (2)**, for detailed results and experimental settings. As expected, task vectors showed significantly lower retention accuracy due to their global nature, which lacks selectivity. These results highlight the advantage of class vectors for localized, class-specific editing.\n\nWe will include these results in Table 1, and provide additional comparisons with task vectors for adversarial trigger optimization in Table 3.\n\n### **(2) Scenarios with only fine-tuned Models**\n\nThank you for raising this important concern. We would first like to clarify that in all ImageNet experiments (Sections 4.3 and 4.4), class vectors are computed as adaptation directions from the initialized representation to the pretrained representation, and this approach worked reliably.\n\nSimilarly, during the rebuttal period, we further conducted additional experiments based on the valuable scenario you proposed, where only a fine-tuned model is available. In this setting, instead of using a pretrained model, we compute the class vector using the fine-tuned model and an initialized model, which can be easily obtained via random initialization or alternative strategies [1]. For this experiment, we used randomly initialized weights for ViT-B-16. The resulting class unlearning performance is reported below in terms of forgetting and retaining accuracies.  We set $\\alpha =-1.0$ for all experiments.\n\n| Dataset | Class Vector+ | Class Vector+ (w/ random init.) |\n| --- | --- | --- |\n| MNIST | 0.0, 96.2 | 0.0, 95.4 |\n| EuroSAT | 0.0, 99.7 | 0.0, 99.4 |\n| GTSRB | 0.0, 93.4 | 0.0, 99.9 |\n\nAs a result, we find that class vectors computed using an initialized encoder perform well. We attribute this to the independence of fine-tuned representations (see Figure 8), as conditions (i) and (iii) of Theorem 3.3 suggest that class vectors derived from initialized encoders can still preserve sufficient independence. In the final version, we will include empirical validations of conditions (i) and (iii), along with analyses of their linearity, in the Appendix together with the above results.\n\n### **(3) Effect of sample size on baseline performances**\n\nThank you for pointing this out. To demonstrate the scalability of class vectors with respect to sample size, we conducted additional experiments comparing them with retraining-based methods on Snowy ImageNet and Typo Attack.\n\n> **Snowy ImageNet**\n     \n| # samples | Retrained | Class Vector | Class Vector+ |\n| --- | --- | --- | --- |\n| 1 | 16.3 | 65.2 | 63.6 |\n| 2 | 23.1 | 70.1 | 67.1 |\n| 5 | 62.8 | 72.1 | 76.2 |\n| 12 | 96.1 | 92.3 | 90.7 |\n\n> **Typo Attack**\n    \n| # samples | Retrained | Class Vector | Class Vector+ |\n| --- | --- | --- | --- |\n| 1 | 52.4 | 65.5 | 67.6 |\n| 4 | 59.5 | 66.7 | 70.1 |\n| 8 | 79.8 | 61.9 | 76.4 |\n\nAs a result, we find that retraining-based methods tend to perform better with more samples (e.g., 12 samples on Snowy ImageNet and 8 samples on Typo Attack). However, we note that these correspond to approximately 60% and 50% of the respective validation sets, an unrealistic assumption in real-world scenarios.\n\nDue to the limited sample sizes available in Snowy ImageNet and Typo Attack, we additionally conducted experiments on MNIST to evaluate baseline performance as a function of sample size. The MNIST test set contains 10,000 images, allowing for more granular analysis. The results, shown below, highlight that the class vector method consistently demonstrates effective model editing with an extremely small number of samples, reinforcing its practicality in low-resource settings, and in some cases, even outperforming training-based methods when data is sufficient.\n\n> **MNIST unlearning**\n    \n| # samples | Retrained | NegGrad | Class Vector | Class Vector+ |\n| --- | --- | --- | --- | --- |\n| 10 | 95.2, 53.4 | 82.2, 94.9 | 0.0, 99.6 | 14.1, 94.7 |\n| 50 | 39.6, 75.4 | 32.8, 93.2 | 0.0, 99.7 | 8.0, 96.7 |\n| 100 | 0.1, 76.4 | 0.0, 43.4 | 0.0, 99.7 | 0.0, 96.2 |\n| 250 | 4.6, 92.0 | 0.0, 23.8 | 0.0, 99.7 | 0.0, 96.8 |\n| 500 | 0.0, 96.0 | 0.0, 33.4 | 0.0, 99.7 | 0.0, 99.4 |\n\nBased on these results,, we will include the results in Section 4.6 and add a corresponding discussion in the revised version.\n\n### **(4) Scenarios without fine-tuned models**\n\nThank you for the question. It addresses a core aspect of how we define the proposed “class vector.” To be direct, obtaining a class vector without fine-tuning is conceptually impossible under our framework. A class vector is, by definition, a delta vector that captures how the representation of a specific class shifts in semantic space during fine-tuning. Without this shift, the delta simply does not exist, making the computation of a class vector infeasible. The key contribution of our work is to demonstrate that this delta vector locally encapsulates the knowledge of a class and enables efficient model editing via simple arithmetic operations. For this reason, our method fundamentally presupposes a fine-tuning step.\n\n### **(5) Paper formatting concerns**\n\nThank you for pointing out the legibility issues with our figures. In the revised version, we will increase the font size on all figures to ensure they are clear and easy to read.\n\n---\n\n[1] Glorot et al. “Understanding the Difficulty of Training Deep Feedforward Neural Networks”, AISTATS 2010"}, {"review_id": "g9sq3XCmQd", "reviewer": "Reviewer_Jxub", "summary": "The paper introduces class vectors, a method for model editing of image classification models. \nClass vectors capture class-specific representation adjustments during fine-tuning (compared to pretraining weights), leading to efficient modification of classifier behavior without extensive retraining.\nOne important finding is that these vectors exhibit linearity and independence, allowing for intuitive edits via simple **arithmetic** operations. Class vectors is evaluated on four types of tasks, including model unlearning, adaptation to snowy environments, defense against typographic attacks and adversarial trigger optimization.", "questions": "1. Class vectors demonstrate that modifying a target class’s representations does not influence other classes. I wonder if this phenomena still exist in datasets with a more complex label space(>=100 labels)?\n\n2. For Figure 6c, I wonder if the sample number keeps increasing, when will retrain surpass Class Vectors? Increasing the sample size to 100 or more is worth trying.", "limitations": "yes.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n1. The findings are novel and interesting. The authors points out that during fine-tuning, the weight changes of pretrained models of each class are linear and independent, editing for one class may not affect other classes. These findings can be connected to established phenomena like Neural Collapse and Cross-Task Linearity.\n\n2. Class vectors is computationally efficient, requiring minimal data and parameters, which is a major advantage over retraining or other related methods.\n\n3. Class vectors is evaluated across 4 different tasks (unlearning, environmental adaptation, adversarial defense...), proving its applicability in real-world scenarios.\n\n**Weaknesses**:\n\n1. Based on the used datasets and writing, I think the method focuses on single-label classification tasks, without the multi-label classification problem which is more in line with the real-world scene. In the multi-label classification scenario, a sample may belong to multiple labels, and the centroids between different labels must be closer.  This setting brings greater challenges to Class vectors' findings and methods. Can Class vectors adapt to multi-label tasks? \n\n2. Empirical Scope: Most experiments are conducted on ViT, not on other convolutional nets (ResNet, ConvNext...). The features of the convolutional nets are very different from the patch-level token features of ViT. Although Figure 2 shows similar phenomena on ResNet18 and MLP, the experimental part does not present experiments on them. The effectiveness of Class vectors on the conv nets needs to be verified.", "was_revised": true, "final_justification": "The rebuttal addressed most of my concern, I increase the score", "rebuttal": "Dear Reviewer Jxub, we are grateful for your insightful comments and thoughtful questions. We address the key concerns below.\n\n### **(1) Additional results on multi-class classification**\n\nThank you for the insightful comment. To further validate the effectiveness of class vectors in multi-class settings, we conducted an additional unlearning experiment targeting class 0 in a multi-label classification task on PASCAL-VOC-2008 [1].\n\nFor unlearning efficacy, we report recall on the forgotten class, which directly measures the proportion of successfully erased predictions. We use F1-score to evaluate retention performance, as it is a standard metric that balances precision and recall in multi-label tasks [2, 3].  We conducted experiments on ViT-B/16 and report both (Recall, F1-score) for each method.\n\n| **Fine-tuned** | **Pretrained** | **Retrained**  | **NegGrad** | **Class Vector**  | **Class Vector+**  |\n| --- | --- | --- | --- | --- | --- |\n| 98.3, 90.1 | 100, 13.2 | 0.0, 74.7 | 0.9, 64.8 | 23.8, 69.2 | 5.1, 80.5 |\n\nAs a result, class vectors, particularly the training-free Class Vector+, show strong performance in multi-label unlearning, achieving near-complete forgetting with competitive retention. However, most methods appear to struggle with retention in the multi-label setting. For example, even Class Vector+ shows a drop from 90.1 to 80.5. As you pointed out, this may be due to label centroids being densely clustered in multi-label scenarios. While the latent-to-weight mapping in Class Vector was trained with default unlearning hyperparameters due to time constraints, Class Vector+ offers a practical and effective upper bound for our approach.\n\n### **(2) Additional experiments with multiple architectures**\n\nThank you for giving us the valuable comment. Following your suggestion, we conducted additional class unlearning experiments on ResNet18, ResNet50 [4], and ConvNeXT-Tiny [5] for image classification, as well as a BERT-Base encoder [6] for text classification. For detailed experimental settings and results, please refer to our response to **Reviewer QfPd, item (2)**.\n\nOverall, the class vector methods outperform baseline approaches and demonstrate effective editing across architectures, highlighting both the effectiveness and robustness of class vectors across a variety of model architectures.\n\nWe will include all results in the Appendix in the revised version.\n\n### **(3) Scalability of class vectors on class number**\n\nThank you for pointing out this important issue. We acknowledge that many-class classification problems are highly relevant in real-world scenarios. To validate the effectiveness of class vectors in settings with over 100 labels, we conducted additional unlearning experiments on CIFAR-100 and the Stanford Cars dataset (196 classes), using only 0.1% of the validation set for training.\n\n| Dataset | **Retrained** | **NegGrad** | **Random Vector** | **Class Vector** | **Class Vector+** |\n| --- | --- | --- | --- | --- | --- |\n| CIFAR100 | 53.8, 78.8 | 3.0, 60.0 | 67.8, 71.8 | 5.0, 62.2 | 0.2, 69.0 |\n| CARS (196) | 56.0, 66.2 | 8.0, 53.6 | 65.1, 60.5 | 6.2, 56.6 | 9.8, 54.6 |\n\nAs the number of classes increases, class vectors remain effective, demonstrating their scalability to many-class tasks. However, we also observe a noticeable drop in retention accuracy when the number of labels exceeds 100 (e.g., from 69.2 to 56.6 / 54.6), suggesting increased interference among classes, similar to the findings discussed in our response to **Reviewer Jxub, item (1)**. We leave further investigation of this phenomenon as an important direction for future work. These experimental results and corresponding limitations will be included in the revised Appendix.\n\n### **(4) Scalability of class vectors on sample number**\n\nThank you for pointing this out. To demonstrate the scalability of class vectors with respect to sample size, we conducted additional experiments comparing them with retraining-based methods on MNIST unlearning, Snowy ImageNet and Typo Attack. Please see our response to **Reviewer tBPC, item (3)**, for results and details.\n\nThe results highlight that the class vector method consistently demonstrates effective model editing with an extremely small number of samples, reinforcing its practicality in low-resource settings—and in some cases, even outperforming retraining-based methods when even data is sufficient. In contrast, we find that retraining-based methods often perform better with more samples (e.g., 12 samples on Snowy ImageNet and 8 samples on Typo Attack). However, these sample sizes correspond to approximately 60% and 50% of the respective validation sets, which is an unrealistic assumption in real-world scenarios.\n\n---\n\n[1] Everingham et al. “The PASCAL Visual Object Classes (VOC) Challenge”, IJCV 2010\n\n[2] Su et al. “Contrastive Learning-Enhanced Nearest Neighbor Mechanism for Multi-Label Text Classification”, ACL 2022\n\n[3] Lanchantin et al. “General Multi-label Image Classification with Transformers”, CVPR 2021\n\n[4] He et al. “Deep Residual Learning for Image Recognition”, CVPR 2016\n\n[5] Liu et al. “A ConvNet for the 2020s”, CVPR 2022\n\n[6] Devlin et al. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, ACL 2019"}]

Author rebuttal:

Meta-review: This work introduces a novel approach to model editing using class vectors, which manipulate hidden representations, as opposed to task vectors that modify model weights.  The effectiveness of the proposed methods is demonstrated through experiments using three sizes of ViTs, showcasing the utility of class vectors in various applications, including class unlearning, environmental adaptation, defense against typographic attacks, and adversarial trigger optimization.

Reviewers praised the clarify of writing, the practicality and efficiency of the methods, and the comprehensiveness of experiments.

Other the other hand, they had concerns about issues including comparisons with task vectors, limitations in relation to unlearning, need to have both pre-trained and fine-tuned models, dependence of performance on sample size, and focus on ViTs.

The authors provided a detailed rebuttal, including a summary of further experiments, and some discussion with reviewers followed, during which some reviewers raised their ratings.

Overall, this submission is borderline, where a consensus among reviewers was not reached and most reviewers had low confidence.  On balance, I recommend acceptance, partly due to the positive trajectory after rebuttal.
Decision: Accept (poster)