Title: Exploring and Leveraging Class Vectors for Classifier Editing
Venue: NeurIPS 2025
Accepted: yes
Keywords: Classifier Editing

Vision Transformer

Class Vector
Abstract: Image classifiers play a critical role in detecting diseases in medical imaging and identifying anomalies in manufacturing processes. However, their predefined behaviors after extensive training make post hoc model editing difficult, especially when it comes to forgetting specific classes or adapting to distribution shifts. Existing classifier editing methods either focus narrowly on correcting errors or incur extensive retraining costs, creating a bottleneck for flexible editing. Moreover, such editing has seen limited investigation in image classification. To overcome these challenges, we introduce class vectors, which capture class-specific representation adjustments during fine-tuning. Whereas task vectors encode task-level changes in weight space, class vectors disentangle each class’s adaptation in the latent space. We show that class vectors capture each class’s semantic shift and that classifier editing can be achieved either by steering latent features along these vectors or by mapping them into weight space to update the decision boundaries. We also demonstrate that the inherent linearity and orthogonality of class vectors support efficient, flexible, and high-level concept editing via simple class arithmetic. Finally, we validate their utility in applications such as unlearning, environmental adaptation, adversarial defense, and adversarial trigger optimization.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces class vectors, a method for model editing of image classification models. \nClass vectors capture class-specific representation adjustments during fine-tuning (compared to pretraining weights), leading to efficient modification of classifier behavior without extensive retraining.\nOne important finding is that these vectors exhibit linearity and independence, allowing for intuitive edits via simple **arithmetic** operations. Class vectors is evaluated on four types of tasks, including model unlearning, adaptation to snowy environments, defense against typographic attacks and adversarial trigger optimization.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n1. The findings are novel and interesting. The authors points out that during fine-tuning, the weight changes of pretrained models of each class are linear and independent, editing for one class may not affect other classes. These findings can be connected to established phenomena like Neural Collapse and Cross-Task Linearity.\n\n2. Class vectors is computationally efficient, requiring minimal data and parameters, which is a major advantage over retraining or other related methods.\n\n3. Class vectors is evaluated across 4 different tasks (unlearning, environmental adaptation, adversarial defense...), proving its applicability in real-world scenarios.\n\n**Weaknesses**:\n\n1. Based on the used datasets and writing, I think the method focuses on single-label classification tasks, without the multi-label classification problem which is more in line with the real-world scene. In the multi-label classification scenario, a sample may belong to multiple labels, and the centroids between different labels must be closer.  This setting brings greater challenges to Class vectors' findings and methods. Can Class vectors adapt to multi-label tasks? \n\n2. Empirical Scope: Most experiments are conducted on ViT, not on other convolutional nets (ResNet, ConvNext...). The features of the convolutional nets are very different from the patch-level token features of ViT. Although Figure 2 shows similar phenomena on ResNet18 and MLP, the experimental part does not present experiments on them. The effectiveness of Class vectors on the conv nets needs to be verified.\n\nQUESTIONS:\n1. Class vectors demonstrate that modifying a target class’s representations does not influence other classes. I wonder if this phenomena still exist in datasets with a more complex label space(>=100 labels)?\n\n2. For Figure 6c, I wonder if the sample number keeps increasing, when will retrain surpass Class Vectors? Increasing the sample size to 100 or more is worth trying.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a novel approach to model editing using class vectors, which manipulate hidden representations, as opposed to task vectors that modify model weights. The authors provide theoretical backing for their method, citing task-level weight linearity and CTL. They present two main techniques: training-free Latent-space injection and Weight-space mapping, the latter of which fine-tunes the model for better alignment with the identified class vectors. The effectiveness of these methods is demonstrated through experiments using three sizes of ViTs, showcasing the utility of class vectors in various applications, including class unlearning, environmental adaptation, defense against typographic attacks, and adversarial trigger optimization.\n\nSTRENGTHS AND WEAKNESSES:\n#Strengths \n\nS1. The paper's primary strength lies in its strong theoretical justification for the proposed method. By grounding the methodology, the authors provide a compelling argument for why it should work. This goes beyond simple empirical success and provides confidence that the results are not just an artifact of the chosen datasets. \n\nS2. The method offers a highly practical and efficient framework for classifier modification. The most important part is performing those edits without additional training with latent-space injection and with weight-space mapping method that requires a small fine-tuning. \n \nS3. The paper demonstrates the effectiveness of class vectors across a diverse and practical set of classifier editing tasks. The applications, from unlearning and environmental adaptation to defending against typographic attacks, are well-motivated and showcase the flexibility of the proposed framework. The results, particularly for unlearning and adaptation, are strong, showing significant improvements over baselines with remarkable efficiency. \n\n#Weaknesses \n\nW1. The central weakness is the lack of a direct comparison to task vector arithmetic. Since the cost of fine-tuning is a prerequisite for both methods, the paper must empirically demonstrate why class vectors are superior for class-specific editing. The authors should include task vectors as a baseline in the class-unlearning experiments (e.g., Table 1) to validate their claim of improved granularity. Without this comparison, the novelty and practical advantage of the proposed method are not fully substantiated. \n\nW2. The methodology requires access to both a pretrained model and a corresponding fine-tuned version to compute the class vectors. This could limit its applicability in scenarios where only the final, fine-tuned model is available, which is common in the context of using models from open-source hubs. \n\nW3. The paper's analysis of how performance scales with sample size is limited. The only direct ablation study (Figure 6c) is confined to a very small data regime (1-5 samples) for a single task. While this demonstrates the method's strength in data-scarce scenarios, it fails to provide a clear picture of the trade-offs when more data is available. A more thorough analysis showing how the method compares to standard fine-tuning at larger sample sizes (e.g., 10, 50, 100) is needed to understand its scalability and when it might be preferable to a full retraining.\n\nQUESTIONS:\n1. Can the authors evaluate how task vectors compare to their method? \n\n2. Have the authors considered getting class vectors without using a finetuned model to make this method even more efficient? \n\n3. Can authors provide more points in Figure 6c as e.g., for 10, 5,0, and 100 examples to have a clearer picture of the scalability of the proposed method\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
The paper introduces class vectors, latent difference vectors between a class’s pretrained and fine-tuned centroids, and shows they enable fast, targeted edits to vision encoders via latent steering or a tiny weight-mapping adapter. Experiments on multiple ViT-CLIP models demonstrate consistent gains over retraining and recent editing baselines, highlighting CVs’ practicality but also revealing limitations in generality and baseline coverage.

STRENGTHS AND WEAKNESSES:
Strengths
1. The motivation of the paper is natural, which moves task-level vectors to class vectors to capture each class’s semantic shift.
2. The proposed method is theoretically sound, with connecting class vectors to Cross-task linearity and proofs, and empirical experiments in appendix.
3. The ablation studies and analyses are thorough and insightful.
Weaknesses
1. Building on prior work like Task Vectors, In-Context Vectors, and CTL, the authors should clarify how Class Vectors differ from concept activation and representation surgery methods.
2. Authors should conduct experiments to compare the task-vectors with class-vectors.
3. Typo in algorithm title, “classfier”.

QUESTIONS:
1. Have you tried CV editing on a traditional CNN or a BERT-style text encoder?
2. The current study modifies one class at a time. What happens when several mutually interacting classes (e.g., “cat”, “lion”, “tiger”) are edited together?

LIMITATIONS:
See weaknesses
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes class vectors for editing image classifiers efficiently. A class vector represents how a specific class's latent features change during fine-tuning. Unlike task vectors, class vectors allow per-class fine grained edits. This paper also found that class vectors follow linear trajectories in latent space (Cross-Task Linearity). Besides the insights above, this paper also explores the application of class vectors on several application domains like class unlearning, environmental adaptation, adversarial defense and etc. And the method is fast, flexible, and works well with very few samples. It shows strong results across several benchmarks like MNIST and ImageNet.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper is logically organized and clearly written, making it easy to follow the proposed ideas and methodology.\n\n2. Theoretical analysis is well-grounded, and the empirical results are convincing, demonstrating the effectiveness of class vectors across multiple settings.\n\n3. The experimental validation is extensive, covering a wide range of applications such as class unlearning, environmental adaptation, typographic attack defense, and backdoor trigger optimization.\n\nWeakness:\n\n1. The machine unlearning experiments lack strong baselines. Although simple comparisons are acceptable for exploring feasibility, including more competitive or state-of-the-art baselines like salun [1] would provide a better benchmark for evaluating performance.\n\n> [1] Fan, Chongyu, et al. \"Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation.\" arXiv preprint arXiv:231\n\n2. The paper is inspired by task vectors but does not include a direct comparison between task vectors and class vectors. Since both approaches aim to edit model behavior, especially in applications like unlearning, it would be helpful to include empirical comparisons or theoretical discussion to clarify their respective advantages and limitations. At least, it is better to make it clear for the comparison between two methods\n\n3. Minor issue: the Cross-Task Linearity equation on page 3 appears to contain an error and should be double-checked for correctness.\n\nQUESTIONS:\nPlease refer to weakness part for the questions:\n\n1. Could you please add more baselines for each applications?\n\n2. Any experiments to show the benefits for using class vector instead of task vector?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}