Approximate Domain Unlearning for Vision-Language Models
Authors: Kodai Kawamura
Yuta Goto
Rintaro Yanagi
Hirokatsu Kataoka
Go Irie
Venue: NeurIPS 2025 spotlight ()
Keywords: Approximate unlearning
Vision-Language Models
Primary area: deep_learning
TL;DR: We propose Approximate Domain Unlearning (ADU), a new task for selectively removing domain-specific knowledge from pre-trained vision-language models, and introduce two key techniques, domain disentangling and instance-wise prompting, to solve it.

Abstract: Pre-trained Vision-Language Models (VLMs) exhibit strong generalization capabilities, enabling them to recognize a wide range of objects across diverse domains without additional training. However, they often retain irrelevant information beyond the requirements of specific target downstream tasks, raising concerns about computational efficiency and potential information leakage. This has motivated growing interest in approximate unlearning, which aims to selectively remove unnecessary knowledge while preserving overall model performance. Existing approaches to approximate unlearning have primarily focused on {\em class unlearning}, where a VLM is retrained to fail to recognize specified object classes while maintaining accuracy for others. However, merely forgetting object classes is often insufficient in practical applications. For instance, an autonomous driving system should accurately recognize {\em real} cars, while avoiding misrecognition of {\em illustrated} cars depicted in roadside advertisements as {\em real} cars, which could be hazardous. In this paper, we introduce {\em Approximate Domain Unlearning (ADU)}, a novel problem setting that requires reducing recognition accuracy for images from specified domains (e.g., {\em illustration}) while preserving accuracy for other domains (e.g., {\em real}). ADU presents new technical challenges: due to the strong domain generalization capability of pre-trained VLMs, domain distributions are highly entangled in the feature space, making naive approaches based on penalizing target domains ineffective. To tackle this limitation, we propose a novel approach that explicitly disentangles domain distributions and adaptively captures instance-specific domain information. Extensive experiments on four multi-domain benchmark datasets demonstrate that our approach significantly outperforms strong baselines built upon state-of-the-art VLM tuning techniques, paving the way for practical and fine-grained unlearning in VLMs. Code : https://kodaikawamura.github.io/Domain_Unlearning/.

Approximate Domain Unlearning for Vision-Language Models
Kodai Kawamura 1.2, Yuta Goto"!, Rintaro Yanagi', Hirokatsu Kataoka 34, Go Irie!
Tooky University of Science
National University of Singapore
National Institute of Advanced Industrial Science and Technology (AIST)
University of Oxford
Abstract
Pre-trained Visgon-Language Models (VLMs) exhibit strong generalization capa- bilities, enabling them to recognize a wide range of objects across diverse domains without additional training. However, they often retain irrelevant information be- yond the requirements of specific target downstream tasks, raising concerns about computational efficiency and potential information leakage. This has motivated growing interest in approximate unlearning, which aims to selectively remove unnecessary knowledge while preserving overall model performance. Existing approaches to approximate unlearning have primarily focused on class unlearn- ing. where VLM is retrained to fail to recognize specified object classes while maintaining accuracy for others. However, merely forgetting object classes is often insufficient in practical applications. For instance, an autonomous driv- ing system should accurately recognize real cars, while avoiding misrecogni- tion of illustrated cars depicted in roadside advertisements as real cars, which could be hazardous. In this paper, we introduce Approximate Domain Unlearn- ing (ADU), a novel problem setting that requires reducing recognition accuracy for images from specified domains (e.g., illustration) while preserving accuracy for other domains (e.g., real). ADU presents new technical challenges: due to the strong domain generalization capability of pre-trained VLMs, domain dis- tributions are highly entangled in the feature space, making naive approaches based on penalizing target domains ineffective. To tackle this limitation, we propose a novel approach that explicitly disentangles domain distributions and adaptively captures instance-specifie domain information. Extensive experiments on four multi-domain benchmark datasets demonstrate that our approach sig- nificantly outperforms strong baselines built upon state-of-the-art VLM tuning techniques, paving the way for practical and fine-grained unlearning in VLMs. Code : /tttps://kodaalawaamuz gi github. io/ io/Domain, http:aio/ooooonnunnaaaaaaaaaaa Unlearning/
Introduction
Pre-trained Vision-Language Models (VLMs) exhibit remarkable generalization capabilities, enabling them recognize wide range of object classes without any additional training [Radford et al. 2021 Jia et al. 1 Li et al. 2022 2023 Yu et al., 2022]. However, many practical downstream tasks do require leveraging their full generalization capability. For instance, autonomous driving system must recognize "carss and "pedestrians" but does not need to identify "foods" or "groceries", Re- taining unnecessary knowledge introduces serious risks, including excessive computational resource
"qqual contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

consumption and potential information leakage [Fredrikson et al. 2015 Bommasani et al. 2021 Shokri et al., 2017]. To address these concerns, approximate unlearning (also known as selective forgetting), which aims to make classification models forget specified knowledge while preserving the rest, has gained significant attention [Shibata et al. 2021 Kuwana et al. 2024 Graves et al. 2021
Ye et al. 2022 Tarun et al. 2023 Fan et al.] 2024 Chen et al. 2023
Huang et al. 2024bb]
Previous studies on approximate unlearning for pre-trained VLMs have primarily focused on class unlearning, which aims to reduce the recognition accuracy of specified object classes while preserving that of the others (Huang et al.) 2024b Kuwana et al. 2024). A common ap- proach involves increasing the classification loss for the classes to be forgotten (thus reduc- ing their accuracy) while decreasing for the classes to be retained (thus improving their ac- curacy). Building on this idea, several methods have been developed, including a technique that optimizes the gradient direction for more effec- tive unlearning (Huang et al.] 2024b and approach tailored for black-box VLMs Kuwana ct al. 2024].
Preserved Domain Roal
Forgotten Domains
Domain
Painting
Clipart
Input Image
Zoro-shot CLIP
Ours (ADU)
Figure I: Illustration of Approximate Domain Unlearning (ADU). ADU is novel approxi- mate unlearning problem introduced in this pa- per. Unlike existing approximate class unlearn- ing tasks, ADU requires retraining a pre-trained Vision- Languae Model (VLM) so that it cannot recogmize images from specified domains (paint- ing, clipart, sketch in the figure) while preserving its ability to recognize images from other domains (real in the figure).
However, we argue that merely forgetting object classes is often insufficient for real- real-world appli- cations. Recall the example of an autonomous driving system. Such system must accurately recognize real cars to control following dis- tances and prevent collisions. In contrast, if an advertisement depicting illustrated cars on the roadside is mistakenly recognized real cars, undesirable behaviors could be triggered, potentially compromising safety. Indeed, due to their strong domain generalization capability, pre-trained VLMs can indiscriminately recognize both real and illustrated cars as "cars." In such cases, simply forgetting the object class "car" is inadequate; a more fine ine-grained selective forgetting technique required to preserve the recognition accuracy of real cars while reducing it for illustrated cars.
In this paper, we introduce a novel variant of the approximate unlearning problem, called Approximate Domain Unlearning (ADU). Fig. [ provides an illustration of the proposed ADU task. While conventional approximate class unlearning aims retrain pre-trained VLM so that it cannot recognize specified object classes, ADU requires tuning the model so that cannot recognize images from specified domains while preserving its recognition ability for other domains. For example, a pre-trained VLM may initially recognize a car regardless of its domain, whether it appears in a real photograph, painting, clipart, sketch. If the real domain is designated to be retained, ADU selectively untrains the model so that only real images of cars remain recognizable, while recognition accuracy for cars in other domains is reduced. ADU is fundamentally different from the traditional class unlearning setting and has not been explored in prior research. As discussed above, it opens new direction in approximate unlearning, motivated by practical considerations.
ADU also introduces a new technical challenge. A straightforward idea to address ADU would be to adapt the common approximate class unlearning strategy minimizing the classification loss for domains to be retained while maximizing it for those to be forgotten [Huang et al. 2024b]] Kuwana 2024]. However, this approach fails to deliver satisfactory domain unlearning performance. This limitation arises from the strong domain generalization capability of pre-trained VLMs, leading to significant entanglement between different domain distributions in the latent feature space [Kumar
et al.
ct al. 2022 Wang et al., 2024), unlike class distributions, which are typically well-separated. Conse- quently, attempts preserve or degrade recognition accuracy for a specific domain inevitably affect all domains, making domain-wine wise control over feature representations inherently difficult. To address this issue, we propose novel approach specifically tailored the problem of ADU. Specifically, we introduce the Domain Disentangling Loss (DDL), which explicitly encourages separation of domain distributions in the latent space. Furthermore, recognizing that domain characteristies may vary in

---

Cross Entropy
Instance-wise
Prompt Generator (InstaPG)
photo of [CLASS].
Text Encoder
Entropy Maximization
Soaled Dot Product
Car
Domains to be memorized
Spoun Aeaasaaaa PU
Image Encoder
Domain Classifier
Domains to forgotten
Instance Prompt Generator
Image
Vision Prompt
Clase Takon Emibedding
tretance-wtla Prompt
Domain Disentangling Loss (DOL)
(a) Overview of Our Method
(b) Instance-wist -iie Prompt Generator (InstaPG)
Figure 2: Overview of Proposed Method. (a) The common approach to approximate unlearning is to minimize the cross-entropy to the ground truth class labels for the domains to memorized and to maximize the entropy for the domains to be forgotten. This approach alone is not satisfactory, due to the strong generalization performance of pre-trained VLMs. We therefore introduce two techniques to facilitate ADU; Domain Disentangling Loss (DDL) to disentangle the domain distrubtions in the latent feature space and Instance- wise Prompt Generator (InstaPG) to capture image-level differences of domains. (b) InstaPG utilizes attention mechanism where the vision prompt acts the query and the image patch features serve as the key and value. Through this mechanism, instance- wise prompts dynamically generated, allowing the model to adaptively refine prompts based individual image characteristics.
strength and spatial extent across images, we introduce an Instance-wise Prompt Generator (InstaPG) to adaptively model these properties on a per-image basis. Experiments on four multi-domain image benchmark datasets demonstrate that our method significantly outperforms strong baselines built upon state-of-the-art VLM tuning techniques, achieving superior domain unlearning performance.
The main contributions of this paper are summarized follows:
We introduce Approximate Domain Unlearning (ADU), a novel problem setting that extends the notion of approximate unlearning to the domain level, addressing practical limitations of conventional class unlearning.
We propose a novel approach to ADU, featuring the Domain Disentangling Loss (DDL) to explicitly disentangle domain distributions in the latent space and the Instance-wise Prompt Generator (InstaPG) to adaptively model instance-level domain variations.
Extensive experiments on four multi-domain image benchmark datasets validate the effectiveness of our approach, showing substantial improvements over strong baselines built upon state-of-the- art VLM tuning techniques.
2
Related Work
2.1 Machine Unlearning
Machine unlearning aims to remove the influence of specific samples from pre-trained models,
motivated by the growing need eliminate traces of particular data from such models (Shaik et al.
2024. Xu et al. 2024 Cao and Yang. 2015 Neel et al. 202T Sekhari et al., 202T Ullah et al.) 2021
Guo et al./ 2220, Golatkar et al., [021 Chen al. [019 Brophy and Lowd 2021] Sun et al. 2023]
Existing methods are broadly categorized into two approaches: exact unlearning and approximate
unlearning. Exact unlearning either retrains a model from scratch after removing the target samples, or alternatively modifies the model so that its parameter distribution exactly matches that of a model trained from scratch without those samples (Thudi et al.) 2022a b). While exact unlearning has been extensively studied in the context of convex optimization of linear machine learning models [Guo al. 2020 Izzo et al. 2021 Neel al. 2021], applying it to deep neural networks typically requires full retraining [Bourtoule et al. 2021] process that is computationally prohibitive for practical applications. To address this limitation. recent research has increasingly focused on approximate
unlearning Fan et al. 2024
Ginart al. 2019 Golatkar et al. 2020 Guo al. 2020 Jia et al. 2023
3

---

Warnecke et al. 2023 Graves et al., 2021 Chundawat et al. 2023 Huang et al. 2024b Kuwana et al. 20244 Shibata et al. 2021]. which seeks to efficiently approximate the result exact un unlearning without incurring the full computational cost. Huang et al. 2024b) proposed an update rule based on information- -egoomtiionn-eommrriic gradient directions, which mitigates interference with the classification accuracy of retained classes. Similarly, Kuwana et al. [2024) introduced black-box unlearning method for VLMs, enabling forgetting in proprietary models with undisclosed internal architectures.
Despite these advances, existing approximate unlearning approaches have focused exclusively on class-level forgetting. To the best of our knowledge, domain-level unlearning has not been explored. This study presents the first investigation of Approximate Domain Unlearning (ADU), introduc- ing novel framework for selectively unlearning domain-specific information while preserving generalization capability.
2.2 Domain Adaptation / Generalization
Domain Adaptation (DA) and Domain Generalization (DG) are two prominent strategies for tackling domain shifts in machine learning. DA assumes access to (labeled or unlabeled) target-domain data during training, while DG requires models generalize to unseen domains without exposure to target data [Patel et al., 2015 Zhou et al.] 2022a]. Recent efforts in these areas have explored various strategies, including adversarial training Ganin al., 2016),
invariant representation learning [Li et al., 2018b]. meta meta-learning (Li et 201 and prompt tuning for large-scale pre- trained models (hhou et al., 2022b Shu et al.) 2022]. A classical and widely adopted technique for distribution alignment in DA and DG is Maximum Mean Discrepancy (MMD) [Gretton et al.) 2012]. non-parametric measure that estimates the distance between probability distributions in a Reproducing Kernel Hilbert Space (RKHS). Many domain adaptation methods incorporate MMD as a regularizer to minimize the distance between marginal or joint feature distributions of source and target domains. Representative examples include Deep Adaptation Network (DAN) [Long et al.) 2013) and Joint Adaptation Network (JAN) [Long al.] 2017], as well as follow-up works that further refine the aligmment process Yan et al. 2017 Wang et al. 2020 Mekhazni et al. 2020).
Unlike these methods, which minimize MMD enforce domain invariance, our approach maximizes MMD within a Domain Disentangling Loss (DDL) to deliberately separate domain-specific represen- tations. This inversion of the conventional alignment objective reflects the fundamentally different nature of ADU, which requires not domain invariance but domain disentanglement for the selective removal of unwanted domain-specinn knowledge.
Method
3.1
Approximate Domain Unlearning (ADU)
Given a set of training data {xx y, J where x E X' represents an input image, y E is the class label, and d E D is the domain label, with X, C and D denoting the input space, the set of all classes, and the set of all domains, respectively. We define Dimeorrze C D as the set of domains to be preserved and D Piorget D \ D Pmemortie the set of domains to be forgotten. Our goal is to retrain a pre-trained VLM f to maintain the classification accuracy for {(xx y, d) E Dmmemonze.. while reducing it for {(x, y, d)ld )-
3.2 Applying Common Approximate Class Unlearning Approach to ADU
The common approach to the conventional approximate class unlearning is to use two different loss functions; one for retaining the classification accuracy for the classes to be retained and the other for reducing the accuracy for those to be forgotten (Huang et al., 2024b Kuwana et al. 2024). Specifically, given mini-bated B {(x; Vs. d;) ) /==1 the cross-entropy to the ground truth class labels Lmemorize is minimized for the classes to retained, and the cross-entropy to the uniform class labels Liorger is minimized (which corresponds to entropy maximization) for the classes to be forgotten [Kuwana et al. 2024]
C
mmmmorize (B)
yki log Pijs
(I)
B

---

[88
1
Ltorgot (B)
log Pij.
B
where Di (pi1. Pi2: .i2:....Pipp ) represents the confidence scores of sample x, output by the model, and y; = (Wh1, .0....... Mee))) denotes the one-hot encoding the class label yi-
Given this idea, a straightforward approach to ADU would be to adapt these two loss functions, that is, minimizing Cmemorize for {(z, y, d)ld 2 Pmemerize J and Lorget for {(z, y, d)]d toorett) However, as we will show later in our experiments, this straightforward approach alone is insufficient achieve satisfactory ADU performance. This is primarily due the strong domain generalization capability of pre-trained VLMs. As evidenced by their robustness to domain shifts, the latent space of VLMs highly aligns data distributions across different domains, meaning that covariate shifts are minimal. Consequently, the feature distributions across different domains are highly entangled, making it difficult effectively control memorization and forgetting on per-domain basis.
3.3 Domain Disentangling Loss (DDL)
To address this issue, we propose Domain Disentangling Loss (DDL), which aims explicitly disentangle the feature distributions among different domains in the latent feature space. The core idea that if the feature distributions of individual domains are well-separated, the domain label d of a given sample x can be accurately predicted, and vice versa. Based on this insight, DDL encourages the domain label of sample be predictable from its latent feature through an auxiliary domain classifier. More specifically, we introduce a standard cross-entropy loss that requires the model to correctly predict the domain labels of the samples:
B
CCr(B)
dig log ply
BB
(p Pl .l...... PID)) represents the confidence scores of a sample x, output by the domain classifier (a fully connected layer), and d, = (d,, d2 . d is the one-hot encoding of the domain label d.
where D
To further enhance domain separability, we additionally incorporate the Maximum Mean Discrepancy (MMD) into DDL as auxiliary loss term. MMD estimates the pairwise distance between domain distributions a Reproducing Kernel Hilbert Space (RKHS) as:
MMD2(B) [DI[D] 1) 1sded'g(p)
Σ
(xx,
Σ
o(x;)
By
B
X.EB.
x
H
where & denotes a kermell-nduued feature mapping, B3 a subset of mini-batch within the domain d E D. Intuitively, maximizing MMD increases the inter-domain divergence in the latent space.
Given the above formulations, our final DDL loss is defined as:
Caomain ( (B) AMMD"(BB,
where y and are balaneing hyperparameters. Combining with the standard loss functions Lmemorize and Liorget, the learnable prompts and the domain classifier are jointly optimized by minimizing the total loss:
Liotal (B) = Ememorize (B) Ltorget e (B) + Ledomain (B).
3.4 Instance-wise Prompt Generator (InstaPG)
Domain is often ambiguous. The term "illustration," for example, encompasses broad spectrum of styles, from highly realistic renderings that closely resemble real-world images to highly stylized depictions resembling clipart, with each image varying in style. Given this nature of the domain, only learnable prompt that is uniform across all the images cannot account for such an instance-level variation in the images, which may degrade the performance of domain unlearning.
5

---

Table 1: Comparison with Baseline CLIP Fine-tuning Methods. The average performance of all possible combinations of the domains to be forgotten/retained are reported for various number of domains to be forgotten [Pinget E (1,2,3) ImageNet has only two domains, so only the case (Druggel 1 is tested. Performance is evaluated using the three metrics: (i) Mem: the accuracy of the classes for data from memorized domains, (ii) For: the error of the classes for data from forgotten domains, and (iii) H: the harmonic mean of Mem and For. Higher values mean better performance. The Baseline in the table refers to the method that trains the vision prompt using Lemorrize and
Ltorget in Sec. 3.2 representing a straightforward and conventional approach to unlearning.
ImageNet
Office-Home
Mini DomainNet
iDirgal
Method
H
Memt
Fort
H f
Ment Fort
H
Mem
Fort
LP+ [Huan et al.
2024a
50.69
62.22
42.76
30.46
85.06
18.55
31.73
84.72
19.52
CLIPPI Li al. 222
71.31
63.04
82.13
43.44
83.20
29.40
53.56
84.32
39.24
[Droger|
BBF (uwaana el al. Baseline
2024
45.56
35.92
65.91
31.25
80.94
19.74
32.12
81.80
20.03
74.66
61.23
96.43
52.59
79.9
39.88
62.07
85.15
49.40
Ours
77.02
64.13
96.73
69.96
77.93
64.34
75.56
78.32
73.06
LP++ (Huang et al. 2024a
x
x
31.11
85.54
19.01
32.21
85,29
19.85
CLIPPit [L et al. 024
X
x
40.53
83.64
26.74
51.35
84.74
36.83
iDirger =2 BBF [Kuwana el al.
2024
X
X
32.54
80.42
20.54
47.95
61.48
40.52
Baseline
X
x 54.19
80.23
41.23
61.97
84.71
48.96
Ours
X
X
x
73.58
75.61
71.89
77.03
76.51
77.66
LP++ (Huang et al 2024a
X
X
x
33.57
86.22
20.84
34.73
85.87
21.77
CLIPFit [Li ct all 2024
x
50.02
85.68
35.32
52.88
85.39
38.30
[Drogal = 3 BBF [Kuwana el al.
2024
39.60
78.13
26.80
39.29
80.29
26.05
Baseline
X
59.47
80.96
48.79
68.82
84.25
58.27
Ours
X
X
75.89
72.15
80.77
79.00
75.00
83.94
To address this problem, we introduce an Instance-wise Prompt Generator (InstaPG) to adjust the learnable vision prompts according to the input image patches. The illustration given in Fig. 2b InstaPG is embedded in an intermediate layer (i.e., Transformer block) of the image encoder to generate additional instance-win wise prompts be fed to the subsequent layer via cross-attention mechanism, where the learnable vision prompts serve as queries, while the image patch features act keys and values. The generated prompts are conditioned on the input image patch features, allowing the model effectively captures the property of the input image.
4 Experiments
4.1 Settings
Datasets. We evaluate our method on four public
image classification datasets:
ImageNet (Deng et al. [2009] Office-Home /Venkateswara et al. [017], Mini DomainNet [Zhou] al. 2021 and DomainNet [Peng et al. 2019], which are widely used for evaluating domain adapratiiongenerrllaatiooo methods. For ImageNet, we treat ImageNet- K (Deng et al. 2009 and ImageNet-Sketch (Wang et al. 2019] as two distinet domains, containing approximately 1.28M and 50K samples, respectively, across 1,000 object classes. Office- (Venkateswara et al.) 2017) contains 15.5K samples of 65 object classes over four domains, namely, art, clipart, product, and real-world. Mini DomainNet (Zhou et al. 2021] consists of 140K images of 126 object classes from four domains: clipart. painting, real, and sketch. Details and results on DomainNet [Peng et al. 2019 provided in Appendix A Unless otherwise noted, use eight labeled samples per domain (both class and domain labels) for training, following the feo-shot setting commonly adopted in
recent VLM tuning (hou et al.] 2022b Khattak al. 2023 Li et al. 2024 Huang et al. 2024a] and
machine unlearning studies Kuwana et al. 2024].
Baselines.
Since this is the first work that introduces the task of ADU, there is no existing method
directly applicable to the task. Thus, we designed dedicated baselines for comparative experiments. Specifically, we evaluate the two state-of-the-an CL.IP fine-tuning methods, namely LP++ HHuag let al. [0244] and CLIPFit [Li al., 2024], tuned with the same loss functions given in Sec. 3.2 i.e.,
Ememorize and Liorget. We also compare method with the state-of- ofate-of-the-an art machine unlcarning

---

Table Ablation Study. The ablation results of Domain Disentangling Loss (DDL) and Instance- wise Prompt Generator (InstaPG) are reported. While our method achieves improvements with either DDL or InstaPG alone, combining both leads to even better balanced trade-off performance.
Diogel
1
Dreget 2
Dieegrr
Datasel
DDL InstaPG
H Mentt
Fort
H
Memt
Fort
H 1
Mem Fort
52.59 79.96
39.88
54.19
80.23
41.23
59.47 80.96 48.79
56.41
83.55
44.05 61.01
83.33
48.60
70.60
81.55 63.28
Office-Homee
60.82
74.51
51.72
60.01
76.22
49.94
63.12
77.74 54.57
69.96
77.93
64.34
73.58
75.61
71.89
75.89
72.15 80.77
62.07
85.15
49.40
61.97
84.71
48.96
68.82
84.25 58.27
64.06
85.46
51.64 65.92 85.12 53.96 74.17 83.68 66.68
Mini DomainNet
74.23 78.00
70.87
75.28 77.57 73.20
77.60 77.42 77.90
75.56 78.32 73.06
77.03 76.51 77.66 81.78
77.96 86.12
Table 3: Ablation Studies of Loss Functions in Domain Disentangling Loss (DDL). DDL employs Cross-Entropy (CE) and Maximum Mean Discrepancy (MMD). Using both losses together yields the best-balanced trade-off, as shown by H.
two loss functions:
[Progel
DDogel = =
Diagel
3
CE
MMD
H
t
Mem For t
Mem 1 For
H
Mem f For t
56.41
83.55
44.05
61.01
83.33
48.60
70.60
81.55
63.28
68.62
82.41
59.47
72.21
81.01
65.66
66.76
82.38
57.12
V
64.01
82.97
53.62
69.33
80.73
61.24
71.53
81.43
64.55
V
69.96
77.93
64.34
73.58
75.61
71.89
75.89
72.15
80.77
In addition, evaluate "Baseline" method that learns the learnable vision prompts with 4memorrze and Liorget.
method for VLMs, Black-Box Forgetting (BBF) Kuwana al. 2024
Implementation Details. We use pre-trained CLIP model with ViT-B/16 [Dosovitskiy et al. 2021 as the image encoder. The text prompt is set to "a photo [class]". For vision prompts, we adopt deep prompting [Khattak et al. 2023 with eight learnable context tokens and train the model for 50 epochs using SGD with a learning rate of 0.0025, The vision prompts are optimized within the first nine transformer layers of the image encoder. We consistently set the weights in loss functions y = 30 and À = 10.
Evaluation Metrics. We use the following three metrics: (i) Mem: the accuracy of the classes for data from memorized domains {(x, y, d)ld E Dimemorize:: (ii) For: the error of the classes for data from forgotten domains {(x, y. d'ld E Di ); (iii) H: the harmonic mean of Mem and For, representing the overall unlearning performance as it balances the forgetting rate for domains to be forgotten and the classification accuracy for domains to be memorized. Higher values for all these metrics are desirable. We report the average results over three runs using different random seeds.
4.2 Results
The comparative results are shown in Table methods on all the datasets in both Hl and For.
I Our method significantly outperforms all the compared
Comparing our method with the two state-of-the-a CLIP fine-tuning methods (i.e., LP++ and CLIP- Fit), our method ourperforms them by more than 20% H on Office-Home and Mini DomainNet, and by 5.71% on ImageNet, regardless of the number of domains to be forgotten. Furthermore, forgetting performance (For) of CLIPFit and LP++ is below 40% on Office-Home and Mini NotainNet, whereas our method achieves over 60%. These results demonstrate that even when state- of-the-art VL.M fine-tuning methods are equipped with common approximate unlearning strategies, they still struggle with ADU - highlighting the unique difficulty of domain unlearning.
7

---

00
50
50
25
16
(a)
(b) X
Figure 3: Impact of Loss Weights y and Domain Disentangling Loss (DDL). We analyze the effect of varying the loss weights and X, which control the weights of the cross-entropy loss and the Maximum Mean Discrepancy (MMD) loss, respectively. Performance remains stable across wide range of values once both and À exceed certain threshold, indicating that the proposed method is not highly sensitive to the choice these hyperparameters.
02
C0
70-
75.
19
15
Number training samples per domain
Number of tratiinng samples per domain
Number of training camples per domain
(a) Office Home
(b) Mini DomainNet
(c) ImageNet
Figure 4: Sensitivity to The Number of Training Samples per Domain. We compare our method with Baseline, which uses Lmemorize and Liorget, for vision prompt learning. While Baseline shows limited improvement with more shots, especially on Mini DomainNet, our method consistently improves, demonstrating better generalization and reduced overfitting.
When compared with BBF, the current state-of-the satte-of-thearre art class unlearning method for vision-language models applied to ADU, our method achieves over 30% higher For all datasets. These results suggest that class unlearning methods are not sufficient for domain unlearning, even a state-of-the-art method, and further validate the effectiveness of approach.
The Baseline in the table refers to the method that trains the vision prompt using only the common class unlearning losses, Ammmorizee and Ltorget (see Sec. 3.2), representing straightforward and conventional approach to unlearning. On OfficeHome and Mini DomainNet, our method surpasses Baseline by over 20% in For, demonstrating the forgetting capability of our method and showing the limitations of naive unlearning strategies in ADU.
4.3 Analysis
Ablation Study. Table shows ablation study of the proposed Domain Disentangling Loss (DDL) and Instance-wis Prompt Generator (InstaPG) on the Office-Home and Mini DomainNet datasets. Although DDL or InstaPG alone improves trade-off performance, combining it with InstaPG further boosts the scores both and For. encourage disentanglement the latent feature space, our DDL employs
(CE) and Maximum Mean Discrepancy (MMD) losses (see Eq. (5). BableBJsowss an ablation of these two components on effective forgetting while preserving aceuracy memorized domains. Using both losses together yields the balanced trade-off performance, demonstrating the complementary effectiveness of CE and MMD losses within DDL.
Both CE and MMD promote
Sensitivity Loss Weights. We analyze the sensitivity of our method two hyperparameters, ? and X, which respectively balance CE and MMD losses in DDL. Fig. shows the effect of varying and À on performance. As increases, performance improves and saturates around y = 10. Notably, it remains stable across wide range of values once 7 exceeds certain threshold, suggesting that the method not overly sensitive the choice of r.. similar trend observed for X. Performance peaks around À 10, and remains stable across wide range beyond that. This suggests that long

---

Table 4: Domain Classification Accuracy when Art is Forgotten on Office-Home.
Method
Domain Classification Accuracy [%]
Before Unlearning
25,80
Ours w/o DDL and InstaPG
31.06
Ours
79.43
dpor
panndeg real
penting
real seatch
(a) Zero-shot CL.IP
(b) Ours
Figure 5: t-SNE Visualization, Where the Domain to Be Forgotten is Real. (a) In the feature space of zero-shot CLIP, features from different domains are entangled, indicating poor domain separation. which makes domain-wise control over feature representations difficult. (b) By applying our method, the features are effectively disentangled from their domains, facilitating domain-wiss control.
as Å set above a certain threshold, the exact value has limited impact, making the method relatively insensitive to the choice of A.
Sensitivity to Number of Training Samples. In practical scenarios, the number of available samples per domain can vary substantially, making important to assess how unlcarning methods respond to different data regimes. We therefore evaluate the sensitivity of our approach to the number of training samples (i.e., shots) per domain, and compare it with the Baseline method. As shown in Fig. 4 method consistently improves as the number of shots increases, demonstrating its ability to effectively leverage additional data. In contrast, Baseline struggles to benefit from more shots on Mini DomainNet, suggesting a tendency toward overfitting- Our method continues gain performance with additional training samples, highlighting its robustness and stronger generalization capability.
Domain Classification Accuracy. An essential prerequisite for effective domain unlearning is that feature representations from different domains are well separated; in other words, domains should be easily distinguishable. Otherwise, attempts to suppress recognition accuracy on specific domain would inevitably affect others. Therefore, evaluating not only the final unlearning performance but also the domain classification accuracy is crucial, it provides direct evidence of the effectiveness of our key components, namely, DDL and InstaPG, in enhancing domain separability. We report domain classification accuracy before and after unlearning Office-Home in Table The accuracy increases substantially from 25.80% before unlearning to 79.43% after applying our method. When the DDL and InstaPG are removed, the accuracy drops remarkably to 31.06%, highlighting the critical role of these components in achieving effective domain separation. These findings provide strong additional evidence of the effectiveness of our method, particularly in facilitating domain-level unlearning.
I-SNE Visualization. Fig. 5 presennss& a t-SNE visualization of image features from Mini DomainNet, where the domain to forget is real. In Fig. 5a the features extracted by Zero-shot CLIP are heavily entangled across domains, indicating poor domain separation, which makes domain-wiss control difficult. Fig. 5b demonstrates that our method separates the domains in the feature space, enabling control over memorization and forgetting on per-domain basis.
Attention Map Visualization. To further investigate and understand the behavior of our unlearned model, we show the attention heatmaps [Selvaraju et al.) [017]] before and after applying our method on DomainNet Mini dataset in Fig. 6 where the domain to be forgotten is real. For the forgetting data, the Zero-shot CLIP attention concentrates on the objects. After applying our method, the attention on the objects disappears or significantly weakens for the data from the domain to be forgotten (i.e., real). For the data from the domain to be memorized (i.e., painting, clipart, and sketch), our method fully maintains strengthens previous attention on the objects. Our method suppresses

---

Real (Forgotten)
Painting
Clipart
Sketch
Original
Zero-shot
CLIP
Ours
Figure Visualization of Attention Maps. From top to bottom: original image, Zero-shot CLIP attention, and unlearned model attention by our method, where the domain to be forgotten real. Zero-shot CLIP attention concentrates on objects regardless of domains, demonstrating its generalizability across various domains. Our method successfully distracts the model's attention from the semantic regions of the forgetting data, while preserving for the remaining data.
prediction sensitivity for data from domains to be forgotten while preserving or enhancing sensitivity for data from domains to be memorized, enabling the model to effectively forget unwanted domain information while maintaining high accuracy on the retained domains.
Limitations
Our method performs Approximate Domain Unlearning (ADU) under the assumption that domain labels are available for training samples. While this setting enables explicit and fine-grained control over domain-specine forgetting, it may not always hold in real-world scenarios, where such comprehensive domain information is often incomplete or unavailable. Nevertheless, this limitation may be mitigated by integrating domain estimation techniques, which estimate domain clusters without requiring prior knowledge the domains. Indeed, we present additional experiments in Ap- pendix C.3 where simulate missing domain labels and evaluate the effectiveness of incorporating domain estimation through simple pseudo-labeling technique. The results show that, even when large fraction of domain labels is absent, our method maintains substantially better performance with domain estimation than without it. This suggests that combining ADU with more advanced domain estimation techniques (e.g.. [Mitsuzumi al. 2021]) provides a practical path toward handling more realistic settings with incomplete domain information. Although such integration lies beyond the scope of this work, exploring this direction offers a promising avenue for realizing ADU under more challenging conditions.
6 Conclusions
We introduced Approximate Domain Unlearning (ADU), a brand new variant of approximate un- learning that prevents pre-trained Vision- LssionLLanguage Models (VLMs) from recognizing only data in specified domains. We also devised a novel solution to this task based on the idea of allowing domain unlearning by disentangling the strong domain generalizability of pre-trained VLMs. Experimental results showed that our method outperformed the dedicated baselines. We believe this paper opens up new research direction approximate unlearning and provides challenge to the community.
7 Acknowledgements
This paper is partially based on results obtained from a project, JPNP25006, commissioned by the New Energy and Industrial Technology Development Organization (NEDO).
10

---

References
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
Lucas Bourtoule, Varun Chandrasekaran, Christopher Choquette-
Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In Proceedings of the IEEE Symposium on Security and Privacy (SP), 2021.
Jonathan Brophy and Daniel Lowd. Machine unlearning for random forests. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In Proceedings of the IEEE Symposium on Security and Privacy (SP), 2015.
Min Chen, Weizhuo Gao, Gaoyang Liu, Kai Peng, and Chen Wang. Boundary unlearning: Rapid forgetting deep networks via shifting the decision boundary. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
Yuantao Chen, Jie Xiong, Weihong Xu, and Jingwen Zuo. A novel online incremental and decremental learning algorithm based on variable support vector machine. Cluster Computing, 22:7435-744 7445, 2019.
Junhyeong Cho, Gilhyun Nam, Sungyeon Kim, Hunmin Yang. and Suha Kwak. Promptstyler: Prompt-driven style generation for source-free domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition (CVPR), 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x16 16 words: Transformers for image recognition at scale. In Proceedings of the International Conference Learning Representations (ICLR), 2021.
Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, and Sijia Liu. Salun: Em- powering machine unlearning via gradient-based weight saliency in both image classification and generation. In Proceedings of the International Conference Learning Representations (ICLR), 2024.
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security (CCS), 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Dovain-adversarial training of neural networks. Journal of Machine Learning Research (JMLR), 17(1):2096-200 2030. 2016.
Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2019.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Aditya Golatkar, Alessandro Achille, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. privacy forgetting deep networks. Proceedings of the IEFF/CVF conference on computer vision and pattern recognition (CVPR), 2021.

---

Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2021.
Arthur Gretton, Karsten M Borgwardt, Malte] J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research (JMLR), 13(1):723-773, 2012.
Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. In Proceedings of the International Conference on Machine Learning (ICML), 2020.
Yunshi Huang, Fereshteh Shakeri, Jose Dolz, Malik Boudiaf, Houda Bahig, and Ismail Ben Ayed. Lp++: A surprisingly strong linear probe for few-shot clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024a.
Zhehao Huang, Xinwen Cheng, JingHao Zheng, Haoran Wang, Zhengbao He, Tao Li, and Xiaolin Huang. Unified gradient-based machine unlearning with remain geometry enhancement. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2024b.
Zachary 1zzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), 2021.
Won Young Jhoo and Jae-Pil Heo. Collaborative learning with disentangled features for zero-shot domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the International Conference Machine Learning (ICML), 2021.
Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsity can simplify machine unlearning. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2023.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
Yusuke Kuwana, Yuta Goto, Takashi Shibata, and Go Irie. Black-box forgetting. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2024.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta-learning for domain generalization. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2018a.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2018b.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the International Conference on Machine Learning (ICML), 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 22ip-2: Bootstrapping language-iman -mmge pre- training with frozen image encoders and large language models. In Proceedings of the International Conference on Machine Learning (ICML), 2023.
Ming Li, Jike Zhong, Chenxin Li, Liuzhuozheng Li, Nie Lin, and Masashi Sugiyama. Vision- language model fine-tuning via simple parameter-efficient modification. Proceedings of the Conference Empirical Methods in Natural Language Processing (EMNLP), 2024.
12

---

Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks, In Proceedings of the International Conference on Machine Learning (ICMI), 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In Proceedings of the International Conference on Machine Learning (ICML), 2017.
Djebril Mekhazni, Amran Bhuiyan, George Ekladious, and Eric Granger. Unsupervised domain adaptation in the dissimilarity space for person re-identification, In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Yu Mitsuzumi, Go Irie, Daiki Ikami, and Takashi Shibata. Generalized domain adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021.
Seth Neel, Aaron Roth, and Saeed Sharifi- -haljir-Malvajerdi.. Descent-to-delete: Gradient-bart bbaed methods for machine unlearning. In Proceedings of the 31st International Conference on Algorithmic Learning Theory (ALT), 2021.
Vishal M Patel, Raghuraman Gopalan, Ruonan Li, and Rama Chellappa. Visual domain adaptation: A survey of recent advances. IEEE Signal Processing Magazine, 32(3):53-69 69, 2015.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2021.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza- tion. In Proceedings of the International Conference on Computer Vision (ICCV), 2017.
Thanveer Shaik, Xiaohui Tao, Haoran Xie, Lin Li, Xiaofeng Zhu, and Qing Li. Exploring the landscape of machine unlearning: A survey and taxonomy. IEEE Transactions on Neural Networks and Learning Systems (TNNLS), pages 1-21, 2024.
Takashi Shibata, Go Irie, Daiki Ikami, and Mitsuzumi. Learning with selective forgetting. In Proceedings of the International Joint Conference Artificial Intelligence (IJCAI), 2021.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In Proceedings of the IEEE Symposium Security and Privacy (SP), 2017.
Ting Shu, Shalini Mo, Arjun Ramaswamy, Xinlei Zhao, Devi Parikh, and Marcus Rohrbach. Test- time prompt tuning for zero-shot generalization vision-language models. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2022.
Nan Sun, Ning Wang, Zhigang Wang, Jie Nie, Zhiqiang Wei, Peishun Liu, Xiaodong Wang, and Haipeng Qu. Lazy machine unlearning strategy for random forests. In Proceedings of the International Conference on Web Information Systems and Applications, 2023.
Ayush K Tarun, Vikram S Chundawat, Murari Mandal, and Mohan Kankanhalli. Fast yet effective machine unlearning. IEFE Transactions on Neural Networks and Learning Systems (TNNLS), 2023.
13

---

Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Under- standing factors influencing machine unlearning, In Proceedings of the IEEE European Symposium on Security and Privacy, 2022a.
Anvith Thudi, Hengrui Jia, Ili Shumailov, and Nicolas Papernot. On the necessity of auditable algorithmic definitions for machine unlearning, In Proceedings of USENIX security symposium (USENIX Security), 2022b.
Enayat Ullah, Tung Mai, Anup Rao, Ryan A Rossi, and Raman Arora. Machine unlearning via algorithmic stability. In Proceedings of the Conference on Learning Theory (CoLT), 2021.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Haohan Wang. Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2019.
Liyuan Wang, Yan Jin, Zhen Chen, Jinlin Wu, Mengke Li, Yang Lu, and Hanzi Wang. Transitive vision-language prompt learning for domain generalization. arXiv preprint arXiv:240.11755, 2024.
Wei Wang, Haojie Li, Zhengming Ding, and Zhihui Wang. Rethink maximum mean discrepancy for domain adaptation. arXiv preprint arXiv:2007.00689 2020.
Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine unlearning of features and labels. In Proceedings of the Anmual Network and Distributed System Security Symposium, 2023.
Yanan Wu, Zhixiang Chi, Yang Wang, Konstantinos N Plataniotis, and Songhe Feng. Test-time domain adaptation by learning domain-aware batch normalization. In Proceedings of the AAAI Conference Artificial Intelligence (AAAI). 2024.
Jie Xu, Zihan Wu, Cong Wang, and Xiaohua Jia. Machine unlearning: Solutions and challenges. IEFE Transactions on Emerging Topics in Computational Intelligence, 2024.
Hongliang Yan, Yukang Ding, Peihua Li, Qilong Wang, Yong Xu, and Wangmeng Zuo. Mind the class weight bias: Weighted maximum mean discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Jingwen Ye, Yifang Fu, Jie Song, Xingyi Yang, Songhua Liu, Xin Jin, Mingli Song, and Xinchao Wang. Learning with recoverable forgetting. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini. and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research (TMLR), 2022.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), 2023.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain adaptive ensemble learning. IEEE Transactions on Image Processing (TIP), 30:8008 8018, 2021.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 4564):439-44455 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision- language models. In Proceedings of the International Conference on Computer Vision (ICCV), 2022b.
14

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction accurately describe the proposed problem setting, method, and experimental scope, which align with the content presented in the paper (Sec. and Sec. 4
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
15

---

Justification: Sec. 5 discusses the limitations of our method, noting that it is sometimes difficult to obtain sufficient information suchh as domain labels, images from all domains, and knowledge of the domain set-to achieve Approximate Domain Unlearning (ADU).
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our paper does not present formal theoretical results with assumptions and proofs. Instead, we provide empirical evidence-suee as performance metrics and t-SNE visualizations support our claims about domain entanglement and the limitations of conventional unlearning methods.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
16

---

Answer: [Yes]
Justification: The paper describes the experimental setup, datasets, evaluation metrics, network architecture, training specifics, and hyperparameter details in Sec. and Sec. 4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the code and instructions in the supplementary material, and is intended to be open open-sourced upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https! for more details.
//nips //npps.ccppubbll http:cccppppccc/tettette.e..cclt http://lllggguude/eeeoee /tt.uudeeeedddeuuuuissisoooolooo
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
17

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental settings, including network architecture, optimizer, data splits, iterations, and hyperparameters are detailed in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We do report error bars other appropriate information about the statistical significance of the experiment. Instead, tables I 2 and the appendix tables report results as the mean over three runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
18

---

Answer: [No]
Justification: Our method is computationally efficient and can be implemented using single RTX A4000 16GB GPU. However, specific computational resource details are not included in the main text or appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttts::/neurips.ccco http://rrripp.ccouuuuiii////aaaa
Answer: [Yes]
Justification: This research proposes novel problem setting and its corresponding approach. It does not involve human subjects or ethically sensitive applications, and we assume it aligns with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: While the targeted application areas (e.g.. autonomous driving) imply po- tential positive impact, this is not explicitly elaborated, nor potential negative societal consequences discussed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
19

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Neither the model nor the standard benchmark data used appear to pose high risk for misuse necessitating specifie release safeguards beyond standard open-source practices.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper properly cites the sources for existing assets like baseline methods and datasets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pawerssipppsiiiiie... thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The primary new asset is the source code for the proposed method. It is provided in the supplementary material and intended to be open-sourced.
20

---

Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the datasct/codc/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to amonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The research does not involve crowdsourcing experiments research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The research does not involve human subjects, therefore IRB approval is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The research does not incorporate any LLM as a key element.
21

---

Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy http:///aaarrpssccoooooooererr/e for what should or should not be described.
22

---

A Results on DomainNet
We evaluated our method on DomainNet [Png et 2019]. an established multi-domain benchmark dataset that comprises 177K test samples across six domains and 345 classes, making it one of the most complex and large-scale datasets of its kind. The results are shown in Table Ours is clearly better than Baseline, which emphasizes the strong effectiveness of our method.
Table 5: Results on DomainNet.
DDogel
=
Diagel
= 2
Diorget
=
Method
H Memt
Fort
H t
Memt
Fort
H Memt Fort
Zero-shot CLIP 36.85 53.28 28.16 36.85 53.28 28.16 36.85 53.28 28.16
Baseline
38.45
55.69
29.36
39.36
55.81 30.40 38.79
55.26 29.88
Ours
66.81
58.86
77.23
67.81
58.89 79.90
68.83
59.06 82.48
B Per-Domain Accuracy
We report per-domain accuracy Office-Home, Mini DomainNet, and DomainNet, which include domains with substantial visual similarity (e.g., sketch vs. quickdraw). The results are shown in Tables 6 7 and 8 Overall, they demonstrate that method can selectively suppress classification accuracy on the forgotten domain while maintaining performance on the others, even when domains share similar visual characteristics. For instance, in Mini DomainNet, forgetting sketch reduces its accuracy significantly from 72.54% to 20.64%, whereas visually similar domains such as clipart and painting only minimally affected (within 1%). These results indicate that method offers fine-grained control even under significant domain overlap.
We found that unlearning real in Office-Home, our method decreases the accuracy from 81 81.29% to 55.48%. Although this is substantially lower than the original zero-shot CLIP, the model still retains moderate recognition ability. One possible reason is that CLIP heavily pre-trained on image-text pairs dominated by real- world photos, making the real domain more strongly encoded and thus harder to forget. Nonetheless, as shown in Tf the main paper, our method consistently achieves strong forgetting performance on average
be more difficult for certain domains. For example, when forgetting
datasets. These results suggest that while approach is broadly effective, slight variations may arise depending on the VLM's pretraining bias; mitigating dataset bias could serve as potential remedy.
Table 6: Per-Domain Accuracy on Office-Home.
Method
art
clipart
product
real
Zero-shot CLIP
74.34
60.97
80.43
81.29
Ours (D larget {art])
39.25
70.18
88.72
75.91
Ours (D Rarget clipart))
77.36
15.13
87.87
80.00
Ours (D larget producr))
80.38
72.15
32.77
77.85
Ours (D (Dinget real))
68.30
67.98
78.94
55.48
Table 7: Per-Domain Accuracy on Mini DomainNet.
Method
clipart
painting
real
sketch
Zero-shot CLIP
80.64
78.10
87.94
72.54
Ours (D Piogge (clipart))
24.76
75.40
83.97
73.97
Ours (D Dooggtt- = (painting!)
81.59
30.32
84.92
73.97
Ours (D forger (real))
77.94
69.52
32.06
74.44
Ours Doogett (sketch))
79.05
77.62
87.46
20.64
23

---

Table 8: Per-Domain Accuracy on DomainNet.
Method
clipart
infograph
painting
quickdraw
real
sketch
Zero-shot CLIP
71.84
50.01
65.36
14.91
83,42
63.07
Ours (D torget cclipart))
33.37
51.55
67.72
28.93
79.87
62.22
Ours (D torget infograph))
71.84
13.05
67.77
32.07
79.50
63.55
Ours (D forget painningg)
73.59
52.21
22.77
29.64
79.86
63.21
Ours forget quick)rraw)
73.60
51.33
67.99
7.01
81.79
63.79
Ours D forget real])
71.61
50.52
64.40
29.56
35,89
64.05
Ours (D torget (sketch))
70.81
51.62
67.20
29.62
81.26
24.51
Table 9: Robustness to Domain Imbalance on Office-Home.
Doogee
[Droggt]]
2
Droope = 3
Number of shots for selected domains
Men't
Fort
H
Men't Fort H f
Mem't Fort
8 shots
69.96 77.93
64.34
73.58 75.61 71.89 75.89 72.15 80.77
4 shots
68.23
82.24
59.55
71.24
81.21
63.82
74.34
74.61 74.08
shot
66.89 77.64 59.61 63.80
78.16
56.53
68.75
64.31 73.85
C Robustness to More Complex Scenarios
In addition to the standard settings, we further evaluate the robustness of our method under more complex and realistic scenarios that may arise practice. Specifically, we consider three challenging conditions: (i) domain imbalance, where some domains contain far fewer samples than others; (ii) partial domain-class overlap, where certain classes appear only in a subset of domains; and (iii) partial domain labels, where domain annotations are missing for a portion of the training samples. These experiments aim to validate whether our approach remains effective handling such imbalanced, heterogencous, and incomplete settings.
C.1 Robustness to Domain Imbalance
Domain imbalance can arise in practical scenarios, where certain domains have substantially fewer samples than others (e.g., abundant real images vs. sparse driving). To assess the robustness of our DDL under such imbalance, we conducted experiments on Office-Home by reducing the number of training samples from selected domains (art and clipart), while keeping the other domains fixed. Our default setting uses eight shots, and we additionally tested reduced settings with four or one shot(s).
clipart images in autonomous
As shown in Table 9 our method maintains stable performance even with severe imbalance. Remark- ably. with only single sample from the art and elipart domains, the model still achieves competitive performance. These results indicate that our method can still perform effective domain disentangling and selective forgetting even in highly imbalanced scenarios.
C.2 Robustness to Partial Domain-Class Overlap
We also investigated scenario where domain-class distributions are partially overlapping, Specif- ically, in Office-Home removed samples from three random classes only in the and clipart domains, thereby creating a setting where some classes are missing in certain domains. As presented Table I0 our method retains competitive performance under this condition. These results suggest that our method can robustly handle realistic domain-class imbalances without critical loss in effectiveness.
C.3 Robustness to Partial Domain Labels
As we discuss in Sec. 5 we acknowledge that domain labels may not always be available for all training samples in real-world scenarios. However, it is important to note that most standard formulations in domain adaptation and domain generalization (c.g., (Wu al. 2024 Cho et al. 2023

---

Table 10: Robustness to Partial Domain-Clan Class Overlap on Office-Home.
Diorges
[Dioget
Dioneer
=3
Setting
H 1 Memt Fort H 1 Memt
Fort
H
Ment Fort
Original Dataset
69.96 77.93 64.34 73.58 75.61
71.89
75.89
72.15 80.77
Dataset w/ Partial Domain Class Overlap 67.93 81.81 58.69
72.03 79.61 66.24 74.51
75.12 73.90
Table 11: Robustness to Partial Domain Labels on Office-Home. Without domain estimation, the performance drops sharply the proportion of unlabeled samples increases. In contrast, with domain estimation, the performance remains substantially higher,
w/o Domain Estimation w/ Domain Estimation
Unlabeled sample ratio
H f Memt
Fort
I
Menct
Fort
0.0
75.89
72.15
80.77
75.89
72.15
80.77
0.3
63.38
83.35
52.68
65.81
83.06
55.98
55.77
84.63
43.66
63.64
84.69
52.20
49.97
83.84
36.98
61.21
83.37
50.00
Thoo and Heo. ))21]) assume full access to domain labels during training. Therefore, our setting aligns with this widely accepted convention and should not be considered unrealistic simplification. Meanwhile, to evaluate the performance of our method under partial domain label availability, we consider setting which portion of the training samples (ranging from 30% 70%) lack domain annotations. To assess the effectiveness of combining domain estimation with our method in such cases, we adopt a simple sreudo-labeling approach: domain classifier is trained using only the samples with known domain labels and then used to assign pseudo labels to the unlabeled samples. As shown in Table []] without domain estimation, the performance drops sharply as the proportion of unlabeled samples increases. In contrast, with domain estimation, the performance remains substantially higher, even under 70% missing domain labels. These results support our claim that our approach can be extended to more realistic cases where only partial domain annotations are available.
D Computational Complexity
Table 12 summarizes GPU memory usage and training time with an NVIDIA RTX A4000 GPU on Office-Home. While our method incurs only slightly higher computational cost than lightweight baselines [Zhou al., [022b] Huang et al., (2024). it remains on par with advanced CLIP fine-tuning methods Khattak et al. 2023 Li et al. [024]. These results prove that our method is sufficiently efficient for practical use. Notably, adding InstaPG increases memory usage by only GB and training time by less than minute, indicating minimal overhead.
E
Additional Analysis
Beyond the main experiments, we conduct a series additional analyses to further examine the robustness, sensitivity, generality, and real-world applicability of our method. Specifically, in- vestigate (i) ablation studies of the loss functions; (ii) sensitivity to the kernel choice in MMD; (iii) sensitivity to prompt depth; (iv) performance with different pre-trained VLM; (v) comparison between prompt tuning and parameter tuning approaches; (vi) accuracy over training; and (vii) perfor- mance on vehicle-related data. Together, these analyses provide more comprehensive understanding of the factors influencing the effectiveness of our method and its practical applicability.
F.1 Ablation Study of Loss Functions
As described in Eq. O of the main paper, total loss combines three terms: Lmemorizas Liorget. and Ldomains which are jointly optimized balance memorization, forgetting, and domain separation. To gain deeper insights into their individual roles, conduct ablation studies where each loss function
25

---

Table 12: Computational Complexity on Office-Home.
Method
Memory [GB]
Time
CoOp Zhou al. 2022b
1.9
283
MaPLe Khattak et al 2023
10.4
539
BBF
Kuwana et al 2024
2.5
2882
CLIPFit
L1 al.
11.0
340
LP++ Huang et 2024a
3.8
35
Ours w/o InstaPG
9.7
501
Ours
10.7
550
Table 13: Ablation Study of Loss functions. We evaluate the individual and combined impact of memmrrize Liorget and Ldmmain on memorization (Mem). forgetting (For), and the overall balance metric H. The results show that each loss function contributes to different aspects of performance, and their combination achieves the best trade-off.
[Paage
Dioges
Deonge 3
Lunemotize Ltorget Ldomain
H 1 Mem For 1
H
Mem 1 For H
Mem f For
27.52
86.22
16.92
30.11
86.60
18.36
36.15
86.20
22.88
d
5.37
2.76
97.36
4.58
2.35
97.70
4.92
2.53
97.52
v
60.82
74.51
51.72
60.01
76.22
49.94
63.12
77.74
54.57
V
52.59
79.96
39.88
54.19
80.23
41.23
59.47
80.96
48.79
v
48.64
81.41
38.52
57.18
79.92
45.98
65.03
79.17
55,46
V
v
2.74
1.39
98.83
2.77
1.41
98.54
2.68
1.36
98.37
69.96
77.93
64.34
73.58
75.61
71.89
75.89
72.15
80.77
applied separately or in combination. This analysis allows us to disentangle their contributions to different aspects of performance and verify that the full combination provides the best trade-off.
Table presents the results. The loss function Ammmorize predominantly enhances memoriza- tion performance (Mem), while Ctorget and Ldomain mainly contribute to forgetting performance (For). Combining all three loss functions yields the best overall balance between memorization and forgetting, as reflected by the harmonic mean H.
E.2 Choice of Kernel for MMD
Since our DDL based on MMD, its effectiveness could principle sensitive to choice of kernel. To assess sensitivity of our method to kernel choice, tested four different kernels for MMD on Office-Home, as shown in Table The results show that the linear, Laplacian, and REF kernels yield generally stable performance, with only modest differences across metrics, while the polynomial kernel performs notably worse. Given these observations, we adopt the RBF kernel in the main experiments, but when validation-based kemel selection feasible, any of the three stable kernels can be chosen; otherwise, the linear kernel provides a conservative and reliable default.
E.3 Sensitivity Prompt Depth
The depth which prompts are inserted is an important design choice, as it determines how much information from intermediate representations is influenced by the prompts. To examine the sensitivity of our method to this factor, vary the insertion layer of the InstaPG, while ensuring that the standard vision prompt is applied up to the same layer.
As shown in Fig. 7 the performance remains stable across different depths. This indicates that our method is robust the choice of insertion point, reducing the need for extensive hyperparameter tuning and simplifying practical deployment.
26

---

Table 14: Impact Kernel Choice on Office-Home.
Drogget
Diorgel
[Diogel
3
Kernel
H Mem't
Fort
H
Mem
Fort
H
Memt
Fort
Linear
66.83 79.85 58.43 69.32 77.35 63.46 73.91 74.02 74.50
Laplacian
68.50 79.00 61.44 71.18
75.93 67.17 73.66 71.69 76.54
Polynomial
36.53 23.29 84.68 49.73
41.79
61.40
53.65
44.49 67.56
RBF
69.96 77.93 64.34
73.58
75.61
71.89
75.89
72.15 80.77
60
80
40
40
20
20
6 Prompt depth
2
10
3
10
Prompt depth
Promot depth
(a) Home
(b) Mini DomainNet
(c) ImageNet
Figure 7: Sensitivity to Propmt Depth. We report the performance when varying the depth at which the InstaPG inserted. The standard vision prompt is also applied up the same layer. The stable results across depths demonstrate the robustness of our method to the choice of insertion point.
E.4 Performance with Different Pre-trained VLMs
We also verify the generality of our method beyond CLIP by evaluating it on another pre-trained vision-language model, SigI.IP [Zhai et al. 2023), using ImageNet. As shown in Table our method outperforms Baseline, confirming that it is effective even when applied to different backbone. This result suggests that our approach is not tied CLIP and can be extended other pre-trained VLMs.
E.5
Prompt Tuning vs. Parameter Tuning
Prompt tuning with frozen model parameters has become widely adopted alternative full fine- tuning, as prior studies have shown that updating all parameters of CLIP often leads to catastrophic forgetting, especially in few-shot regimes [Zhou al. 2022b Kumar al. 20022.. One possible concern, however, is that updating only the vision prompts might limit the ability to sufficiently alter internal representations if domain-specific features are already entangled in early layers. To examine this, we also evaluate setting where the model parameters are updated alongside the prompts. As shown in Table [6] this setting results in severe catastrophic forgetting, whereas prompt tuning preserves much better balance between memorization and forgetting. These findings support prompt tuning as a more reliable strategy for domain unlearning.
E.6 Accuracy over Training Iterations
To better understand the learning dynamics of our method, we track how the memorization (Mem) and forgetting (For) accuracies evolve during training. Fig. shows the results on the Office-Home dataset. We see that Mem drops in the early stages while For increases rapidly while suggesting a conflict between the two. However, both scores improve steadily after this phase, indicating that the model gradually learns balance forgetting and retention. These results suggest a two-stage process: initial domain disentanglement followed by controlled forgetting and memorization.
27

---

Table 15: Performance with SigLIP on ImageNet.
Method
H 1
Menr
Fort
Zero-shot
46.47
63.26
36.73
Baseline
48,99
60.32
41.23
Ours
64.97
48.42
98.71
Table 16: Prompt Tuning vs. Parameter Tuning on Office-Home.
[Piogel
Diorgll
Diogal
=
Tuning Approach
H 1 Memt
Fort
H 1
Ment Fort
H
Mem Fort
Parameter Tuning
3.90
2.00
98.31
2.95
1.50
98.52
3.64
1.86 98.26
Prompt Tuning (Ours)
66.81
58.86
77.23
67.81
58.89
79.90
68.83
59.06 82.48
100
Accuracy
40
20
Mern
200
500
800
Iterations
Figure 8: Mem and For over Training Iterations Office-Homer
E.7 Proxy Evaluation for Autonomous Driving Scenarios
To examine the practical applicability of our method in autonomous driving scenarios, we design proxy experiment that simulates domain discrepancies commonly encountered in such systems; in real-world driving environments, vehicles may appear not only as physical objects but also as illustrations, for example roadside advertisements and billboards (see Sec. DT. While no public autonomous driving dataset currently provides annotated examples of such illustrated vehicles, we emulate this situation using the Mini DomainNet dataset. Specifically, we select seven vehicle-related categories, namely, "bus", "car", "motorbike", "pickup truck", "police car", "school bus", and "tractor", and evaluate our method in a setting where the model unlearned retain only the real domain while forgetting the other illustration domains, i.e., clipart, painting, and sketch.
The results are shown in Table 17
Our method effectively preserves recognition of real vehicles, while substantially suppressing classification on illustrated ones. These results suggest that our approach holds promise for practical applications such as autonomous driving or car counting.
We acknowledge that full-scale validation on real-world deployment tasks remains an important direction for future work. Nonetheless, believe findings serve informative first step toward domain-specific unlearning under realistic constraints.
F Instance-level Diversity within Domain
We present example images from the art domain Office-Home, Even within the same domain, the visual styles vary significantly from image to image. For instance, (a) appears to be a mix of photo and artwork. Images (b), (c), and (d) resemble typical art-style images, while (e) and (f) look more like clipart or cartoons. In contrast, (g) and (h) resemble sketch. This diversity in style within single domain motivates the use of the Instance-wis wise Prompt Generator (InstaPG), which generates tailored prompts for each individual image.
28

---

Table 17: Per-Domain Accuracy on Vehicle-Related Classes in Mini DomainNet.
Per-domain accuracy
clipart +
painting +
real
sketch
Before unlearning
80,64
78.10
87.94
72.54
Ours w/o DDL and InstaPG
31.27
34.29
82.54
16.03
Ours
22.38
21.43
84.44
14.29
3
5
(a)
(b)
(c)
(d)
Rainbou Oash Alaco Block.
3
4
(e)
(f)
g
(h)
Figure 9: Instance-level Diversity within Art Domain of Office-Home. Sample images from the art domain exhibit substantial style variation, ranging from artwork to sketch. This intra-domain diversity motivates the use of the InstaPG, which generates image-specific prompts better handle such variability.
29


Reviews:
[{"review_id": "BzITUuLVKo", "reviewer": "Reviewer_gH1G", "summary": "This paper introduces a new machine unlearning framework for Vision-Language Models. Unlike traditional class-level unlearning that removes specific object categories, this work selectively removes recognition capability for images from certain domains (e.g., illustrations, sketches) while preserving performance on other domains (e.g., real photos). The authors propose two key technical components: Domain Disentangling Loss to separate domain distributions in feature space, and Instance-wise Prompt Generator to handle within-domain style variations.", "questions": "Why not include additional metrics like domain classification accuracy after unlearning, or measures of knowledge retention for related but not-forgotten content?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Pros:\n\n1. The problem is genuinely innovative and addresses practical limitations of existing unlearning approaches. The autonomous driving example (distinguishing real cars from illustrated advertisements) is compelling and highlights real-world relevance.\n\n2. This work effectively tackles the core challenge of domain entanglement in VLMs by combining cross-entropy loss with MMD maximization. It intelligently handles intra-domain variability through attention-based prompt generation. The combination addresses both macro-level (domain separation) and micro-level (instance variation) challenges.\n\n3. Extensive evaluation on three multi-domain datasets with different numbers of forgotten domains. Strong baselines adapted from state-of-the-art VLM fine-tuning and unlearning methods. And thorough ablation studies demonstrating the contribution of each component.\n\nCons:\n\n1. Experiments use only 8 samples per domain for training, which seems quite small. Evaluation limited to relatively simple multi-domain datasets. Unclear how the approach scales to more complex domain distributions or larger datasets\n\n2. The paper lacks discussion of scenarios where the method might fail or perform poorly. What happens when domain boundaries are ambiguous or when domains share significant visual similarity?\n\n3. No detailed analysis of computational overhead, training time, or memory requirements compared to baseline methods.", "was_revised": true, "final_justification": "Applying machine unlearning to domain generalization is well-motivated and novel. My major concerns like few-shot training examples and failure analysis have been adequately addressed.", "rebuttal": "We sincerely thank Reviewer gH1G for their thoughtful and constructive review. We are especially grateful that the reviewer **clearly acknowledged all the main contributions of our paper**, including the novelty of the problem setting, the effectiveness of our proposed solution, and the comprehensiveness of our experiments. We believe that our paper makes the first attempt at approximate domain unlearning in Vision-Language Models and can meaningfully contribute to the community by addressing key challenges of this emerging task. Below, we provide point-by-point responses to all questions and concerns raised. All the comments are constructive and help strengthen our work. We also appreciate that none of the points raised indicate fundamental flaws in our core ideas or methodology.\n\n### **Q1-1. 8 training samples per domain quite small.**\n\nTuning of VLMs is typically performed under a few-shot protocol [a-d], and recent work on unlearning for VLMs [e] also adopts this standard setting. Accordingly, we follow the same protocol. As shown in Appendix A.1, we report results for varying numbers of samples (from 1 to 32), where our method consistently and significantly outperforms the baseline across all cases.\n\n### **Q1-2. Results on more complex domain distributions or larger datasets?**\n\nWe additionally evaluated our method on DomainNet, an established multi-domain benchmark dataset that comprises 177K test samples over six domains and 345 classes, making it one of the complex and large-scale datasets of its kind. The results are shown in Table A-1. Ours is clearly better than the baseline (the best competitor compared with our method in our main paper), which emphasizes the strong effectiveness of our method.\n\n*Table A-1. Results on DomainNet.*\n\n| | | $\\|D_{for}\\|\\=1$ | | | $\\|D_{for}\\|\\=2$ | | | $\\|D_{for}\\|\\=3$ | |\n|:----|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n|Method|$H\\uparrow$|$Mem\\uparrow$|$For\\uparrow$|$H\\uparrow$|$Mem\\uparrow$|$For\\uparrow$|$H\\uparrow$|$Mem\\uparrow$|$For\\uparrow$|\n|Zero-shot CLIP|36.85|53.28|28.16|36.85|53.28|28.16|36.85|53.28|28.16|\n|Baseline|38.45|55.69|29.36|39.36|55.81|30.40|38.79|55.26|29.88|\n|Ours|**66.81**|**58.86**|**77.23**|**67.81**|**58.89**|**79.90**|**68.83**|**59.06**|**82.48**|\n\n### **Q2. Discussion on ambiguous or similar domains, with a note on failure cases.**\n\nTo the best of our knowledge, there is currently no established multi-domain benchmark datasets having ambiguous domain boundaries. Instead, we report per-domain accuracy on Office-Home, Mini DomainNet, and DomainNet, those which include domains with substantial visual similarity (e.g., \"sketch\" vs. \"quickdraw\"). \n\nThe results are shown in Tables A-2 to A-4 below. Overall, they demonstrate that our method can selectively suppress classification accuracy on the forgotten domain while maintaining performance on the others, even those with similar visual properties. For instance, in Mini DomainNet, forgetting \"sketch\" reduces its accuracy significantly from 72.54% to 20.64%, whereas similar domains such as \"clipart\" and \"painting\" experience only minor drops (within 1%). These results indicate that our method provides fine-grained control even in the presence of overlapping domain features.\n\nRegarding a failure case, we found that unlearning can be more difficult for certain domains. For example, when forgetting \"real\" in Office-Home, our method decreases the accuracy from 81.29% to 55.48%, which, although significantly lower than the original zero-shot CLIP, still retains moderate recognition ability. One possible reason is that CLIP is extensively pre-trained on image-text pairs dominated by real-world photos, making the \"real\" domain more strongly encoded and harder to forget. Nonetheless, as shown in Table 1 of the main paper, our method consistently achieves strong forgetting performance on average across datasets. These results suggest that although our approach is generally effective, slight variations may occur depending on the VLM’s pretraining bias; as a potential remedy, one may consider incorporating techniques for mitigating dataset bias.\n\n*Table A-2. Per-domain Accuracy on Office-Home.*\n\n||art|clipart|product|real|\n|:---|:---:|:---:|:---:|:---:|\n|Zero-shot CLIP|74.34|60.97|80.43|81.29|\n|Ours (Forgotten = art)|**39.25**|70.18|88.72|75.91|\n|Ours (Forgotten = clipart)|77.36|**15.13**|87.87|80.00|\n|Ours (Forgotten = product)|80.38|72.15|**32.77**|77.85|\n|Ours (Forgotten = real)|68.30|67.98|78.94|**55.48**|\n\n*Table A-3. Per-domain Accuracy on Mini DomainNet.*\n\n||clipart|painting|real|sketch|\n|:---|:---:|:---:|:---:|:---:|\n|Zero-shot CLIP|80.64|78.10|87.94|72.54|\n|Ours (Forgotten = clipart)|**24.76**|75.40|83.97|73.97|\n|Ours (Forgotten = painting)|81.59|**30.32**|84.92|73.97|\n|Ours (Forgotten = real)|77.94|69.52|**32.06**|74.44|\n|Ours (Forgotten = sketch)|79.05|77.62|87.46|**20.64**|\n\n*Table A-4. Per-domain Accuracy on DomainNet.*\n\n||clipart |infograph |painting |quickdraw |real |sketch|\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|\n|Zero-shot CLIP|71.84|50.01|65.36|14.91|83.42|63.07|\n|Ours (Forgotten = clipart)|**33.37**|51.55|67.72|28.93|79.87|62.22|\n|Ours (Forgotten = infograph)|71.84|**13.05**|67.77|32.07|79.50|63.55|\n|Ours (Forgotten = painting)|73.59|52.21|**22.77**|29.64|79.86|63.21|\n|Ours (Forgotten = quickdraw)|73.60|51.33|67.99|**7.01**|81.79|63.79|\n|Ours (Forgotten = real)|71.61|50.52|64.40|29.56|**35.89**|64.05|\n|Ours (Forgotten = sketch)|70.81|51.62|67.20|29.62|81.26|**24.51**|\n\n\n### **Q3. Computational complexity?**\n\nTable A-5 below summarizes GPU memory usage and training time with NVIDIA RTX A4000 GPU on Office-Home. While our method incurs only slightly higher computational cost than lightweight baselines such as CoOp and LP++, it remains on par with advanced CLIP fine-tuning methods such as MaPLe and CLIPFit. These results prove that our method is sufficiently efficient for practical use. Notably, adding InstaPG increases memory usage by only 1 GB and training time by just 1 minute, indicating minimal overhead.\n\n*Table A-5. Computational Complexity.*\n\n| |Memory [GB]|Time [s]|\n|:----|:----|:----|\n|CoOp [a]|1.9 | 283 |\n|MaPLe [b]| 10.4 | 539 |\n|BBF|2.5| 2882 |\n|CLIPFit| 11.0 | 340 |\n|LP++| 3.8 | 35 |\n|Ours w/o  InstaPG |9.7| 501 |\n|Ours |10.7| 550 |\n\n### **Q4. Why not report domain classification accuracy?**\n\nAccording to the reviewer's suggestion, we reported domain classification accuracy before and after unlearning on Office-Home in Table A-6. Accuracy improved significantly from 25.80% before unlearning to 79.43% after unlearning when using our full method. Furthermore, when our two key components, i.e., DDL and InstaPG, are removed, the domain classification accuracy drops substantially to 31.06%, indicating that our method plays a critical role in achieving effective domain separation. We believe these results provide strong additional evidence of the effectiveness of our method, particularly in facilitating domain unlearning. We are grateful to the reviewer for directing our attention to this important perspective!\n\n*Table A-6. Domain Classification Accuracy When \"Art\" is Forgotten.*\n\n||Domain Classification Accuracy|\n|:---|:---:|\n|Before Unlearning |25.80|\n|Ours w/o DDL and InstaPG|31.06|\n|Ours|**79.43**|\n\n---------------------------------------------\n**References**\n\n[a] Zhou et al., Learning to Prompt for Vision-Language Models, IJCV 2022.\n\n[b] Khattak et al., MaPLe: Multi-modal Prompt Learning, CVPR 2023.\n\n[c] Li et al., Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification, EMNLP 2024.\n\n[d] Huang et al., LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIP, CVPR 2024.\n\n[e] Kuwana et al., Black-box Forgetting, NeurIPS 2024."}, {"review_id": "fU5dpWXruY", "reviewer": "Reviewer_ZXGU", "summary": "This paper introduces the problem of Approximate Domain Unlearning (ADU), which aims to reduce recognition accuracy for images from specified domains (e.g., clip art) while preserving accuracy for other domains (e.g., real images). Compared to existing approximate class unlearning tasks, ADU can be considered as a finer-grain task as it needs to distinguish different domains within the same class. With this finer-grain task, there are new challenges. For example, domain distributions are highly entangled in the feature space (e.g. real car images vs. illustrated car images) due to the strong domain generalization capability of pre-trained VLMs, which makes domain unclearning difficult. This paper proposes two approaches to tackle the challenges: Domain Disentangling Loss (DDL) and Instance-wise Prompt Generator (InstaPG).  Domain disentangling loss explicitly disentangles domain distributions in the latent space. Instance-wise Prompt Generator adaptively models instance-level domain variations. \n\nExperiments and ablation studies are done on multi-domain image benchmark datasets, showing the effectiveness of the approach.", "questions": "1) In applications such as autonomous driving, data may be very unbalanced among different domains. For example, there would be a lot more real car images than illustrated car images (car on a poster).  Have you studied how sensitive domain disentangling loss is to such kind of data imbalance among domains?  \n2) As domain labels are not always readily available. How sensitive is your approach to noise in domain labels?\n3) Have you tried other pre-trained VLMs?\n4) Do you have any analysis on the computation cost, e.g., the increase with the use of InstaPG?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1) The paper proposes a new problem: Approximate Domain Unlearning (ADU).  This is useful for applications such as in self-driving, where car recognition would mean recognition of “real” cars on the road, instead of pictures of a car on a poster.\n2) The paper proposes two methods to alleviate the entanglement between different domain distributions in the latent feature space: Domain Disentangling Loss (DDL) and Instance-wise Prompt Generator (InstaPG). DDL is based on the idea that if the feature distributions of individual domains are well-separated, the domain labels can be accurately predicted, and vice versa. InstaPG is to account for  instance-level variation in the images. The proposed approaches are intuitive and seem to be effective, as shown via experiments. \n3) Experiments and ablation studies are done on multi-domain image benchmark datasets, including ImageNet, Office-Home, and Mini DomainNet. The experimental results demonstrated the effectiveness of the proposed approach. \n4) The paper is generally well written.\n  \n\nWeakness:\n1) One limitation of the proposed approach is that, for using the domain disentangling loss,  it assumes the availability of domain labels. As the authors pointed out, this may be a strong assumption, for example, domain labels are not readily available in autonomous driving data.\n2) In applications such as autonomous driving, data may be very unbalanced among different domains. For example, there would be a lot more “real” car data compared to illustrated car data. It would be interesting to see how domain disentangling loss works in those situations.\n3) The paper is based on a pre-trained CLIP model. It would be interesting to see results on other pre-trained VLMs.\n4) The Instance-wise Prompt Generator (InstaPG) increases the training and inference latency. It would be nice to have some analysis on the computational cost.", "was_revised": true, "final_justification": "The paper introduces a new problem: Approximate Domain Unlearning (ADU). It also proposes two approaches to tackle the challenges in ADU. The proposed approach seems to be effective.  \n\nMy questions were adequately answered in the rebuttal. Feedback from other reviewers is also generally positive. I would like to maintain my rating for acceptance.", "rebuttal": "We sincerely thank Reviewer ZXGU for their thoughtful and insightful review. We are especially grateful that the reviewer **clearly recognized all the key contributions of our work**, including the novelty of the Approximate Domain Unlearning task, the effectiveness of our proposed methods (DDL and InstaPG), and the clarity and comprehensiveness of our experiments. We believe this paper makes the first concrete attempt at Approximate Domain Unlearning for Vision-Language Models, and we are confident that our contributions will meaningfully benefit the community by highlighting important challenges and proposing practical solutions in this emerging area. Below, we provide detailed responses to each comment and question. We greatly appreciate that all feedback was constructive and aimed at further strengthening the contribution, without pointing out any fundamental flaws.\n\n### **Q1. Domain labels not always fully available.**\nAs we discuss in the Limitations section of our paper, we acknowledge that domain labels may not always be available for all training samples in real-world scenarios. However, it is important to note that most standard formulations in domain adaptation and domain generalization (e.g., [a–c]) assume full access to domain labels during training. Therefore, our setting aligns with this widely accepted convention and should not be considered an unrealistic simplification.\n\nMeanwhile, to evaluate the performance of our method under partial domain label availability, we consider a setting in which a portion of the training samples (ranging from 30% to 70%) lack domain annotations.\nTo assess the effectiveness of combining domain estimation with our method in such cases, we adopt a simple pseudo-labeling approach: a domain classifier is trained using only the samples with known domain labels and then used to assign pseudo labels to the unlabeled samples.\n\nAs shown in Table B-1, without domain estimation, the $H$-score drops from 75.89% to 49.97% under 70% missing domain labels. In contrast, with domain estimation, the $H$-score remains substantially higher at 61.21% even under the same missing rate. These results support our claim that our approach can be extended to more realistic cases where only partial domain annotations are available.\n\n*Table B-1. Performance on Office-Home with Partial Domain Labels.*\n\n| | |w/o Domain Estimation| | | w/ Domain Estimation| |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|Unlabeled sample ratio|$H\\uparrow$|$Mem\\uparrow$|$For\\uparrow$|$H\\uparrow$|$Mem\\uparrow$|$For\\uparrow$|\n|0|75.89|72.15|80.77|75.89|72.15|80.77|\n|0.3|**63.38**|83.35|52.68|**65.81**|83.06|55.98|\n|0.5|**55.77**|84.63|43.66|**63.64**|84.69|52.20|\n|0.7|**49.97**|83.84|36.98|**61.21**|83.37|50.00|\n\n### **Q2. Robustness of DDL to domain imbalance?**\nWe agree that domain imbalance can arise in practical scenarios, where some domains may contain fewer samples than others (e.g., abundant \"real\" images vs. sparse \"art\" or \"clipart\" images in autonomous driving). To evaluate the robustness of our DDL under such imbalance, we conducted experiments on Office-Home by reducing the number of training samples from selected domains (i.e., \"art\" and \"clipart\"), while keeping the other domains fixed. Our default setting uses 8 shots, and we test the performance when this is reduced to 4 or 1 shot(s).\n\nAs shown in Table B-2, our method maintains stable performance under domain imbalance. Even with just one sample from each of the \"art\" and \"clipart\" domains, the model achieves a $For$ of 73.85 and a $Mem$ of 64.31 for $|D_{for}| = 3$. These results indicate that our method can still perform effective domain disentangling and selective forgetting even in severely imbalanced scenarios.\n\nWe appreciate this suggestion, as it offers an additional valuable perspective for further demonstrating the robustness and practical utility of our approach!\n\n*Table B-2. Robustness to Domain Imbalance.*\n| | | $\\|D_{for}\\|\\=1$ | | | $\\|D_{for}\\|\\=2$ | | | $\\|D_{for}\\|\\=3$ | |\n|:----|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n|Number of shots for selected domains|$H\\uparrow$|$Mem\\uparrow$|$For\\uparrow$|$H\\uparrow$|$Mem\\uparrow$|$For\\uparrow$|$H\\uparrow$|$Mem\\uparrow$|$For\\uparrow$|\n|8 shots|69.96|77.93|64.34|73.58|75.61|71.89|75.89|72.15|80.77|\n|4 shots|68.23|82.24|59.55|71.24|81.21|63.82|74.34|74.61|74.08|\n|1 shot|66.89|77.64|59.61|63.80|78.16|56.53|68.75|64.31|73.85|\n\n\n### **Q3. Performance with other VLMs?**\n\nAlthough evaluating on diverse pre-trained VLMs would be ideal, such efforts are limited by the scarcity of publicly available CLIP-like pre-trained models. This would be a reason that most existing studies on CLIP fine-tuning or unlearning (e.g., [f–j]) focus solely on CLIP. \n\nNevertheless, we report the performance of our method applied to another VLM, SigLIP [e], in Table B-3 using ImageNet. As shown, our method outperforms the baseline (the best competitor compared with our method in our main paper), indicating its strong effectiveness even if not coupled with CLIP. These results demonstrates that our approach is not specific to CLIP and can be extended to other VLMs. \n\n*Table B-3. Performance with SigLIP on ImageNet.*\n| |$H\\uparrow$|$Mem\\uparrow$|$For\\uparrow$|\n|:----|:----:|:----:|:----:|\n|Zero-shot |46.47 |**63.26**| 36.73|\n|Baseline | 48.98| 60.32 |41.23|\n|Ours| **64.97**| 48.42 |**98.71**|\n\n\n### **Q4. Computation complexity?**\nTable B-4 shows computation time of our method with and without InstaPG using NVIDIA RTX A4000 GPU. The results indicate that adding InstaPG introduces a very minor overhead, namely an additional 50 seconds for training and only 0.2 seconds for inference.\n\n*Table B-4. Training and Inference Times.*\n| |Training [s] |Inference [s] |\n|:----|:----|:----|\n|w/o InstaPG|501|13.1|\n|Ours|550|13.3|\n\n---\n**References**\n\n[a] Wu et al., Test-Time Domain Adaptation by Learning Domain-Aware Batch Normalization, AAAI 2024.\n\n[b] Cho et al., PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalization, ICCV 2023.\n\n[c] Jhoo and Heo, Collaborative Learning with Disentangled Features for Zero-shot Domain Adaptation, ICCV 2021.\n\n[d] Mitsuzumi et al., Generalized Domain Adaptation, CVPR 2021.\n\n[e] Zhai et al., Sigmoid Loss for Language Image Pre-Training, ICCV 2023.\n\n[f] Zhou et al., Learning to Prompt for Vision-Language Models, IJCV 2022.\n\n[g] Khattak et al., MaPLe: Multi-modal Prompt Learning, CVPR 2023.\n\n[h] Li et al., Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification, EMNLP 2024.\n\n[i] Huang et al., LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIP, CVPR 2024.\n\n[j] Kuwana et al., Black-box Forgetting, NeurIPS 2024."}, {"review_id": "GBH7mhEL4T", "reviewer": "Reviewer_tNbB", "summary": "This paper introduces Approximate Domain Unlearning (ADU), a new task that aims to make vision-language models forget specific domains (e.g., sketches) while retaining performance on others (e.g., real images). The authors propose a Domain Disentangling Loss and an Instance-wise Prompt Generator to address domain entanglement. Experiments on multiple benchmarks show clear improvements over existing unlearning and CLIP tuning baselines.", "questions": "- How does the method perform when domain definitions are noisy or ambiguous? For instance, can it still disentangle domains effectively if the boundary between “art” and “sketch” is unclear?\n\n- Can the method selectively forget domain-specific features within a class without hurting class-level performance?\n\n- It would be helpful to see how the Mem and For metrics evolve over training. Does forgetting conflict with memorization at certain stages, or are the gradients well-behaved?\n\n- Is the method inherently tied to VLMs like CLIP, or could it work with plain vision encoders?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n-  The paper introduces an interesting problem setting called Approximate Domain Unlearning. Instead of forgetting specific object classes, the goal here is to make the model forget certain domains like sketches or illustrations, which makes sense in scenarios where domain confusion can cause real issues.\n\n\n- The method is interesting, it combines a domain disentangling loss to separate different domains in the feature space, and an Instance-wise Prompt Generator that lets the model adjust prompts based on each image, which helps deal with how mixed-up the domains usually are in VLMs.\n\n- Well written, clear visualization.\n\nWeaknesses\n\n-  The method assumes you know the domain label for every training image, which might not always be the case. The paper mentions this, but doesn’t really give a backup plan if those labels aren’t available.\n\n- The proposed DDL seems to work well, but it uses MMD, which is known to be sensitive to things like the choice of kernel and how batches are formed. The paper doesn’t really explain why their particular setup was chosen or how stable it is.\n\n- The method only updates the vision prompts and keeps the rest of the model frozen. That makes it efficient, but also limits how much it can actually change the internal representations, especially if the domain-specific stuff is already mixed in early on. The paper doesn’t really talk about this trade-off or look into which layers are actually doing the forgetting.\n\n- In all experiments, the domains to forget and retain are predefined and balanced in terms of class coverage. But in practice, the distribution of domains and classes is rarely so clean, (e.g., some domains might contain only a subset of classes or very different class frequencies). The paper doesn’t explore how the method handles such imbalanced or partially overlapping domain-class settings, which limits the understanding of its robustness.", "was_revised": true, "final_justification": "Thank you very much for the author’s response. It addressed all of my questions, so I’ll stick with my original score.", "rebuttal": "We sincerely thank Reviewer tNbB for their thoughtful and constructive review. We are especially grateful that the reviewer **recognized all the main contributions of our work**, including the introduction of Approximate Domain Unlearning (ADU) as a novel problem setting, the effectiveness of our proposed solution, and the clarity of our presentation. We believe that, as the first study to explore ADU, our paper can meaningfully contribute to the community. We also appreciate that the reviewer's comments were framed not as criticisms of fundamental flaws, but rather as constructive suggestions for further development. We view these points as valuable opportunities to deepen and expand our work, and we address them point-by-point below.\n\n### **Q1. Domain labels not always fully available.**\nAs we discuss in the Limitations section of our paper, we acknowledge that domain labels may not always be available for all training samples in real-world scenarios. However, it is important to note that most standard formulations in domain adaptation and domain generalization (e.g., [a–c]) assume full access to domain labels during training. Therefore, our setting aligns with this widely accepted convention and should not be considered an unrealistic simplification.\n\nMeanwhile, to evaluate the performance of our method under partial domain label availability, we consider a setting in which a portion of the training samples (ranging from 30% to 70%) lack domain annotations.\nTo assess the effectiveness of combining domain estimation with our method in such cases, we adopt a simple pseudo-labeling approach: a domain classifier is trained using only the samples with known domain labels and then used to assign pseudo labels to the unlabeled samples.\n\nAs shown in Table C-1, without domain estimation, the $H$-score drops from 75.89% to 49.97% under 70% missing domain labels. In contrast, with domain estimation, the $H$-score remains substantially higher at 61.21% even under the same missing rate. These results support our claim that our approach can be extended to more realistic cases where only partial domain annotations are available.\n\n*Table C-1. Performance on Office-Home with Partial Domain Labels.*\n\n| | |w/o Domain Estimation| | | w/ Domain Estimation| |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|Unlabeled sample ratio|$H\\uparrow$|$Mem\\uparrow$|$For\\uparrow$|$H\\uparrow$|$Mem\\uparrow$|$For\\uparrow$|\n|0|75.89|72.15|80.77|75.89|72.15|80.77|\n|0.3|**63.38**|83.35|52.68|**65.81**|83.06|55.98|\n|0.5|**55.77**|84.63|43.66|**63.64**|84.69|52.20|\n|0.7|**49.97**|83.84|36.98|**61.21**|83.37|50.00|\n\n### **Q2. Choice of kernel and batch formation for MMD?**\nAll results reported in the paper use the RBF (Gaussian) kernel. To assess sensitivity to kernel choice, we additionally tested four different kernels on Office-Home in Table C-2. Among them, the linear, Laplacian, and RBF kernels demonstrated generally stable performance, with only modest differences in accuracy. When validation-based kernel selection is feasible, we recommend choosing the best among these three; otherwise, the linear kernel serves as a conservative and reliable default. \n\nRegarding batch formation, we use a batch size of 128 and randomly select 128 samples per batch from the entire training data.\n\n*Table C-2. Impact of Kernel Choice.*\n\n| | |$\\|D_{for}\\|\\=1$| | |$\\|D_{for}\\|\\=2$| | |$\\|D_{for}\\|\\=3$| |\n|:----|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n|Kernel|$H\\uparrow$|$Mem\\uparrow$|$For\\uparrow$|$H\\uparrow$|$Mem\\uparrow$|$For\\uparrow$|$H\\uparrow$|$Mem\\uparrow$|$For\\uparrow$|\n|Linear|66.83|**79.85**|58.43|69.32|**77.35**|63.46|73.91|**74.02**|74.50|\n|Laplacian|68.50|79.00|61.44|71.18|75.93|67.17|73.66|71.69|76.54|\n|Polynomial|36.53|23.29|**84.68**|49.73|41.79|61.40|53.65|44.49|67.56|\n|RBF|**69.96**|77.93|64.34|**73.58**|75.61|**71.89**|**75.89**|72.15|**80.77**|\n\n### **Q3. Concern about limited representation change in early layers due to model freezing.**\nMany prior studies have proven that fully fine-tuning CLIP often leads to catastrophic forgetting, especially in few-shot regimes [e, f]. Prompt tuning with frozen model parameters has thus become a standard alternative, which we follow. As shown in Table C-3, updating model parameters in our task indeed causes severe catastrophic forgetting, reinforcing this concern.\n\nDespite freezing, our method affects both low- and high-level representations. Fig. 7 shows stable unlearning performance across various prompt insertion depths, including early layers. This suggests that the learned prompts induce meaningful changes throughout the network, addressing the concern about limited adaptability in early stages.\n\n*Table C-3. Impact of Updating Model Parameters on Office-Home.*\n\n| | | $\\|D_{for}\\|\\=1$ | | |$\\|D_{for}\\|\\=2$| | | $\\|D_{for}\\|\\=3$| | \n|:----|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n| Update Model Parameters?|$H\\uparrow$|$Mem\\uparrow$|$For\\uparrow$|$H\\uparrow$|$Mem\\uparrow$|$For\\uparrow$|$H\\uparrow$|$Mem\\uparrow$|$For\\uparrow$|\n|Yes|3.90|2.00|**98.31**|2.95|1.50|**98.52**|3.64|1.86|**98.26**|\n|No (Ours)|**66.81**|**58.86**|77.23|**67.81**|**58.89**|79.90|**68.83**|**59.06**|82.48|\n\n### **Q4. Robustness to Partial Domain-Class Overlap.**\nWe conducted additional experiments on Office-Home where samples within three random classes (specifically, mug, radio, and hammer) were excluded from only \"art\" and \"clipart\" domains, creating a setting with partial domain-class overlap. As shown in Table C-4, even under such imbalance, our method maintains competitive $H$-scores. These results suggest that our method can handle realistic class-domain imbalances without critical loss in effectiveness.\n\n*Table C-4. Robstness to Partial Domain-Class Overlap.*\n| | | $\\|D_{for}\\|\\=1$ | | | $\\|D_{for}\\|\\=2$| | | $\\|D_{for}\\|\\=3$| |\n|:----|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n||$H\\uparrow$|$Mem\\uparrow$|$For\\uparrow$|$H\\uparrow$|$Mem\\uparrow$|$For\\uparrow$|$H\\uparrow$|$Mem\\uparrow$|$For\\uparrow$|\n|Original Dataset|**69.96**|77.93|**64.34**|**73.58**|75.61|**71.89**|**75.89**|72.15|**80.77**|\n|Dataset w/ Partial Domain-Class Overlap |67.93|**81.81**|58.69|72.03|**79.61**|66.24|74.51|**75.12**|73.90|\n\n### **Q5. Robustness to ambiguous domain definitions.**\nWe agree that domains may overlap semantically or visually, making clear boundaries difficult. To examine this, we report per-domain accuracy on Office-Home in Tables C-5. Our method demonstrates selective forgetting and memorization even between highly similar domains, indicating effective disentanglement despite domain ambiguity.\n\nFor robustness to label noise, our response to Q1 may offer a partial answer, as it addresses the case where domain labels are only partially available — a setting that can also be interpreted as involving random label noise on unlabeled samples (Table C-1).\n\n*Table C-5. Per-domain Accuracy on Office-Home.*\n||art|clipart|product|real|\n|:---|:---:|:---:|:---:|:---:|\n|Zero-shot CLIP|74.34|60.97|80.43|81.29|\n|Ours (Forgotten = art)|**39.25**|70.18|88.72|75.91|\n|Ours (Forgotten = clipart)|77.36|**15.13**|87.87|80.00|\n|Ours (Forgotten = product)|80.38|72.15|**32.77**|77.85|\n|Ours (Forgotten = real)|68.30|67.98|78.94|**55.48**|\n\n### **Q6. The method can forget domain knowledge without hurting class knowledge?**\nYes, this is precisely the goal of ADU. The task is to forget domain-specific information (e.g., \"sketch\") while preserving class-relevant, domain-agnostic features. As shown in Table 1, our method significantly improves the forgetting score ($For$) on the target domain, while maintaining the memorization score ($Mem$) on the rest. These results demonstrate that the model can forget domain-specific representations of a class (e.g., \"sketch\" dog) without degrading its classification accuracy on the same class in other domains (e.g., \"real\" dog).\n\n### **Q7. How $Mem$ and $For$ scores evolve over training?**\nAs shown in Table C-6, we see that $For$ increases rapidly while $Mem$ drops in the early stages, suggesting a conflict between the two. However, both socres improve steadily after this phase, indicating that the model gradually learns to balance forgetting and retention. These results suggest a two-stage process: initial domain disentanglement followed by controlled forgetting and memorization.\n\n*Table C-6. $Mem$ and $For$ over Training Iterations on Office-Home.*\n\n|Iteration|0|20|40|60|80|100|200|500|800|\n|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|\n|$Mem\\uparrow$|59.48|19.67|33.97|48.55|50.39|60.61|66.15|76.58|79.19|\n|$For \\uparrow$ |48.06|85.61|67.56|64.27|64.84|63.41|64.48|84.77|85.36|\n\n### **Q8. The method work with plain vision encoders?**\nThank you for this valuable suggestion! Our primary focus is on pre-trained VLMs, particularly CLIP, where domain-specific representations are highly entangled. While our DDL and InstaPG are in principle applicable to plain vision encoders, the absence of public pre-trained vision-only models with domain generalization capabilities comparable to CLIP makes immediate evaluation difficult. We are interested in exploring this point in the final version.\n\n---\n**References**\n\n[a] Wu et al., Test-Time Domain Adaptation by Learning Domain-Aware Batch Normalization, AAAI 2024.\n\n[b] Cho et al., PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalization, ICCV 2023.\n\n[c] Jhoo and Heo, Collaborative Learning with Disentangled Features for Zero-shot Domain Adaptation, ICCV 2021.\n\n[d] Mitsuzumi et al., Generalized Domain Adaptation, CVPR 2021.\n\n[e] Zhou et al., Learning to Prompt for Vision-Language Models, IJCV 2022.\n\n[f] Kumar et al., Fine-tuning Can Distort Pretrained Features and Underperform Out-of-Distribution, ICLR 2022."}, {"review_id": "fdnBFIpVzn", "reviewer": "Reviewer_ubkk", "summary": "This paper introduces Approximate Domain Unlearning (ADU), a novel task aiming to enable pre-trained Vision-Language Models (VLMs) to selectively forget recognition capabilities for specified domains (e.g., illustrations, paintings) while preserving accuracy for target domains (e.g., real images). Conventional class unlearning methods fail to address feature entanglement caused by strong domain generalization in VLMs. The authors propose Domain Disentangling Loss (DDL) and an Instance-wise Prompt Generator (InstaPG), which enforce domain feature separation and adaptively capture instance-level variations, respectively. Experiments on three multi-domain datasets validate the method’s effectiveness", "questions": "1.Can the authors supplement experiments with partial domain labels to validate adaptability in real scenarios?\n2.Test the method on autonomous driving datasets to verify forgetting effects on complex scenes like roadside advertisements.\n3.Provide comparisons of training time and GPU resource consumption with baselines to clarify engineering costs.\n4.Using DDL will reduce accuracy on memorized domains,why and how to solve it?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1.First to define domain-level unlearning (ADU), filling the application gap of traditional class unlearning, particularly in scenarios like autonomous driving\n2.DDL maximizes inter-domain feature distances via cross-entropy and MMD, while InstaPG generates instance-specific prompts to handle intra-domain style variations, supported by solid theoretical foundations.\nWeaknesses:\nDependence on Domain Labels: The method assumes complete domain labels during training, without validating robustness under partial label scenarios.\nLimited Real-world Validation: Untested on practical datasets (e.g., autonomous driving), leaving domain generalization in real applications unproven.\nUnclear Computational Efficiency: No comparison of training time or parameter scale with baselines, hindering evaluation of engineering feasibility.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer ubkk for their thoughtful and constructive review. We are especially grateful that the reviewer **acknowledged all the main contributions of our paper**, including the novelty of the problem setting and the effectiveness of our proposed solution. We believe this paper represents the first attempt at Approximate Domain Unlearning (ADU) in Vision-Language Models (VLMs) and offers a meaningful contribution by tackling key challenges in this emerging area. Below, we provide detailed, point-by-point responses to all questions and concerns raised. We appreciate the constructive feedback, which we believe improve our work. \n\n### **Q1. Dependence on Domain Labels.**\n\nAs we discuss in the Limitations section of our paper, we acknowledge that domain labels may not always be available for all training samples in real-world scenarios. However, it is important to note that most standard formulations in domain adaptation and domain generalization (e.g., [a–c]) assume full access to domain labels during training. Therefore, our setting aligns with this widely accepted convention and should not be considered an unrealistic simplification.\n\nMeanwhile, to evaluate the performance of our method under partial domain label availability, we consider a setting in which a portion of the training samples (ranging from 30% to 70%) lack domain annotations.\nTo assess the effectiveness of combining domain estimation with our method in such cases, we adopt a simple pseudo-labeling approach: a domain classifier is trained using only the samples with known domain labels and then used to assign pseudo labels to the unlabeled samples.\n\nAs shown in Table D-1, without domain estimation, the $H$-score drops from 75.89% to 49.97% under 70% missing domain labels. In contrast, with domain estimation, the $H$-score remains substantially higher at 61.21% even under the same missing rate. These results support our claim that our approach can be extended to more realistic cases where only partial domain annotations are available.\n\n*Table D-1. Performance on Office-Home with Partial Domain Labels.*\n\n| | |w/o Domain Estimation| | | w/ Domain Estimation| |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|Unlabeled sample ratio|$H\\uparrow$|$Mem\\uparrow$|$For\\uparrow$|$H\\uparrow$|$Mem\\uparrow$|$For\\uparrow$|\n|0|75.89|72.15|80.77|75.89|72.15|80.77|\n|0.3|**63.38**|83.35|52.68|**65.81**|83.06|55.98|\n|0.5|**55.77**|84.63|43.66|**63.64**|84.69|52.20|\n|0.7|**49.97**|83.84|36.98|**61.21**|83.37|50.00|\n\n### **Q2: Limited Real-world Validation.**\n\nTo the best of our knowledge, no public autonomous driving dataset contains annotated roadside advertisements and multi-domain versions of objects like vehicles, making direct empirical evaluation infeasible at this time.\n\nInstead, the best possible evaluation we can currently perform is a proxy experiment that partially simulates such scenarios using an existing multi-domain dataset, Mini DomainNet. Specifically, we focus on seven vehicle-related classes (bus, car, motorbike, pickup truck, police car, school bus, and tractor), and evaluate our method when retaining only the \"real\" domain while forgetting the other illustration domains (\"clipart,\" \"painting,\" and \"sketch\"). As shown in Table D-2, our method effectively preserves recognition of real vehicles while substantially suppressing classification on illustrated ones. These results suggest that our approach holds promise for practical applications such as autonomous driving or car counting.\n\nWe acknowledge that full-scale validation on real-world deployment tasks remains an important direction for future work. Nonetheless, we believe our findings serve as an informative first step toward domain-specific unlearning under realistic constraints.\n\n*Table D-2. Per-domain Accuracy on Vehicle-Related Classes in Mini DomainNet.*\n\n|Per-domain accuracy|clipart ↓|painting ↓|real ↑|sketch ↓|\n|:----|:----|:----|:----|:----|\n|Before unlearning|80.64|78.10|**87.94**|72.54|\n|Ours w/o DDL and InstaPG|31.27|34.29|82.54|16.03|\n|Ours|**22.38**|**21.43**|84.44|**14.29**|\n\n### **Q3. Unclear Computational Efficiency.**\n\nTable D-3 below summarizes GPU memory usage and training time with NVIDIA RTX A4000 GPU on Office-Home. While our method incurs only slightly higher computational cost than lightweight baselines such as CoOp and LP++, it remains on par with advanced CLIP fine-tuning methods such as MaPLe and CLIPFit. These results prove that our method is sufficiently efficient for practical use. Notably, adding InstaPG increases memory usage by only 1 GB and training time by just 1 minute, indicating minimal overhead.\n\n*Table D-3. Computational Complexity.*\n\n| |Memory [GB]|Time [s]|\n|:----|:----|:----|\n|CoOp [a]|1.9 | 283 |\n|MaPLe [b]| 10.4 | 539 |\n|BBF|2.5| 2882 |\n|CLIPFit| 11.0 | 340 |\n|LP++| 3.8 | 35 |\n|Ours w/o  InstaPG |9.7| 501 |\n|Ours |10.7| 550 |\n\n### **Q4. DDL reduce accuracy on memorized domains?**\n\nPartially true. As shown in Table D-4, the per-domain accuracy on Office-Home illustrates that DDL can have varying effects across domains. For example, when \"art\" is forgotten, the accuracy for the retained domain \"real\" slightly drops from 81.29% to 75.91%, whereas the accuracy for \"product\" improves from 80.43% to 88.72%. These results indicate that the effect of DDL depends on inter-domain correlations and task structure, leading to either small gains or small losses in retained domains. Note that such degradations are modest and do not lead to critical failure. Overall, DDL serves as an effective mechanism for suppressing domain-specific information while maintaining a favorable balance between forgetting and retention.\n\n*Table D-4. Per-domain Accuracy on Office-Home.*\n\n||art|clipart|product|real|\n|:---|:---:|:---:|:---:|:---:|\n|Zero-shot CLIP|74.34|60.97|80.43|81.29|\n|Ours (Forgotten = art)|**39.25**|70.18|88.72|75.91|\n|Ours (Forgotten = clipart)|77.36|**15.13**|87.87|80.00|\n|Ours (Forgotten = product)|80.38|72.15|**32.77**|77.85|\n|Ours (Forgotten = real)|68.30|67.98|78.94|**55.48**|\n\n---\n**References**\n\n[a] Wu et al., Test-Time Domain Adaptation by Learning Domain-Aware Batch Normalization, AAAI 2024.\n\n[b] Cho et al., PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalization, ICCV 2023.\n\n[c] Jhoo and Heo, Collaborative Learning with Disentangled Features for Zero-shot Domain Adaptation, ICCV 2021.\n\n[d] Mitsuzumi et al., Generalized Domain Adaptation, CVPR 2021."}]

Author rebuttal:

Meta-review: This work proposes the new problem setting of Approximate *Domain* Unlearning (ADU). Where prior Unlearning works focused on removing knowledge of specific classes, ADU focuses on removing the ability of pretrained models to recognize images from a given domain (e.g., removing knowledge of clip art images) while still maintaining performance on images from remaining domains. The authors argue that ADU is even more difficult than traditional class unlearning, as VLM domain generalization capabilities lead domains to have overlapping representations in the model's feature space, while classes are typically more easily distinguished in the embedding spaces. To overcome this, the authors propose a Domain Disentangling Loss (DDL) to explicitly disentangle domains in the embedding space, and an Instance-wise Prompt Generator (InstaPG) component to adaptively model instance-level domain variations. While no other methods designed for the proposed ADU problem setting exist at this time, the authors develop reasonable baselines (e.g. finetuning, adapting a leading unlearning approach), which their method performs well against on evaluations over three multi-domain image benchmark datasets.

Reviewers initial evaluation was largely positive. Still, reviewers raised concerns regarding computational overhead [gH1G, ZXGU,  ubkk], the requirement of having domain labels [ZXGU,  tNbB,  ubkk], domain imbalance [ZXGU], experiments using only CLIP and no additional VLMs [ZXGU], and sensitivity to choice of kernel for MMD [tNbB]. However, the authors response to these concerns, including additional results, was satisfactory to the point where all active reviewers have acknowledged that their concerns have been adequately addressed.

The proposed problem setting of ADU is novel while being well-justified, and one can imagine the utility and necessity of a solution to this problem, indicating that this is a worthy line of research. The technical challenges of ADU, specifically domains not being easily separated in VLM latent spaces, are interesting and the proposed approach is novel and sound. The authors' rebuttal was comprehensive, and the new results strengthen the already good work. For these reasons, I recommend acceptance. 

Furthermore, I recommend a spotlight for this paper. The novel problem setting introduced in this work is, in my opinion, an interesting and important task. Highlighting this newly defined task setting would be valuable for the research community. The proposed approach for this new task is likewise novel and we'll justified. While some reviewers still had reservations about the need for domain labels, I do not believe this is a critical flaw. The need for domain labels is standard for most domain shift approaches, so I don't believe this is too strong of an assumption, and the semi-supervised results strengthen the work. So while it's a limitation, I don't think it's too strong of one.
Decision: Accept (spotlight)