Title: Approximate Domain Unlearning for Vision-Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Approximate unlearning

Vision-Language Models
Abstract: Pre-trained Vision-Language Models (VLMs) exhibit strong generalization capabilities, enabling them to recognize a wide range of objects across diverse domains without additional training. However, they often retain irrelevant information beyond the requirements of specific target downstream tasks, raising concerns about computational efficiency and potential information leakage. This has motivated growing interest in approximate unlearning, which aims to selectively remove unnecessary knowledge while preserving overall model performance. Existing approaches to approximate unlearning have primarily focused on {\em class unlearning}, where a VLM is retrained to fail to recognize specified object classes while maintaining accuracy for others. However, merely forgetting object classes is often insufficient in practical applications. For instance, an autonomous driving system should accurately recognize {\em real} cars, while avoiding misrecognition of {\em illustrated} cars depicted in roadside advertisements as {\em real} cars, which could be hazardous. In this paper, we introduce {\em Approximate Domain Unlearning (ADU)}, a novel problem setting that requires reducing recognition accuracy for images from specified domains (e.g., {\em illustration}) while preserving accuracy for other domains (e.g., {\em real}). ADU presents new technical challenges: due to the strong domain generalization capability of pre-trained VLMs, domain distributions are highly entangled in the feature space, making naive approaches based on penalizing target domains ineffective. To tackle this limitation, we propose a novel approach that explicitly disentangles domain distributions and adaptively captures instance-specific domain information. Extensive experiments on four multi-domain benchmark datasets demonstrate that our approach significantly outperforms strong baselines built upon state-of-the-art VLM tuning techniques, paving the way for practical and fine-grained unlearning in VLMs. Code : https://kodaikawamura.github.io/Domain_Unlearning/.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Approximate Domain Unlearning (ADU), a novel task aiming to enable pre-trained Vision-Language Models (VLMs) to selectively forget recognition capabilities for specified domains (e.g., illustrations, paintings) while preserving accuracy for target domains (e.g., real images). Conventional class unlearning methods fail to address feature entanglement caused by strong domain generalization in VLMs. The authors propose Domain Disentangling Loss (DDL) and an Instance-wise Prompt Generator (InstaPG), which enforce domain feature separation and adaptively capture instance-level variations, respectively. Experiments on three multi-domain datasets validate the method’s effectiveness\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.First to define domain-level unlearning (ADU), filling the application gap of traditional class unlearning, particularly in scenarios like autonomous driving\n2.DDL maximizes inter-domain feature distances via cross-entropy and MMD, while InstaPG generates instance-specific prompts to handle intra-domain style variations, supported by solid theoretical foundations.\nWeaknesses:\nDependence on Domain Labels: The method assumes complete domain labels during training, without validating robustness under partial label scenarios.\nLimited Real-world Validation: Untested on practical datasets (e.g., autonomous driving), leaving domain generalization in real applications unproven.\nUnclear Computational Efficiency: No comparison of training time or parameter scale with baselines, hindering evaluation of engineering feasibility.\n\nQUESTIONS:\n1.Can the authors supplement experiments with partial domain labels to validate adaptability in real scenarios?\n2.Test the method on autonomous driving datasets to verify forgetting effects on complex scenes like roadside advertisements.\n3.Provide comparisons of training time and GPU resource consumption with baselines to clarify engineering costs.\n4.Using DDL will reduce accuracy on memorized domains,why and how to solve it?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Approximate Domain Unlearning (ADU), a new task that aims to make vision-language models forget specific domains (e.g., sketches) while retaining performance on others (e.g., real images). The authors propose a Domain Disentangling Loss and an Instance-wise Prompt Generator to address domain entanglement. Experiments on multiple benchmarks show clear improvements over existing unlearning and CLIP tuning baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n-  The paper introduces an interesting problem setting called Approximate Domain Unlearning. Instead of forgetting specific object classes, the goal here is to make the model forget certain domains like sketches or illustrations, which makes sense in scenarios where domain confusion can cause real issues.\n\n\n- The method is interesting, it combines a domain disentangling loss to separate different domains in the feature space, and an Instance-wise Prompt Generator that lets the model adjust prompts based on each image, which helps deal with how mixed-up the domains usually are in VLMs.\n\n- Well written, clear visualization.\n\nWeaknesses\n\n-  The method assumes you know the domain label for every training image, which might not always be the case. The paper mentions this, but doesn’t really give a backup plan if those labels aren’t available.\n\n- The proposed DDL seems to work well, but it uses MMD, which is known to be sensitive to things like the choice of kernel and how batches are formed. The paper doesn’t really explain why their particular setup was chosen or how stable it is.\n\n- The method only updates the vision prompts and keeps the rest of the model frozen. That makes it efficient, but also limits how much it can actually change the internal representations, especially if the domain-specific stuff is already mixed in early on. The paper doesn’t really talk about this trade-off or look into which layers are actually doing the forgetting.\n\n- In all experiments, the domains to forget and retain are predefined and balanced in terms of class coverage. But in practice, the distribution of domains and classes is rarely so clean, (e.g., some domains might contain only a subset of classes or very different class frequencies). The paper doesn’t explore how the method handles such imbalanced or partially overlapping domain-class settings, which limits the understanding of its robustness.\n\nQUESTIONS:\n- How does the method perform when domain definitions are noisy or ambiguous? For instance, can it still disentangle domains effectively if the boundary between “art” and “sketch” is unclear?\n\n- Can the method selectively forget domain-specific features within a class without hurting class-level performance?\n\n- It would be helpful to see how the Mem and For metrics evolve over training. Does forgetting conflict with memorization at certain stages, or are the gradients well-behaved?\n\n- Is the method inherently tied to VLMs like CLIP, or could it work with plain vision encoders?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the problem of Approximate Domain Unlearning (ADU), which aims to reduce recognition accuracy for images from specified domains (e.g., clip art) while preserving accuracy for other domains (e.g., real images). Compared to existing approximate class unlearning tasks, ADU can be considered as a finer-grain task as it needs to distinguish different domains within the same class. With this finer-grain task, there are new challenges. For example, domain distributions are highly entangled in the feature space (e.g. real car images vs. illustrated car images) due to the strong domain generalization capability of pre-trained VLMs, which makes domain unclearning difficult. This paper proposes two approaches to tackle the challenges: Domain Disentangling Loss (DDL) and Instance-wise Prompt Generator (InstaPG).  Domain disentangling loss explicitly disentangles domain distributions in the latent space. Instance-wise Prompt Generator adaptively models instance-level domain variations. \n\nExperiments and ablation studies are done on multi-domain image benchmark datasets, showing the effectiveness of the approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) The paper proposes a new problem: Approximate Domain Unlearning (ADU).  This is useful for applications such as in self-driving, where car recognition would mean recognition of “real” cars on the road, instead of pictures of a car on a poster.\n2) The paper proposes two methods to alleviate the entanglement between different domain distributions in the latent feature space: Domain Disentangling Loss (DDL) and Instance-wise Prompt Generator (InstaPG). DDL is based on the idea that if the feature distributions of individual domains are well-separated, the domain labels can be accurately predicted, and vice versa. InstaPG is to account for  instance-level variation in the images. The proposed approaches are intuitive and seem to be effective, as shown via experiments. \n3) Experiments and ablation studies are done on multi-domain image benchmark datasets, including ImageNet, Office-Home, and Mini DomainNet. The experimental results demonstrated the effectiveness of the proposed approach. \n4) The paper is generally well written.\n  \n\nWeakness:\n1) One limitation of the proposed approach is that, for using the domain disentangling loss,  it assumes the availability of domain labels. As the authors pointed out, this may be a strong assumption, for example, domain labels are not readily available in autonomous driving data.\n2) In applications such as autonomous driving, data may be very unbalanced among different domains. For example, there would be a lot more “real” car data compared to illustrated car data. It would be interesting to see how domain disentangling loss works in those situations.\n3) The paper is based on a pre-trained CLIP model. It would be interesting to see results on other pre-trained VLMs.\n4) The Instance-wise Prompt Generator (InstaPG) increases the training and inference latency. It would be nice to have some analysis on the computational cost.\n\nQUESTIONS:\n1) In applications such as autonomous driving, data may be very unbalanced among different domains. For example, there would be a lot more real car images than illustrated car images (car on a poster).  Have you studied how sensitive domain disentangling loss is to such kind of data imbalance among domains?  \n2) As domain labels are not always readily available. How sensitive is your approach to noise in domain labels?\n3) Have you tried other pre-trained VLMs?\n4) Do you have any analysis on the computation cost, e.g., the increase with the use of InstaPG?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new machine unlearning framework for Vision-Language Models. Unlike traditional class-level unlearning that removes specific object categories, this work selectively removes recognition capability for images from certain domains (e.g., illustrations, sketches) while preserving performance on other domains (e.g., real photos). The authors propose two key technical components: Domain Disentangling Loss to separate domain distributions in feature space, and Instance-wise Prompt Generator to handle within-domain style variations.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n\n1. The problem is genuinely innovative and addresses practical limitations of existing unlearning approaches. The autonomous driving example (distinguishing real cars from illustrated advertisements) is compelling and highlights real-world relevance.\n\n2. This work effectively tackles the core challenge of domain entanglement in VLMs by combining cross-entropy loss with MMD maximization. It intelligently handles intra-domain variability through attention-based prompt generation. The combination addresses both macro-level (domain separation) and micro-level (instance variation) challenges.\n\n3. Extensive evaluation on three multi-domain datasets with different numbers of forgotten domains. Strong baselines adapted from state-of-the-art VLM fine-tuning and unlearning methods. And thorough ablation studies demonstrating the contribution of each component.\n\nCons:\n\n1. Experiments use only 8 samples per domain for training, which seems quite small. Evaluation limited to relatively simple multi-domain datasets. Unclear how the approach scales to more complex domain distributions or larger datasets\n\n2. The paper lacks discussion of scenarios where the method might fail or perform poorly. What happens when domain boundaries are ambiguous or when domains share significant visual similarity?\n\n3. No detailed analysis of computational overhead, training time, or memory requirements compared to baseline methods.\n\nQUESTIONS:\nWhy not include additional metrics like domain classification accuracy after unlearning, or measures of knowledge retention for related but not-forgotten content?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}