Contrastive Unlearning: A Contrastive Approach to Machine Unlearning

Authors: Hong kyu Lee, Qiuchen Zhang, Carl Yang, Jian Lou, Li Xiong
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Machine unlearning, Privacy, Contrastive learning

Abstract:
Machine unlearning aims to eliminate the influence of a subset of training samples (i.e., unlearning samples) from a trained model. Effectively and efficiently removing the unlearning samples without negatively impacting the overall model performance is challenging. Existing works mainly exploit input and output space and classification loss, which can result in ineffective unlearning or performance loss. In addition, they utilize on unlearning or remaining samples ineffectively, sacrificing either unlearning efficacy or efficiency. Our main insight is that direct optimization on the representation space utilizing both unlearning and remaining samples can effectively remove influence of unlearning samples while maintaining representations learned from remaining samples. We propose a contrastive unlearning framework, leveraging the concept of representation learning for more effective unlearning. It removes the influence of unlearning samples by contrasting their embeddings against the remaining samples' embeddings so that their embeddings are closer to the embeddings of unseen samples. Experiments on a variety of datasets and models on both class unlearning and sample unlearning showed that contrastive unlearning achieves the best unlearning effects and efficiency with the lowest performance loss compared with the state-of-the-art algorithms.

Decision: reject
Reviewer scores: 5, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730380091698

thread_type:
official_review

<thread_object>
creator: reviewer_i83z

<content>
summary:
In this paper, the authors design a contrastive unlearning method. This method could remove the influence of unlearning samples by constrastive their embeddings against the remaining samples' embeddings.

strengths:
1. The paper is well-organized.

2. The motivation is clearly described.

3. The appendix is comprehensive.

weaknesses:
1. Table 1 is confusing. The results for “remain test” on RN18 seem to suggest that higher is better, so why is 85.79 bolded?

2. A similar issue appears in Table 3.

3. The datasets used are only CIFAR-10 and SVHN. It is recommended to include additional datasets.

4. The authors should release the code to ensure reproducibility.

questions:
Please see weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730438237226

thread_type:
official_review

<thread_object>
creator: reviewer_hnfr

<content>
summary:
The author introduces a novel form of contrastive loss tailored to the machine unlearning paradigm. The main insight is to selectively remove the most relevant features of the unlearning samples while preserving the quality of embeddings for the remaining samples. Experimental results demonstrate that the proposed contrastive loss achieves superior performance on single-class and sample-unlearning tasks, surpassing state-of-the-art methods.

strengths:
1. The proposed supervised contrastive learning framework is well-structured and easy to understand. By constructing positive and negative pairs, the encoder effectively deactivates the embedding of unlearning samples.
2. The framework performs well across three benchmarks, showing improved unlearning efficiency and effectiveness over existing methods.

weaknesses:
1. The approach of using supervised contrastive learning for unlearning is somewhat simplistic, as it overlooks the potential generalization decay of the model, as noted in Question 1. Merely introducing a framework that achieves improved performance is not sufficiently insightful. ICLR standards are high, and this work could benefit from a deeper exploration of these issues for the community.
2. The presentation quality could be enhanced, especially regarding punctuation usage. For example, there is a missing comma after Eq. (5) and an extra period before Eq. (6).

questions:
1. Is the metric in Eq. (1) for machine unlearning reasonable? In the case of a well-performing pre-trained model, strong generalization should allow it to handle unseen samples effectively without degradation.
2. In Tables 2 and 5, the efficacy metric is the retraining time. However, retraining time can vary due to multiple factors, such as I/O rates and the number of GPU cores, which may affect the reliability of this metric.
3. Given that cross-entropy loss is included in the retraining, why is the processing time less than other baselines?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730558606613

thread_type:
official_review

<thread_object>
creator: reviewer_9czd

<content>
summary:
The paper introduces a novel framework for machine unlearning, which is the process of removing the influence of specific training samples from a trained model. The authors propose a method called contrastive unlearning that leverages representation learning to effectively eliminate the impact of unlearning samples while maintaining the model's performance on the remaining data. The approach contrasts the embeddings of unlearning samples with those of the remaining samples, adjusting the model's representation space to make the unlearning samples' embeddings similar to those of unseen test samples. The paper concludes that contrastive unlearning is a promising technique for machine unlearning, with potential applications in complying with privacy regulations. Through extensive experiments on various datasets and models, the paper demonstrates that contrastive unlearning outperforms existing methods in terms of unlearning efficacy, model performance preservation, and computational efficiency.

strengths:
1. The paper introduces contrastive learning to the machine unlearning, effectively removing the influence of unlearning samples without significant loss in model performance. Contrastive unlearning is computationally efficient, requiring fewer iterations to achieve the desired unlearning effect compared to other methods.

2. The experiments are comprehensive, including accuracy gap and membership inference attack for unlearning performance. They convinced me of the SOTA performance of the proposed contrastive paradigm.

weaknesses:
1. Weak scalability compared to non-contrastive methods. The proposed contrastive learning method constructs the positive pair and negative pair in a batch, hence, it might need a large batch size in the practical application which has a large class number. So I suggest using a MoCo-like, more advanced contrasive learning method to further enhance the scalability of the proposed method. 

2. The assumption for test samples is a little bit strong, e.g., " If the embeddings of the unlearning samples become indistinguishable from the embeddings of the test samples, we can claim that the model is no longer influenced by the unlearning samples". We usually assume the training and test examples follow I.I.D. However, it seems that this paper did not have this assumption.

questions:
1, Could you provide some discussion for the limits regarding the number of unlearning samples. More unlarning samples in the contrastive framework might damage the representation of ramaining samples.
2. Following the weakness 2, do the existing methods have the same assumption as line 62-65?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730897756548

thread_type:
official_review

<thread_object>
creator: reviewer_hf8q

<content>
summary:
In this paper, the authors propose a new machine unlearning method, inspired by contrastive learning. 

The proposed contrastive unlearning pushes away the positive pairs (from the same class) and pulls the negative pairs close to each other. 

The idea is validated on several backbones and datasets.

strengths:
- It seems that the contrastive paradigm is new for the field of unlearning. 
- The experimental results, from both efficiency and efficacy, seem to show the advantages of the proposed contrastive unlearning (while I'm quite confused about *unlearning acc*, which will be elaborated in Weaknesses and Questions).

weaknesses:
I'm not familiar with machine unlearning. I read some papers from recent ML conferences, such as ICML, NeurIPS, and CVPR, and try my best to provide a fair review. Some comments may be naive and I'd like to update my score after reading the response from the authors and reviews from other reviewers. Here are my concerns:

- I'm quite confused about the setting of single class unlearning. If we expect a model to forget a class, why don't we just add some rules? For a simple classification task, the rule could be very simple: assign random labels if the model outputs the class to unlearn. The rule may be useless for more powerful models, such as zero-shot classifiers (e.g., CLIP) and LLMs. However, the authors fail to conduct experiments on these models. 

- Some visualization may be helpful for the "contrastive" unlearning. It is quite common to show some t-SNE visualization in contrastive learning. With the visualization, we can know whether the unlearning samples are pushed to the decision boundary. 

- In Eq. (5) and Eq. (6), $z_i$ is not normalized, which is a standard step in contrastive learning. In CL, $z_i$ is usually processed by $z_i = \frac{z_i}{\|\|z_i\|\|}$. Is it a special setting to ensure the performance?

- The theoretical analysis is lacking. Although it is not necessary, some theoretical analysis may help to improve the quality of this paper. 

- There are some typos, such as  $ \mathcal D_{ts} \cup \mathcal{D}_{tr} = \emptyset $ in Line-198.

questions:
- Why do the authors test the single class unlearning only on simple classification models? 
- How will the embeddings of unlearning samples change? Are they really pushed to the decision boundary? 
- If we normalize the representation $z_i$ in Eq. (5) and (6), how will the performance change?
</content>
</thread_object>

</thread>


Metareview: