Captured by Captions: On Memorization and its Mitigation in CLIP Models

Authors: Wenhao Wang, Adam Dziedzic, Grace C. Kim, Michael Backes, Franziska Boenisch
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: memorization, multi-modal, clip, vision language models

Abstract:
Multi-modal models, such as CLIP, have demonstrated strong performance in aligning visual and textual representations, excelling in tasks like image retrieval and zero-shot classification. Despite this success, the mechanisms by which these models utilize training data, particularly the role of memorization, remain unclear. In uni-modal models, both supervised and self-supervised, memorization has been shown to be essential for generalization. However, it is not well understood how these findings would apply to CLIP, which incorporates elements from both supervised learning via captions that provide a supervisory signal similar to labels, and from self-supervised learning via the contrastive objective.
To bridge this gap in understanding, we propose a formal definition of memorization in CLIP (CLIPMem) and use it to quantify memorization in CLIP models. Our results indicate that CLIP’s memorization behavior falls between the supervised and self-supervised paradigms, with "mis-captioned" samples exhibiting highest levels of memorization. 
Additionally, we find that the text encoder contributes more to memorization than the image encoder, suggesting that mitigation strategies should focus on the text domain. 
Building on these insights, we propose multiple strategies to reduce memorization while at the same time improving utility---something that had not been shown before for traditional learning paradigms where reducing memorization typically results in utility decrease.

Decision: accept (poster)
Reviewer scores: 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1729817621106

thread_type:
official_review

<thread_object>
creator: reviewer_f6jj

<content>
summary:
They study the memorization problem in the CLIP model unlike existing studies focusing on the unimodal memorization problem. 
They propose a new metric to analyze it and find that memorization seems to be more significant in text encoder than in image encoder. Their analysis indicates that augmenting captions can be a key to mitigating memorization in the CLIP model. Their experiments confirm that augmenting captions can improve the quality of the image encoder's representations while reducing memorization.

strengths:
1. They propose a new metric to measure memorization in the CLIP model. The design of the metric is reasonable. 
2. Their insight that memorization is more significant in the text encoder is new and might interest readers. 
3. They conduct analysis on the augmentation of the text and images, which might be also interesting to some readers. 
4. This paper is well-organized and easy to follow.

weaknesses:
1. They do not discuss how model size can affect memorization. Although I am not very familiar with this topic, I guess the model size can affect their arguments. For example, if they utilize a larger image encoder, the memorization might be more significant on the image side. Therefore, I think their conclusion about which encoders suffer more from memorization can change by the size of the encoders, but they do not discuss much. 

2. Most of their findings sound a bit too reasonable and are not surprising. Their finding that augmenting text improves the CLIP model has already been observed in many previous papers though they probably did not discuss memorization. Also, it is not hard to imagine that augmenting datasets mitigates memorization. In these points, I think their findings are not impressive. 

3. They mention that their metric is effective in removing noisy samples. But, they compare their approach only with random replacement. I think they need to add more baseline, such as naive CLIP's similarity as done in many works. 

4. In Table 1, the authors augment images by using a diffusion model. But, as they imply, such augmentation can cause a distribution shift in the image side and does not give much intuition about image augmentation. 

Overall, I think this paper is well-organized and delivers a clear statement to readers, which I like. However, I think their findings are not very surprising and lack impact due to the reasons described in 1, 2. My rating is based on it.

questions:
My rating is mainly based on 1 and 2. Please respond to those points.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730628905920

thread_type:
official_review

<thread_object>
creator: reviewer_cajs

<content>
summary:
The paper introduces CLIPMem, a novel metric to quantify memorization in CLIP, which combines elements of supervised and self-supervised learning. The paper shows that memorization within CLIP often arises from mis-captioned or atypical samples, particularly within the text modality rather than the image modality. They propose mitigation strategies that reduce memorization without sacrificing model utility, unlike traditional methods that often degrade performance when reducing memorization. It reports several interesting findings, including 1) CLIP's memorization lies between supervised and self-supervised paradigms, with high memorization for data with inaccurate or misaligned captions; 2) Text domain adjustments, such as using varied or augmented captions, reduce memorization and improve generalization, defying the usual trade-offs seen in other paradigms.

strengths:
+ It introduces a new metric -- CLIPMem to provide a new way for measuring memorization in multi-modal settings, a gap in previous research.
+ It performs empirical analysis to show differences in memorization between the text and image modalities, providing actionable insights.
+ It proposes techniques to successfully reduce memorization while preserving or even enhancing model utility, challenging established norms.
+ By highlighting the risks of training with uncurated, potentially mis-captioned data, the paper suggests guidelines that can benefit real-world multi-modal model training practices.

weaknesses:
- While tailored to CLIP, the metric and findings may need adaptation to apply effectively to other multi-modal models with different architectures.
- The experiments focus on datasets like COCO and CC3M, so it’s unclear how well these findings generalize to other large-scale or domain-specific datasets.
- The mitigation strategies, such as augmenting captions or generating variations, may incur additional computational costs in training, which could limit practicality for some users.

questions:
Would you please comment on how the metric adapt to other multi-modal models besides CLIP?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731141210738

thread_type:
official_review

<thread_object>
creator: reviewer_khe4

<content>
summary:
**Summary:** Understanding the memorization / generalization tradeoff is important to properly quantify modern ML models. This work focuses on CLIP which applies InfoNCE between image and text pairs and attempts to quantify the extent to which CLIP models memorize. The authors introduce CLIPMem to quantify memorization and find that the text encoder contributes more to memorization than the vision encoder. They also propose strategies to mitigate / remove memorized samples to improve performance.

strengths:
**Stong points:**
- Well highlighted literature on memorization.
- Defines CLIPMem based on hold one out strategy (similar to Feldman et. al in supervised learning).
- Interesting results on mis-captioned text labels, multi-caption and removal of memorized examples.
- Reasonable pretraining datasets like CC3M
- Clean separation of training and test splits for measuring memorization.

weaknesses:
**Weak points:**
- Missing ability for CLIPMem to be applicable to general off-the-shelf CLIP models. Currently if I understand correctly it requires retraining on specific splits. 
- Clarity of specifics of CLIPMem is used for vision only and joint vision + text can be improved.
- The noising results (Table 5-b) are not very convincing. Almost all the results are within the same +/- std range.
- The linear probe accuracy seems quite low (Table 1, 5-a, 5-b,  6-a/b).  

**Nit:**
- Text and images in Figure one are very hard to read. Suggest larger and fewer images and move rest to appendix. 
- Figure 3 can be make larger / more readable by doing share-y and increasing font sizes.

questions:
**Questions:**

- Is CLIPMem bounded? 
- What dimension is kept constant for Table 1? Are the total training samples [count] seen the same? 
- It would be interesting to evaluate infinite data regimes (i.e. no repeated data) rather than classical K-epoch  runs. Would the results for - memorization hold here? Just like in language this setting is becoming more and more common.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732485184152

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank the Reviewers for their thoughtful and encouraging feedback, which greatly helped us further improve our submission. Our work was described as addressing “a gap in previous research” by offering “a new way for measuring memorization in multi-modal settings (Reviewer CAJS). We are encouraged that the significance of the studied trade-off between memorization and generalization was highlighted (Reviewer khe4), with Reviewer CAJS noting our observed trade-off as “challenging established norms” and stating that the paper “suggests guidelines that can benefit real-world multi-modal model training practices.” Reviewer khe4 also found our experimental results *interesting*, especially the removal of memorized mis-captioned examples, which improves performance, underscoring the practical impact of our findings. Finally, Reviewer F6JJ notes that our “insight that memorization is more significant in the text encoder is new” and that our results and analysis “might interest readers.” We hope that our work will significantly contribute to the community and inspire further research, reaching a broader audience.

1. During the rebuttal, we performed additional experiments and analysis that we present in the individual answers to the reviewers, and update them correspondingly in the paper.
2. Based on the suggestion of Reviewer khe4, we experimented with additional noise strength for caption noising and updated the results in Figure 5 (b) and Table 8. When applying random Gaussian noise with 0 mean and 0.15 standard deviation, the linear probing accuracy is 65.34% ± 0.84%, which is out of the +/- std range of baseline’s 63.11% ± 0.91%. This could now prove our statement “Noising the text embedding during training is highly effective in reducing memorization while improving downstream generalization” in the paper.  
3. Based on the suggestion of Reviewer F6JJ, we compared the effect of noise-sample removal between CLIPMem with naive CLIP's similarity and updated the results in figures 6 (a) and 6 (b) in the paper. The results show that CLIPMem is more effective than using the naive CLIP’s similarity.
4. Based on the suggestions of Reviewer khe4 and Reviewer CAJS, we train new models with infinite data regimes on a 7M subset of YFCC100M, which is a widely used large-scaled multi-modal dataset. The results in Table 6 and Figure 9 indicate that the most-memorized samples (according to CLIPMem) are still obviously mis-captioned samples, which proves CLIPMem is feasible in infinite data regimes training and can be applied to multi-modal datasets besides COCO and CC3M.
5. Based on the suggestion of Reviewer F6JJ, we performed extra experiments on ViT-large/16 to further study the impact of how model size will affect memorization effect. The results in Table 5 show that with more parameters (larger size), encoders will have higher memorization capacity. This aligns with previous research in Supervised Learning and Self-Supervised Learning [1,2,3]. 
6. Following Reviewer F6JJ's suggestion, we conducted additional experiments for Table 1 to further substantiate the claim that augmenting either text or images during training significantly reduces memorization.

**References:**

[1] *”Memorization in self-supervised learning improves downstream generalization”.* Wenhao Wang, Muhammad Ahmad Kaleem, Adam Dziedzic, Michael Backes, Nicolas Papernot, Franziska Boenisch. ICLR, 2024.

[2] *“Does learning require memorization? a short tale about a long tail”.* Vitaly Feldman. 
ACM SIGACT Symposium on Theory of Computing, 2020.

[3] *“Do ssl models have déjà vu? a case of unintended memorization in self-supervised learning.”* Casey Meehan, Florian Bordes, Pascal Vincent, Kamalika Chaudhuri, Chuan Guo. arXiv e-prints, 2023.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732532060790

thread_type:
official_comment

<thread_object>
creator: area_chair_cmp4

<content>
comment:
Dear Reviewers,

This is a gentle reminder that the authors have submitted their rebuttal, and the discussion period will conclude on November 26th AoE. To ensure a constructive and meaningful discussion, we kindly ask that you review the rebuttal as soon as possible and verify if your questions and comments have been adequately addressed.

We greatly appreciate your time, effort, and thoughtful contributions to this process.

Best regards,
AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors study memorization on CLIP models and define CLIPMem as a measure of it. They find that "mis-captioned" samples exhibiting highest levels of memorization and that the text encoder contributes more to memorization than the image encoder. They also propose strategies to reduce memorization and improve performance. 

This result is interesting since it differs from the behavior of other learning paradigms. Reviewers pointed some areas of improvement like exploring the effect of model size, training on infinite data regimes, or experiments with additional noise strengths. The authors did a good job with the rebuttal, and I believe they succeeded at answering all the reviewer’s concerns. However, while khe4 raised their score, CAJS remained inactive during discussion, and F6JJ decided to keep their score of 5 arguing that results are not surprising.

During reviewer/AC discussion, F6JJ restated their concern and khe4 replied that this work is good science and valuable in itself. I also believe that the fact reducing memorization in CLIP increases performance differs from behavior previously seen in other paradigms. 

Overall, and given the enthusiasm of khe4, I believe this is a valuable work and it should be accepted to ICLR 2025.