Multimodal Unlearnable Examples: Protecting Data against Multimodal Contrastive Learning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 16:56:10

Abstract: Multimodal contrastive learning (MCL) has shown remarkable advances in zero-shot classification by learning from millions of image-caption pairs crawled from the Internet. However, this reliance poses privacy risks, as hackers may unauthorizedly exploit image-text data for model training, potentially including personal and privacy-sensitive information. Recent works propose generating unlearnable examples by adding imperceptible perturbations to training images to build shortcuts for protection. However, they are designed for unimodal classification, which remains largely unexplored in MCL. We first explore this context by evaluating the performance of existing methods on image-caption pairs, and they fail to effectively build shortcuts due to the lack of labels and the dispersion of pairs in MCL. In this paper, we propose Multi-step Error Minimization (MEM), a novel optimization process for generating multimodal unlearnable examples. It extends the Error-Minimization (EM) framework to optimize both image noise and an additional text trigger, thereby enlarging the optimized space and effectively misleading the model to learn the shortcut between the noise features and the text trigger. Specifically, we adopt projected gradient descent to solve the noise minimization problem and use HotFlip to approximate the gradient and replace words to find the optimal text trigger. Extensive experiments demonstrate the effectiveness of MEM, with post-protection retrieval results nearly half of random guessing, and its high transferability across different models.

Main paper: 
 related work 2.1 multimodal contrastive learning: 
 Initially designed for self-supervised representation learning in unimodal contexts, contrastive learning methods aim to improve the agreement between views augmented differently in the same instance while reducing the agreement between views of distinct instances [5, 8, 16, 18, 29] . Recently, these techniques have been extended to multimodal domains, notably in the context of paired image-text datasets. Multimodal contrastive models like CLIP [31] and ALIGN [21] have undergone extensive pretraining on vast datasets including hundreds of millions to billions of image-text pairs. Their objective is to maximize the agreement between representations of matched image-caption pairs while minimizing agreement for non-matched pairs. As a result, these models have exhibited exceptional performance in zero-shot classification tasks and have shown robustness to distributional shifts.Data poisoning attacks aim to disrupt the model training process by processing the training dataset, resulting in a significant increase in test errors for some specific samples during the testing phase. [7, 36] .A common type of data poisoning attack is the backdoor attack [7, 15, 26] , which typically involves injecting triggers into training samples, leading to misclassification of images containing these trigger patterns during testing. However, it typically only affects samples with trigger, while clean samples remain unaffected and can be correctly classified. Recent work also explores poisoning attacks on multimodal contrastive learning (MCL). Yang et al. [43] study the poisoning attacks against multimodal models in both visual and linguistic modalities. Carlini and Terzis [3] introduced a framework that effectively poisoned CLIP models with backdoor attacks. In addition, some works aim to train a robust CLIP model against data poisoning and backdoor attacks [1, 42] . Due to the time-consuming of training a high-performance model with a largescale dataset, we will follow the experimental setting of these works to train a rather simpler model with a small dataset in this paper. [12, 37] . The error minimization (EM) [20] minimizes the classification error of images on a surrogate classifier and iteratively updates the surrogate model with these perturbed images. Fowl et al. [13] propose adversarial poisoning, which contains targeted adversarial perturbation (TAP) and untargeted adversarial perturbation (UAP), which uses adversarial examples as poisoned data to make the model unlearn the features. TAP presents an effective protection for the image dataset. While model-based methods are potent, they are often computationally demanding. Some findings reveal that these approaches can be easily neutralized by adversarial training (AT). To address this, several works attempt to leverage robust protection against AT [11, 14] . Model-free methods do not require a surrogate model to generate noise for images. Yu et al. [45] empirically investigate various unlearnable methods and show that all of them use these spurious features to create a shortcut in the model. They also propose the Linear-sperate Synthetic Perturbation (LSP) in response to this characteristic and show great effectiveness. Autoregressive poisoning [33] proposes a generic perturbation generation with each class that can be applied to different datasets, which is a series of dataset-independent perturbations. CUDA [32] is generated using controlled class-wise convolutions with filters that are randomly generated via a private key. These previous works focus on image classification using cross-entropy loss, and He et al. [17] explore the unlearnable examples for unsupervised contrastive learning and discover that extended EM and TAP methods can still effectively protect against unsupervised learning. However, when extending them to multimodal contrastive learning, their effectiveness diminishes due to the increased modality. 
multimodal unlearnable examples 3.1 preliminaries: 
 Our scenario involves two main parties: the protector, and the hacker. We assume that the protector is aware that multimodal privacy data may be unauthorized used by hackers. Consequently, the protector takes measures to render the samples unlearnable by performing operations on the data set. Subsequently, the protector releases these unlearnable multimodal examples to the Internet. Then, when hackers crawl the multimodal data from the web to train multimodal models from scratch with an initial model or fine-tune the pre-trained model, these unlearnable examples will prevent the model from learning the features of private data, and present a poor representation of features across modalities. In this paper, we take the image-text pair as multimodal data as an example, and choose the CLIP [31] as the model used by hackers, which is the most representative multimodal contrastive learning framework.Here, we formulate our problem: we begin by considering a personal image-caption dataset D ⊂ I × T that comprises pairs (𝐼 𝑖 ,𝑇 𝑖 ), where 𝐼 𝑖 is an image and 𝑇 𝑖 is the associated caption. The protector aims to generate an unlearnable set D 𝑢 that contains the unlearnable image-text pair 𝐼 ′ 𝑖 ,𝑇 ′ 𝑖 , and make a CLIP model 𝑓 * trained on it generalize poorly on the clean distribution D:arg max E (𝑰 ,𝑻 )∼D L 𝑓 * (𝑰 , 𝑻 ) , s.t. 𝑓 * ∈ arg min 𝑓 ∑︁ (𝑰 ′ ,𝑻 ′ ) ∈ D 𝑢 L 𝑓 (𝑰 ′ , 𝑻 ′ ) (1)Directly solving Eq. 1 is intractable for deep neural networks, and recent works have designed multiple approximate solutions. We can follow these works and extend them to MCL. However, all model-free methods for classification fail to generate image noise here because these methods aim to find a series of specific noise patterns for images related to one certain class, while there is no label in the image-caption pair data. Therefore, only model-based methods can be applied to the MCL, and we extend two typical methods to generate unlearnable multimodal examples. Specifically, they leveraged the addition of a set of perturbations Δ on the images and employ an 𝑙 ∞ to bound for each noise 𝛿. However, from Table 1 , we experimentally found that although EM and UAP can be applied to image-caption pairs, they fail to achieve highly effective protection, especially UAP. We explore the reasons for the decline in the effectiveness of these methods from image classification to multimodal contrastive learning. In image classification, EM and TAP optimize the images with the same label to converge in a feature space, resulting in the model easily capturing these additive noises and learning the correlation with labels, as shown in Fig. 2 (a). However, in multimodal contrastive learning (MCL), to effectively apply the EM and UAP methods, the direction of the optimized image noise must relate to the features of the caption, causing the image features to become either close to or far away from these features. Nevertheless, the caption features of different pairs may be widely dispersed in the image-caption dataset. Consequently, as illustrated in Fig. 2 (b ) and (c), it becomes more challenging for the model to capture the correlation between captions and noise generated by EM and TAP compared to those in classification. In Fig. 2 (c ), the learning decision space of UAP is much more complex, so its poor protection is to be expected.In the previous section, we showed that the model-based methods still fail to achieve effective protection due to the dispersion of image-text pairs. An intuitive strategy of enhancement entails optimizing both the image and the captions for a larger optimized space, boosting their convergence across different pairs in the feature space. Consequently, the optimized feature representations of the image and caption set exhibit similar distributions, facilitating the model's learning of their shortcut, as illustrated in Fig. 2 (d) .To this end, we take the EM method as the basic framework and propose adding an additional short text trigger to the caption to minimize contrastive loss, following the setting of an adversarial attack on text tasks [41] . Regarding the length of the trigger, longer triggers are more effective, while shorter triggers are more stealthy. Therefore, our method can be conceptualized as a tri-level iterative optimization problem, resembling a multi-step process of EM. Specifically, we sequentially optimize noise 𝛿 and text trigger 𝑡 to reduce contrastive loss between optimized images 𝐼 + 𝛿 and optimized text 𝑇 ⊕ 𝑡, where ⊕ denotes the triggers that can be inserted into clean text 𝑇 at various positions. For simplicity, we choose to add text triggers at the beginning of the text in this paper. As a result, our Multi-step Error Minimization (MEM) method can be formulated as follows:EQUATIONWe can iteratively optimize the above problem sequentially by referring to the method in EM. We use projected gradient descent (PGD) [27] to solve the noise minimization problem in Eq. 4. Notably, to mitigate the noise overfit to the clean captions, we augment them by shuffling the clean caption in a batch and adding the correct matching text triggers on them. Thus, this generated noise can focus more on the text trigger than on part of the captions, when facing the semantic wrong captions. So we can get the optimal 𝛿 according to the following iterative formula,EQUATIONwhere ∇ 𝛿 𝑡 L is the gradient of the loss w.r.t 𝛿 𝑡 . 𝛼 is the step size and 𝑃𝑟𝑜 𝑗 () denotes the project of 𝛿 within the bound of the norm (-𝜖, 𝜖). S(•) means shuffle the order of the clean caption samples in a batch with each iteration. For the text trigger minimization problem, we first initialize the trigger sequence by repeating the word "the" or "a" to the front of all inputs. In addition, we optimize the text trigger based on HotFlip [9] , a method that approximates the effect of replacing a token using the gradient. We denote a text trigger as 𝑡 𝑖 which is a hot vector and can be embedded in the form 𝑒 𝑖 . Thus, we can update the embedding for every trigger token 𝑒 𝑖 to minimize the first-order Taylor approximation of CLIP loss around the current token embedding:arg minEQUATIONwhere V is the set of all token embeddings in the vocabulary and ∇ e 𝑡 is the gradient of loss w.r.t 𝑒 𝑖 . We can calculate a set of candidate tokens 𝑒 ′ 𝑗 using a dot product with embedding of the vocabulary token and gradients. Finally, we can search for each optimal text trigger with a beam search in a set of candidate tokens. We consider the top-k candidates from Eq. 6 and search front to the last in every position in the trigger and score each beam using the loss on the current batch. We follow Wallace et al. [41] and use a small beam size for efficient computation. The pseudocode of our method is We then compute the gradient with current images and tokens. We update the images using Eq. 5 and update the text triggers with Eq. 6. The solid line represents forward propagation, and the dashed line represents backward propagation.shown in Algorithm 1. In Fig. 3 , we can see the framework of the generation of multimodal unlearnable examples with our MEM. 
experiment 4.1 experiment setup: 
 Target models and datasets. Following previous works [1, 42, 43] ,we adopt the open-source implementation of CLIP in the community 1 . For the architecture of the surrogate model, we choose the model with ResNet50 [19] as the image encoder and a Transformer [40] with certain architecture modifications serving as the text encoder. Our experiment involves three datasets: Flickr8K [44] , Flickr30K [44] , and MS-COCO [6] . Specifically, Flickr8K and Flickr30K consist of 8000 and 31,000 images, respectively, each accompanied by five captions. We adhere to the protocol established by [10, 22, 47] , dividing the datasets into training/validation/testing sets with ratios of 6,000/1,000/1,000 and 29,000/1,000/1,000 for Flickr8K and Flickr30K, respectively. MS-COCO comprises 123,287 images, each annotated with five descriptions. As per [22] , MS-COCO underwent division into 82,783 training images, 5,000 validation images, and 5,000 test images. Despite their smaller size compared to the large-scale datasets used to train the original CLIP model, these datasets remain suitable for storage and computational resources and have found widespread usage in MCL studies [42, 43] .Training and Attack Setup. Initially, we focus on training models from scratch in the main experiments and reserve discussion of the fine-tuning scenario to the case study on face privacy protection.1 https://github.com/mlfoundations/open_clip Following the training setup of the CLIP model in [1, 31] , we choose a batch size of 128 and set the initial learning rate to 0.0005. The weight decay rate is set to 0.2, and we employ an Adam optimizer with decoupled weight decay regularization, while the learning rate decays using a cosine scheduler. We train these models from scratch on 1 A100 GPU for 32 epochs. During the attack stage, we extend the EM and UAP methods to MCL. In both cases, we assume that the surrogate models are CLIP models with the ResNet50, while the surrogate model for UAP is already trained on clean data.For our MEM, we choose an initial surrogate CLIP and iteratively optimize the noise and text trigger, stopping when the loss is below a threshold. We select a threshold of 0.01. Regarding the text trigger, we set the text sequence lengths as three and five, denoted MEM-3 and MEM-5, respectively. Typically, we set the noise bound of all methods with the 𝑙 ∞ -norm 𝜖 = 8/255. Evaluation Metrics. We evaluate the impact of data protection by examining the decline in performance in image and text retrieval tasks, drawing from previous work on attacks in MCL [42, 43] . Two metrics are employed to illustrate the retrieval results. Hit@10 measures the proportion of all target images/texts that appear within the top 10 of the list of rank. Higher Hit@10 values indicate that many text/image samples successfully retrieving target images/texts early, reflecting a better rank list. Medr refers to the median position of all target images / texts in the list of test images/texts. Lower Rank values signify earlier access to target images, indicative of a superior rank list. The performance of unlearnable multimodal examples is assessed using Hit@10 and Medr for image retrieval (Image → Text) and text retrieval (Text → Image) across all testing images. Lower Hit@10 and higher Medr-Rank values signify more effective protection of the data.In this section, we compare our method with extensions of existing popular unlearnable methods, including EM [20] and UAP [13] . Table 1 presents their retrieval results on different datasets. It is evident that UAP nearly fails to provide any protection for multimodal data, while EM demonstrates a certain level of protection. Moreover, as the size of the dataset increases, the effect of EM improves due to the denser caption distribution in the feature space, consistent with our analysis in Section 3.2. However, our MEM consistently offers robust protection for multimodal data, reducing retrieval performance to nearly half random guessing. In particular, MEM-5, with a longer text trigger, obtained greater results in reducing the performance of the hacker model compared to MEM-3, likely because longer text triggers enable the model to focus more effectively on the trigger and establish a shortcut. Fig. 4 illustrates the descending curves of the training loss trained on unlearnable examples generated by different methods and the retrieval Medr on the clean test set. From (a), we observe that although EM enables the loss to fall faster compared to normal training, our method, MEM-3 and MEM-5, have a smaller loss on the first epoch, suggesting that the model can quickly learn the shortcuts. The UAP hardly speeds up the loss decline significantly, so it is understandably inefficient. From (b), we find that all models are trained with Medr decreasing compared to random guessing, but the model trained on unlearnable examples generated by MEM stops learning the fastest, reaches the worst retrieval result, and does not learn better further as the epoch increases. The above observations are consistent with the results in Table 1 .We assume that data protection is a completely black-box setting, wherein the protector lacks knowledge of the hacker model's architecture. Thus, we evaluate the performance of our MEM generated on the ResNet50 surrogate model on different hacker models, including ResNet101, and ViT-B/32. The results are presented in Table 2 . We find that these examples can be successfully transferred across different models and can degrade the performance of the CLIP model. Although they generated a surrogate model with a structure different from that of the hacker model can also exhibit the same protective effect, this observation aligns with the one of unlearnable examples in image classification. This is likely because the feature preferences learned by the models are similar.Attention of the models. Fig. 5 presents the heatmaps of images and text, illustrating the attention of models trained on clean and unlearnable examples. The Grad-CAM [35] is utilized to visualize the model attention for images, while the Integrated Gradients [38] is employed to visualize the attention to the text. Lighter colors represent higher attention from the model. Remarkably, for the image, models in 4 shows, our protection power gradually diminishes as the percentage of unlearnable examples decreases. As identified by previous work in classification [13, 20] , the effect of them will quickly drop when the data are not 100% poisoned, and as the number decreases to 50% they have almost no effect. In the case of MCLs, when they only constitute 20% of the dataset, some level of protection is still observed. This difference may be attributed to the dispersion of image-text pairs, unlike in classification where clean images may correct the images with perturbation, resulting in a rapid decline in protection efficacy. This finding underscores the superior robustness of unlearnable multimodal examples.In the previous methodology, we propose to use the gradient by the Hotflip method to select embeddings in the vocabulary list with large inner products to serve as candidate words for replacement. However, this approach may result in the generation of candidate words lacking semantic information, potentially impacting social reading or being detected and removed by hackers. Therefore, here we leverage the pre-training parameters of the BERT model to generate some semantically relevant substitutions as vocabulary lists. Subsequently, we embed these substitutions to obtain V bert , and then compute the inner product with the gradient and token embeddings of the substitutions, which yields replaced words with semantics. Table 4 displays examples of text triggers generated by our MEM-3 method under different datasets, including those generated with and without the BERT method. Text triggers generated with BERT exhibit greater naturalness and semantic coherence compared to those generated without BERT. Additionally, we show the Medr results of these triggers in text-image retrieval. It is observed that triggers generated without BERT have better protection performance, which is attributed to their larger search space, while triggers generated with BERT are restricted to a vocabulary space V bert , resulting in inferior protection efficacy.Here, we conduct a case study to apply our MEM noise to a realworld scenario: protecting personal face images and associated information on social media platforms, such as names. This scenario is arguably one of the most common multimodal privacy protection scenarios. In previous sections, we assumed that hackers train models with multi-modal unlearnable examples from scratch. However, in most real-world scenarios, hackers will opt to fine-tune pre-trained models provided by the community. Therefore, here we assume that the protector aims to prevent their facial images and names from being fine-tuned with a pre-trained CLIP model. We adopt the Open AI's released parameters to initialize our pretrained models. We still assume the protector has access to their image and text data to generate multimodal unlearnable examples before sharing them on on-line social media platforms. However, they are unable to know the hacker's model architecture and the parameters of pre-trained models. In this black-box scenario, we explore whether our method can still successfully prevent the CLIP model from learning the protector's face images. We conducted experiments using the PubFig database [23] , a large real-world face dataset comprising 58,797 images of 200 individuals collected from the internet. For retrieval evaluation, we randomly selected one photo of each celebrity as the test set and utilized all remaining images for training. It's noteworthy that the pre-trained CLIP model already possesses knowledge of the real names and faces of these celebrities. Therefore, for authentic finetuning, we altered their names and provided a set of text templates relevant to the name for caption generation, as shown in Fig. 7 . Subsequently, we generated unlearnable examples using MEM and employed a pre-trained surrogate model with ResNet50 as the image encoder. We fine-tune the pre-trained models for 10 epochs and a small learning rate of 0.00001 and evaluate them with different hacker models. The results are presented in Table 5 . For comparison, we also test the results of the pre-trained models and the benign fine-tuned models. Table 5 shows that the initial model shows poor performance on the test set, since the face and name features are not aligned. However, with only 10 epochs of fine-tuning, the Medr of its retrieved results reaches 1. This may be because of the strong feature representation of the pre-trained model, enabling it to swiftly adapt to the new task during the fine-tuning process. However, our MEM approach can prevent these fine-tuned models from learning the correlation between face and name features, thereby impeding accurate person retrieval on the test set. Our unlearnable examples were generated using the ResNet50 surrogate model and effectively protect against attacks on the ResNet101 hacker model. However, their efficacy is considerably reduced on Vision Transformer, possibly due to significant variations in architecture and initialization parameters between models. 
conclusion: 
 In this paper, we explore multimodal data protection, particularly focusing on image-text pairs, wherein we generate multimodal unlearnable examples to prevent exploitation by MCL. We extend previous classification methods to this context, revealing their limitation in MCL due to the increased modality and data dispersion. In light of these findings, we introduce a novel generation approach named Multi-step Error Minimization (MEM), which is based on the framework of EM. MEM effectively establishes a shortcut between noise and textual triggers and demonstrates transferability across different hacker models. Additionally, we utilize various visualization tools to validate the effectiveness of our approach. Finally, we present a case study on face-privacy protection aimed at preventing fine-tuning on pre-trained models with personal faces and identifying information. Our work opens up a new direction, and our methods are expected to be applicable to other modal pairs, such as audio-text and audio-image pairs.  
: 
 unified framework. One of the most popular approaches for multimodal learning is Multimodal Contrastive Learning (MCL), as demonstrated by models such as CLIP [31] and ALIGN [21] . These models are trained with a contrastive loss, which encourages the correlation between pairs of images and captions while also keeping them distinct from unrelated pairs. This approach reduces the need for extensive manual annotation of training data and allows the use of larger datasets that contain millions of examples. MCL has shown promise in various applications, including image classification [31] , image captioning [24, 28] , image generation [25, 30] .Training of high-performance multimodal models is highly dependent on large amounts of multimodal data, often sourced from publicly available datasets such as CC12M [4] , YFCC100M [39] , and LAION5B [34] . However, as the demand for larger datasets continues to surge in the future, these datasets may be still insufficient. Consequently, malicious actors may resort to unauthorized data acquisition from the web or engage in the crawling of user posts on social networks for commercial training purposes. However, these datasets often contain significant amounts of sensitive personal information, raising concerns among people about the potential unauthorized use of personal data and the leakage of user privacy information.A series of recent works make efforts to prevent unauthorized usage in image classification by making the image unexploitable. Specifically, they poison the data with some imperceptible perturbations, creating 'shortcuts' [45] in the training process that hinder the models from learning the features of the images [46] . This kind of attack is called availability attacks or indiscriminate poisoning attacks, and these poisoned training data are called unlearnable examples. These unlearning methods can be broadly classified into two categories: model-free and model-based attacks. Model-free attacks usually generate unlearnable noise at the pixel level without any knowledge of clean data and directly create shortcuts between image noise and labels, such as LSP [45] and CUDA [32] . Due to their direct association with labels, these patterns often exhibit high efficiency. Model-based attacks typically generate noise through surrogate models. The surrogate model learns the features through the training phase and generates the feature-level noise, such as Error-Minimizing [20] and Adversarial Poisoning [13] . However, there has yet to be research to consider protecting multimodal data in the context of MCL.We are the first to consider a scenario focused on generating multi-modal unlearnable examples against privacy risks associated with MCL. In this context, we concentrate on image-text pairs as a representative multimodal dataset. Users are assumed to frequently share personal photos with text on social media platforms like Facebook, including some private identity information such as faces, names, phone numbers, and addresses. Currently, hackers attempt to collect large amounts of such image-text pairs from the Internet and utilize MCL techniques to train modern foundational models, as illustrated in the left segment of Fig. 1 . These models inadvertently capture user's private information and facial characteristics, leading to potential privacy leakage. Protectors aim to prevent the unauthorized exploitation of these sensitive data by performing unlearning methods on multimodal data. These methods aim to render models trained on such multimodal unlearnable examples incapable of accessing users' privacy features, while not impeding users' social interactions after posting images and text, as depicted in the right segment of Fig. 1 .An intuitive idea to generate unlearnable multimodal examples involves extending the unlearning methods for image classification to MCL. However, we explore the performance of these methods on multimodal data, and all of them fail to present effective protection due to increased data modalities or dispersion of data pairs. For model-free attacks, they fail to generate specific noise patterns that strongly correlate with the category for a shortcut due to the lack of certain labels in the image-text pair. For model-based attacks, while it may be feasible to optimize noise to build shortcuts with clean captions, their efficacy is significantly diminished. Our analysis was primarily attributed to the dispersion of the data pairs, which presents challenges in learning the noise pair and captions compared to the images and labels in classification, as depicted in Fig. 2 . Therefore, establishing more efficient shortcuts is key to generating effective multi-modal unlearnable examples.In this paper, we propose a novel optimization framework that efficiently generates unlearnable multimodal examples for imagecaption pairs. We first adopt the Error-minimizing (EM) [20] framework as a basis for our attack, and we consider optimizing the noise along with an additional text trigger. Specifically, following adversarial triggers in NLP [41] , we propose to add short text sequences as triggers in the front of the clean caption, which does not affect the understanding of the text by the user. Therefore, it can be formulated a multi-step minimization problem to optimize the noise-text trigger pair and build the shortcut between them, which we dubbed Multi-step Minimization (MEM), to prevent the unauthorized model from learning the features of images and captions. During optimization, we consider adopting projected gradient descent (PGD) [27] to solve the noise minimization problem and use the HotFlip [9] method to approximate the gradient and replacement strategy in [41] to select the optimal trigger in the text minimization problem.  
 
 Appendix:  


Decision: A good problem to study and the approach was deemed novel by reviewers.
Two out of three reviewers voted for acceptance, while the other one has reservations. However, looking at that reviewer's remaining concerns, they do not seem to be fundamental, technical issues. Hence I'm siding with the majority opinion.

Reviews: review: paper_summary: paper_summary: In this paper, the authors propose a novel approach to protect multimodal data, specifically image-text pairs, from unauthorized use in Multimodal Contrastive Learning (MCL) models. The authors propose a Multi-step Error Minimization (MEM) method that generates unlearnable examples to prevent models from learning private information. The effectiveness of the MEM approach is demonstrated through extensive experiments.
strength_weakness: strength_weakness: 1.	The paper introduces a new scenario for multimodal data protection against MCL, addressing a gap in current research.
2.	The paper provides a detailed analysis of the limitations of previous methods when applied to multimodal data, offering clear insights into the challenges and their proposed solutions.
3.	This paper is easy to follow.
limitations: limitations: 1.	The paper does not discuss the efficiency of the proposed MEM method compared to existing methods, particularly in terms of computational complexity and resource requirements.
2.	The approach focuses on image-text pairs, which may limit its applicability to other types of multimodal data such as audio-visual pairs or text-sensor data combinations.
paper_summary: paper_summary: This paper proposes MEM (Multi-step Error Minimization), a new method to create unlearnable examples for privacy protection in Multimodal Contrastive Learning. MEM builds on the EM framework to optimize image noise and text triggers, hindering models from learning the shortcut between the noise and the trigger.The article is well-written, with clear and concise language that is easy to follow.
strength_weakness: strength_weakness: 1. I like Figure 1, as it provides an excellent and clear explanation.
2. The article is well-written, with clear and concise language that is easy to follow.
3. The approach of simultaneously optimizing images and text to generate unlearnable examples is novel and demonstrates effective utilization of knowledge in the multimedia domain.
limitations: limitations: 1. I suggest moving the pipeline up and spending more space introducing it. In the current version, it is challenging for readers to grasp the implementation details of the proposed method. The methodology section should be expanded to provide more elaborate explanations. 
2. In line 494, there is no Algorithm 1.
3. Given the existing exploration of defending against availability attacks in classification tasks, I believe it would be beneficial to the community to include a discussion or supplement on how to defend against availability attacks in multimodal tasks.
paper_summary: paper_summary: This paper presents an unlearnable example method for protecting the privacy of image-text pairs from being learned by DNNs in multi-modal scenarios. Experimental results on benchmark datasets demonstrate the effectiveness of the proposed approach.
strength_weakness: strength_weakness: 1.The novelty of the scenario. The consideration of unlearnable examples in multi-modal contexts in this paper is quite novel to me, and it holds significant practical implications in real-world scenarios.

2.This paper is well-written and easy to understand.
limitations: limitations: 1.The puzzling Fig 1. The authors claim in Fig 1 that the proposed unlearnable example can prevent the model from learning features of private information. But what if it's proven that the model hasn't learned these features of private information? How can we be certain of this? There are also no relevant metrics in the experiments to assess this.

2.Inaccurate statement. Lines 122-123 and Lines 251-252 incorrectly equate the concepts of availability/indiscriminate poisoning attack with the unlearnable example attack. In fact, availability/indiscriminate poisoning attack should be categorized based on the attack's objective, where any attack resulting in a decrease in final test accuracy is termed as availability/indiscriminate poisoning attacks, including clean-label scenarios (equivalent to unlearnable examples) and dirty-label scenarios [1].

3.Self contradictory statement. In Lines 157-158, the authors claim that the methods for unlearnable examples in image classification all fail to be effective in multimodal data, but it is evident from Table 1 that the transferred methods EM and UAP are effective.

4.Time complexity of proposed approach. In Lines 167-168, the authors mention the importance of more efficient generation of shortcut for multimodal data. Therefore, I am curious about the efficiency of the proposed solution and its comparison with other methods. However, I did not find any discussion on efficiency in the experimental section.

5.Confused definition. In Lines 226-228, the authors define the target of data poisoning attacks as the decrease in accuracy for specific samples, which is not the objective of indiscriminate/availability poisoning attacks. The indiscriminate/availability poisoning attacks aim to indiscriminately reduce the accuracy of samples in the test set.

[1] Support vector machines under adversarial label noise. In: Proceedings of the 3rd Asian Conference on Machine Learning. (ACML’11)