PAIR: Pre-denosing Augmented Image Retrieval Model for Defending Adversarial Patches

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 03:03:48

Abstract: Deep neural networks are widely used in retrieval systems. However, they are notoriously vulnerable to attack. Among the various forms of adversarial attacks, the patch attack is one of the most threatening forms. This type of attack can introduce cognitive biases into the retrieval system by inserting deceptive patches into images. Despite the seriousness of this threat, there are still no well-established solutions for image retrieval systems. In this paper, we propose the Pre-denosing Augmented Image Retrieval (PAIR) model, a new approach designed to protect image retrieval systems against adversarial patch attacks. The core strategy of PAIR is to dynamically and randomly reconstruct entire images based on their semantic content. This purifies well-designed patch attacks while preserving the semantic integrity of the images. Furthermore, we present a novel training strategy that incorporates a semantic discriminator. This discriminator significantly improves PAIR's ability to capture real semantics and reconstruct images. Experiments show that PAIR significantly outperforms existing defense methods. It effectively reduces the success rate of two state-of-the-art patch attack methods to below 5\%, achieving a 14\% improvement over current leading methods. Moreover, in defending against other forms of attack, such as global perturbation attacks, PAIR also achieves competitive results.  The codes are available at: https://anonymous.4open.science/r/PAIR-8FD2.

Main paper: 
 related work 2.1 adversarial patch attacks: 
 Patch attacks are one of the most threatening forms of adversarial attacks, modifying pixels in a continuous region of an image. AdvHash [15] and TTH [14] have developed patch-based attack methods in content-based and text-based image retrieval systems, respectively. AdvHash inserts an attack patch into the input image, causing the content-based image retrieval system to return irrelevant images. This attack can be used, for example, to protect intellectual property. If the plagiarised works are entered after the attack, the original works cannot be retrieved. TTH inserts adversarial advertising images into the image database of the text-based image retrieval system. After the attack, when users enter text containing specific keywords, the system displays irrelevant advertisements. Adversarial patches can be implemented in a variety of image carriers, such as logos, QR codes, text, etc., and can appear in any position in the image. They wreak havoc on content-based and text-based image retrieval systems.Defenses designed specifically for classification tasks. The representative method is based on Derandomized Smoothing [7, 16, 28] . Derandomized Smoothing divides the image into several image bands. Since the number of image bands affected by the adversarial patch is limited, the final classification result can be obtained by voting according to the classification result of each image band. Besides, PatchGuard [32] designed a defense mechanism in convolutional network based on small receptive fields. However, these methods are specifically designed for image classification. It is difficult to transfer to image retrieval. Universal defenses against patch attacks. Universal methods are mainly preprocessing methods, which purify and denoise the images before input to the model. Representative methods are LGS [24] , PatchZero [33] , SAC [18] , MAEDefense [22] . LGS [24] developed a smoothing strategy based on the image gradient. It was motivated by empirical observation: the attack patch's pixel value changes dramatically. However, LGS can be easily broken [18] and PatchZero [33] used a patch detector to locate and mask attack patches. This is highly dependent on the performance of the patch detector. If the patch detector is inside the attacker's target and locates it inaccurately, the localization-based defense will fail. Our defense, in contrast, randomly masks and recover the image multiple times. This mechanism guarantees that after purification, all patch pixels are reconstructed and denoised.by white-box attacks, as demonstrated by experiments [8] , if the attacker optimizes the adversarial patch against the model that includes all pre-processing steps. SAC [18] and PatchZero [33] utilizes a patch detector to locate and mask attack patches in adversarial images. However, the clever attacker will fool the patch detector. As a result, the attack patch cannot be accurately located by the patch detector. This location-based approach is also easily broken by this adaptive attacks. MAEDefense [22] utilizes a masked autoencoder to partially generate the image, weighted with the original image. Nevertheless, this cleaning mechanism is not thorough and not trained at the semantic level. As a result, the effectiveness of MAEDefense is limited and will have a significant negative impact on the accuracy of downstream retrieval tasks.Enhance robustness based on adversarial training. Some defense methods [38, 39] attempt to conduct adversarial training (AT) on the retrieval model. This strengthens the model's resistance to adversarial samples. However, adversarial training is expensive and less practical than the preprocessing method for models that require large-scale pre-training, such as CLIP [26] . 
problem description 3.1 target retrieval models: 
 • Content-based image retrieval model. Images are the input to content-based image retrieval systems. The retrieval system aims to find similar images in the database. We use CSQ-ResNet50 [34] as our target model in the content-based image retrieval task. The core structure of CSQ-ResNet50 is a feature extractor 𝐻 (•). An image 𝒙 can be transformed into a K-bit hash code 𝒄 as follows:EQUATIONThe Hamming distance of the images' hashcodes is used to calculate the similarity score. The similarity score is then utilized for the similarity ranking [5] .• Text-based image retrieval model. Texts are the input to textbased image retrieval systems. The retrieval system searches for images that match the written description. We use CLIP [26] as the model in the text-based image retrieval task. CLIP is a crossmodal model pretrained on 400 million image-text pairs. Its network structure consists of a text encoder 𝐹 𝑡 (•) and an image encoder 𝐹 𝑖 (•).Text and image representations can be obtained by feeding them into the corresponding encoder. The semantic distance 𝒔 between a text 𝒚 and an image 𝒙 can be calculated as:EQUATIONwhere 𝒔 is the semantic distance, which can be used to rank similarity, and its function 𝑑 (•) selects the Euclidean distance.• Attack in content-based image retrieval systems. AdvHash [15] is the state-of-the-art white box attack method against contentbased image retrieval systems. AdvHash inserts a universal adversarial patch 𝝈 into the input image. As a result, retrieval results may include irrelevant images of the target class. AdvHash first computes the central hash code 𝒄 𝒕 of each image class. The adversarial patch is then optimized so that the hash codes of the adversarial images are close to the 𝒄 𝒕 .• Attack in text-based image retrieval systems. TTH [14] is the state-of-the-art white box attack method for text-based image retrieval systems. TTH inserts advertising images with universal adversarial patches 𝝈 into the image database. When a user enters some keywords, the retrieval result will include irrelevant adversarial images. TTH first computes the embedding of all sentences containing the target keyword to obtain the embedding 𝒆 𝒕 associated with the target keyword. An adversarial patch is then optimized so that the embedding of the images with the attack patch is close to the 𝒆 𝒕 .• Non-adaptive attack. In non-adaptive attacks [18] , the attacker only has knowledge of the retrieval model and no knowledge of defense methods. In our experimental setup, the attacker optimizes the attack patches by gradient backpropagation based on TTH [14] and AdvHash [15] methods, targeting the original model without any defense. Then, defense methods are applied to the optimized patches. Based on the retrieval results, test their defense capabilities.• Adaptive attack. Adaptive attack applies some targeted strategies [1, 2] to attack the whole model including the defense pipline.To effectively attack SAC [18] and PatchZero [33] , we make fooling the patch detector one of the attacker's optimization goals. If the patch detector can't locate the patch correctly, the SAC and PatchZero defenses will fail. Since the defense pipline of LGS [24] is differentiable, attackers can directly perform gradient backpropagation to optimize the attack patch. To effectively attack PAIR, we have experimented with various adaptive attacks, such as BPDA [1] and EOT [2] , as well as their combinations. Due to space limitations, we don't show the details, but we are convinced that EOT is the most effective adaptive attack strategy against PAIR. The following are the new optimized goals for TTH [14] and AdvHash [15] :EQUATIONarg minEQUATIONEQUATIONwhere 𝒙 ′ is the attack image modified by the attacker, 𝑇 is the pre-processing defense method, 𝐻 and 𝐹 𝑖 are content-based and text-based image retrieval models, ℎ 𝑎𝑑𝑣 is the hyperparameter of EOT, 𝒅 is the distance measure, and 𝝈 denotes the modification degree of the image by the attacker. 
our model pair: 
 Our goal is to develop an augmented pre-denoising method that is efficient and insensitive to variations in the shape, position, and form of adversarial patches for content-based and text-based image retrieval systems.In this research, we tackle the problem of defending against patch attacks. These attacks modify parts of an image to mislead retrieval models. To solve this problem, we propose PAIR (Pre-denosing Augmented Image Retrieval Model), a novel solution designed to improve the security of image retrieval systems. The fundamental concept behind PAIR is its ability to randomly and dynamically reconstruct an entire image, following the semantic of the image. This process is achieved through a generative network in the form of a masked auto-encoder. With our designed masking and recovery mechanism, it is guaranteed that all pixels of the attack patch are purified. Such a technique ensures the preservation and capture of the semantics of the processed image, adaptable to any downstream model, while disrupting the adversarial patches. This disruption makes the attackers' optimisation strategies less effective and harder to adapt to.However, the traditional masked autoencoder model is insufficient to fully recover the image, which inevitably leads to a significant compromise in the accuracy of the retrieval model. To improve the recovery of PAIR, we propose a novel training methodology based on a semantic discriminator. This approach, which focuses on training the generative network from a semantic representation standpoint, markedly increases the retrieval model's accuracy. Moreover, it unexpectedly can enhance the model's ability to capture the real semantics of an image, thereby enhancing its defensive capabilities, as demonstrated in our ablation studies.PAIR defends the retrieval model from patch attacks through dynamic and random reconstruction of the entire input image. The defence mechanism we design can be guaranteed to purify all pixels of the attack patch while preserving the semantics of the input image. The operational pipeline of PAIR unfolds through the following sequential steps:• Step 1: Initially, partition an image 𝒙 ∈ 𝑅 𝑐 ×ℎ×𝑤 into 𝑛 image patches {𝑥 𝑖 } 𝑛 𝑖=1, each of size 𝑠 × 𝑠 pixels, where 𝑥 𝑖 ∈ 𝑅 𝑐 ×𝑠 ×𝑠 . The dimensions 𝑐, ℎ, and 𝑤 represent the number of channels, height, and width of the image, respectively. Ensure that 𝑠 is a common divisor of both ℎ and 𝑤, and the total number of patches 𝑛 is ℎ𝑤/𝑠 2 . Subsequently, generate a binary mask sequence 𝑀 of length 𝑛:EQUATIONwhere 𝑚 𝑖 is defined by:EQUATIONwhere 𝛼 denotes the mask ratio, which ranges between 0 and 1. Experimental results indicate that setting it between 0.2 and 0.8 yields satisfactory defensive effects. However, we recommend setting it to 0.5 to achieve optimal retrieval accuracy on clean samples. Due to the space limitation, we won't show this sensitivity analysis.• Step 2: Employ the mask sequence 𝑀 to obtain a set of masked image patches {𝑝 𝑖 } 𝑛 𝑖=1 , where 𝑝 𝑖 is determined by:EQUATIONwhere 𝑧 representing an 𝑐 × 𝑠 × 𝑠 image patch of zero pixel values.A generator then reconstructs the image. As depicted in Figure 4 , the generator 𝐺 (•), structured on a masked autoencoder [12] , encompasses both an encoder and a decoder, each utilizing a multilayer transformer architecture. Initiate the image restoration by mapping the image patches to embedding via a fully connected layer FC(•), as with ViT [10] . Then, add 1-dimensional positional embedding 𝑒 𝑝𝑜𝑠 to the image patch embedding in raster order, following the method in [10] :EQUATIONHere, 𝑑 represents the dimension of each embedding and subsequent feature. Input the embedding set {𝑒 𝑖 } 𝑛 𝑖=1 into the encoder to infer the feature set {𝑓 𝑖 } 𝑛 𝑖=1 : 𝑓 1 , 𝑓 2 , . . . , 𝑓 𝑛 = Encoder(𝑒 1 , 𝑒 2 , . . . , 𝑒 𝑛 ).For masked image patches, introduce a trainable mask embedding 𝑒 𝑚𝑎𝑠𝑘 as a feature and combine it with 1-dimensional positional embedding. The resulting feature set {𝑓 𝑖 } 𝑛 𝑖=1 and the mask embedding 𝑒 𝑚𝑎𝑠𝑘 are then input into the decoder along with 1-dimensional positional embedding 𝑒 𝑝𝑜𝑠 : EQUATIONEQUATIONwhere ⟨•⟩ denotes the concatenation function, sequentially merging the image patches.• Step 3: Finally, perform a second restoration using the image patches reconstructed in Step 2. This ensures that all pixels in the final image are restored in alignment with the intended semantics, rather than derived from the original image. To do this, invert the mask sequence from Step 1, [𝑚 1 , 𝑚 2 , ..., 𝑚 𝑛 ], as follows:EQUATION)Then, replicate the image restoration process from Step 2 using the new mask sequence [𝑚 ′ 1 , 𝑚 ′ 2 , ..., 𝑚 ′ 𝑛 ] and the same generator, but this time input the image restored in Step 2.In the process of training a generative model, the integration of a semantic discriminator is vital. This discriminator, structured as a multi-layer transformer network, can extract semantic features from images. The generator is trained purely on clean images to guarantee that the model is not biased against attack patches. As shown in Figure 4 , unlike the defensive pipeline, the training of the generator is generated only once. The motivation is that, from our observation, iterative generation based on already recovered images will result in training instability.To enhance the semantic consistency between the reconstructed image 𝒙 ′ and its original 𝒙, we have formulated a composite training loss function 𝐿 as follows:EQUATIONwhere 𝐿 𝑝 denotes the pixel-level reconstruction loss and 𝐿 𝑠 denotes the semantic reconstruction loss, with the parameter 𝛽 modulating their respective contributions. The pixel-level loss 𝐿 𝑝 , which provides hard labels for training, is formally defined as:EQUATIONHere SSIM(•) [31] is a metric for measuring the similarity of two images. It takes into account the brightness, contrast and structural information of the images. It yields values in the range of -1 to 1, with higher values being more similar. The following is the formula for SSIM(•):EQUATIONwhere 𝜇 𝑥 𝑖 is the average pixel value, 𝜎 2 𝑥 𝑖 is the variance, 𝜎 𝑥 𝑖 𝑥 ′ 𝑖 is the covariance of 𝑥 𝑖 and 𝑥 ′ 𝑖 . 𝑐 1 and 𝑐 2 are constants, following [31] . The loss 𝐿 𝑠 at the semantic level provides soft labels and is formally defined as:EQUATION)In this equation, the semantic discriminator 𝐷 (•) plays a crucial role in extracting the semantic vector of the image, employing cosine similarity to maximize semantic alignment. 
experiments: 
 In this section, we introduce the benchmark datasets and analyze the results of defending against state-of-the-art adversarial patch attacks in the content-based image retrieval task and the text-based image retrieval task. In addition, we perform ablation studies to investigate the factors that affect defense and clean sample accuracy. Finally, we investigate the sensitivity of PAIR to the size, shape, form, and position of attack patches.Datasets. We demonstrate the effectiveness of our approach on three popular benchmark datasets. ImageNet [9] and MSCOCO [17] are used for the content-based image retrieval task. Flickr30K [25] and MSCOCO are used for the text-based image retrieval task. For ImageNet, following [3, 15] , we use a subset that has 130k images with 100 classes. We apply the COCO2014 dataset for MSCOCO. The training set contains 82,783 images and the validation set contains 40,504 images with a total of 80 categories, and each image has 5 sentences of annotation. Flickr30k contains 31,783 images, each with 5 sentences of annotation. We randomly selected 21,783 images for the training set, 5,000 images for the validation set, and 5,000 images for the test set. Training details. The image patches that divide the image are 16 × 16 in size. The generator and discriminator are initialized with the parameters of the pretrained model MAE [12] and CLIP [26] , respectively. Since CLIP is extensively pre-trained and has high generalisation capacity, the discriminator requires no further training and the generator is trained solely. A total of 104,566 images from MSCOCO and Flickr30k are used to train the generator according to the training approach. The mask radio 𝛼 in Step 1 is set to 0.5. The 𝛽 in equation 14 is set to 1. The 𝑐 1 , 𝑐 2 in equation 3 are set to 1e-4, and 9e-4, respectively. The batch size is set to 32, the optimizer is Adam, and the learning rate is set to 1e-5. The model is trained for 100 epochs. It takes about 20 hours to complete the training process on a Tesla V100.Baselines. To compare with PAIR, we implemented PatchZero [33] , SAC [18] , LGS [24] , MAEDefense [22] , Adversarial Training (AT) [23] . In particular, we changed the uniform mask in MAEDefense's mechanism to the random mask to enhance its defense.Attacks details. In the text-based image retrieval task, following [14] , 23 keywords were set for target attacks. Insert 10 advertising images into the image database for each keyword. The width and height of the adversarial patch was set to 0.35 of the width and height of the original image. In the content-based image retrieval task, following [15] , we tried some categories for target attacks and extracted 50 images to optimize a universal adversarial patch. Then, tested the adversarial patch on another 100 images. The length of the hashcode is set to 64. The width and height of the adversarial Table 3 : Ablation study. R@10 (%) under different defense components in text-based image retrieval systems. "↑" means higher is better. "↓" means lower is better. The best performance of each column is in bold. The attack method is TTH [14] . Following EBM [13] , ℎ 𝑎𝑑𝑣 is set to 7.Evaluation metrics. We use Mean Average Precision (mAP) and Recall@K as evaluation metrics in content-based and text-based image retrieval, respectively. Similar to AdvHash [15] and TTH [14] , we adopt the following three variants:• Clean: test retrieval ability on clean images. To measure the negative impact of these defense methods on the model retrieval capability. The higher this metric is, the better. • Irrelevant image: After the attack, irrelevant images that the attacker wants the user to see will appear in the retrieval results. For Recall@K, a successful attack is defined as an irrelevant image ranking in the top 𝐾. Therefore, for defensive methods, the lower this metric is, the better. • Relevant image: After the attack, the ranking of really relevant images in the results will drop. This metric use ground-truth images to compute Recall@K and mAP.We compared PAIR with several state-of-the-art defense methods from similar tasks. The detailed attack formulation is described in Section 3.2. The experimental results highlight the advantages of PAIR in the context of adaptive attacks. Text-based image retrieval. As shown in Table 1 , PAIR demonstrates superior defense capabilities on both datasets. In particular, against adaptive attacks, the success rate of the attacks is 14.05% lower than the second-best method. Against non-adaptive attacks, methods such as MAEDefense [22] , LGS [24] , and PAIR prove highly effective, reducing the attack success rate to below 1%. While SAC [18] and PatchZero [33] perform not well. If the patch detectors are not located correctly, their defence fails. PatchZero and SAC have more advantages on clean samples due to less image damage. However, PAIR also maintains a high level performance for retrieve clean samples, paralleling the original model.Content-based image retrieval. The results in Table 2 highlight PAIR's strength in dealing with adaptive attacks, reducing the success rate of such attacks to below 2.05%. This level of effectiveness is also seen in the context of non-adaptive attacks. However, in terms of retrieval performance on clean samples, PAIR shows a notable decrease compared to the undefended model. On the one hand, mAP is a more comprehensive evaluation of the ranking results, reflecting the negative impact of the defense. On the other hand, this decrease is a trade-off for its much stronger defense, which results in PAIR still being the best performing method overall.After attackFigure 5 : High dimensional vector visualization. We extract both image and text as high-dimensional vectors and visualize them. Experimental results show that after PAIR defense, the semantic distance between the vectors is restored to the original state. Purple: texts containing the attacked keyword "policeman"; Gray: natural images in the test set; Gold: attack images inserted into the database. The goal of the attacker is to make the gold points close to the purple points.In this section, we evaluate the effectiveness of the various components of PAIR. As shown in Table 3 , when the image is reconstructed without masking, it has a certain defensive effect under a non-adaptive attack. In this experimental setting, the generator's encoder is employed to extract the image features. Then, the decoder is used to restore the image features to the reconstructed image. When reconstructing an image with a fixed mask, it can provide an effective defense against a non-adaptive attack. However, it provides no significant barrier against the adaptive attack. PAIR can achieve excellent defense when reconstructing the input image using random masking under both adaptive and nonadaptive attacks. We believe that random masks will result in random damage to attack patches and that the existing adaptive attack methods are difficult to adapt. Therefore, it has an efficient defense effect under adaptive defense. The ability of the generator to preserve image semantics improves after the addition of a semantic discriminator to its training. Therefore, the retrieval accuracy on clean images is comparable to the original model's effect. The reasons for the stronger defensive effect are as follows: because the ground-truth image is better reconstructed, its retrieval ranking will be higher. As a result, the metric of the relevant image improved.With the enhanced ability of the model to capture real semantics, the ability to purify against adversarial perturbations is increased. Therefore, it benefits the metric of the irrelevant images.We evaluated the robustness of PAIR against adversarial attacks of different patch sizes, shapes, and forms. The results in Table 4 demonstrate that our proposed PAIR approach maintains high effectiveness against attacks of varying sizes. As the size of the adversarial patches increases, the attack success rate generally increases when no defense is applied. However, PAIR consistently demonstrates high efficiency in defense, with attack success rates of less than 5%. Then, we conducted experiments with different patch shapes, including circles and rectangles. Furthermore, we randomly altered the form and position of the attack patches. As shown in Table 4 , we reported the average experimental data. The experimental results show that PAIR remains effective. This is attributed to PAIR's reliance solely on clean samples for training, making it insensitive to diverse attack patches.Visualization. The visualization is designed to confirm that our defense lowered the aggression of the attack patch. The attacks on PAIR are set to be adaptive attacks to achieve better attack effects. As shown in Figure 5 , we visualized the semantic representation of the original images and texts. After the PAIR defense, the semantic distance between the attacker's images and the texts containing the attacked keyword is restored to its original state.To illustrate the generality of PAIR, we experiment against global perturbation attacks. The global perturbation attack allows the attacker to modify all the pixels in the image and set the perturbation individually for each adversarial image. The attack is set up to insert an adversarial advertising image into the database. Modify the TTH [14] to the global perturbation attack on the MSCOCO [17] dataset. The size of perturbation budget is 4/255 in 𝑙 ∞ . Partial defense methods can't defend against global perturbation attacks, such as PatchZero [33] and SAC [18] . Therefore, we introduce other advanced baselines for comparison. Among them, JPEG [11] , ARN [36] , and CAFD [37] are based on pre-processing denoising and HM [39] is based on adversarial training. As shown in Table 5 , compared [36] 96.47 76.57 CAFD [37] 96.24 75.63 MAEDefense [22] 59.42 64.86 JPEG [11] 47.26 55.65 HM [39] 48  
conclusions: 
 In this paper, we have developed PAIR, a new method specifically designed to protect against patch attacks in both content-based and text-based image retrieval systems. This method processes images in a way that reduces the impact of attacks while preserving the semantic information for the downstream model. Extensive experimental evaluations on a variety of benchmark datasets has demonstrated PAIR's exceptional ability to defend against adversarial patch attacks. Impressively, PAIR is insensitive to the shape, form, and position of the attack patches. Furthermore, in terms of retrieval accuracy on clean samples, PAIR's performance is comparable to that of the original, undefended model. Ablation studies further reveal that the PAIR module introduces unpredictable disruptions to attack patches, making existing adaptive attack methods less effective and difficult to adapt to.Limitations and broader impacts. Although our PAIR defense performs well in semantic-based retrieval models, it is not suitable for fine-grained retrieval tasks, such as face retrieval. In terms of compatibility, as a preprocessing defense, our method destroys the attack patches while preserving the image's semantic information, making it compatible with the most of downstream retrieval models. Moreover, our defense strategy may be extended to other domains, such as defense in graphs and texts.  
: 
 in a contiguous region of the image. Some recent studies [14, 15] have shown that the attacker can insert a patch into the image so that the users will see the search results that the attacker wants them to see. As shown in Figure 2 , the attacker inputs adversarial images into a content-based image retrieval system. The search results are all irrelevant images. Such vulnerabilities could be exploited in intellectual property protection systems, allowing plagiarised content to escape detection. Moreover, the threat extends to textbased image retrieval databases, where attackers could introduce advertising images with attack patches. As a result, a user searching for a seemingly innocuous word such as "shoes" could be bombarded with these unwanted advertisements.It is still unknown how to combat against patch attacks on content-based and text-based image retrieval systems. This problem has three main challenges: (1) Adaptation to diverse adversarial patches. Defense strategies need to be robust to the shape, form, and position of the adversarial patch. Current preprocessing defense methods [18, 37] often rely on training with specific types of adversarial images, which limits their effectiveness when encountering previously unseen patch configurations. (2) Resistance to attack without relying on localization methods. LGS [24] locates attack pixels through empirical observation, while SAC [18] and Patchzero [33] use a patch detector for localization. However, according to Chiang et al. [8] and our experiments, these localization-based defenses can be easily fooled by adaptive attacks. (3) Affordability of training and implementation. Some studies employ expensive adversarial training to defend against global perturbation attacks [38, 39] . However, in many cases, the cost of adversarial training is prohibitively expensive, particularly for models pretrained on large-scale datasets such as CLIP [26] .In this study, we propose a novel pre-denoising and purifying defense strategy named the Pre-denoising Augmented Image Retrieval (PAIR) model. As shown in Figure 1 , PAIR achieves a superior trade-off between defense performance and retrieval ability on For the content-based image retrieval system, the adversarial patched image serves as the input. For the text-based image retrieval system, some advertising images with adversarial patches are added to the database. The attack patch format is set as a QR code. We develop a pre-denoising augmented strategy to protect the retrieval system from patch attacks. Above: the retrieval result without defense; Below: the retrieval result with our defense. clean samples. PAIR employs a unique strategy by masking random portions of image patches and reconstructing it repeatedly using a robust generative network. This approach not only guarantees that all attack patch pixels are purified, but also ensures that important semantic content is preserved. Note that as a preprocessing denoising module, PAIR is theoretically compatible with any downstream image retrieval model. Remarkably, PAIR relies solely on clean images and doesn't involve any localization techniques. This attribute makes the model relatively unaffected by variations in the shape, size, and position of attack patches. Moreover, PAIR does not require expensive adversarial training. Experiments on several datasets against patch attacks demonstrate that PAIR is highly effective in both adaptive attack and non-adaptive attack settings for content-based and text-based image retrieval.Further, we present a new training method for PAIR's generative network. As a pre-denoising method, PAIR will inevitably affect the accuracy of the downstream retrieval model. This training method is designed to enhance the generative network's ability in image reconstruction, ensuring the preservation of semantic information from the original image through the implementation of a semantic discriminator. Our ablation experiments show that it can reduce the negative effects caused by PAIR. This means that the defended model has a similar performance to the original model in the case of clean samples, but with a higher defensive capability.In summary, our main contributions are as follows:• To the best of our knowledge, PAIR is the first defense model against patch attacks in content-based and text-based image retrieval. It is insensitive to variations in the shape, form, and position of adversarial patches. • We design a new training method that utilizes a semantic discriminator to improve the PAIR's performance. The defense model performs similarly to the original model on clean samples and has stronger defenses. • Experiments show that PAIR is highly effective in defending against two state-of-the-art attack methods. It reduces the success rate of attacks to less than 5%. Furthermore, PAIR also shows competitive results in defending against other forms of attack, such as global perturbation attacks. 
 
 Appendix:  


Decision: This paper introduces the PAIR model, which defends image retrieval systems against patch attacks by dynamically reconstructing images based on their content. This approach thwarts patch attacks while maintaining semantic integrity. To enhance semantic understanding and reconstruction accuracy, PAIR incorporates a semantic discriminator in training. Experimental results demonstrate the effectiveness of PAIR.

During the review process, this paper received split final reviews (2 borderline reject and 2 borderline accept). After carefully reviewing the paper and considering the reviewers' comments, as well as discussing it with the SAC, I recommend accepting this paper. The reason for my recommendation is that it presents a simple yet effective method that provides valuable insights into defending against adversarial patches.

One main concern raised by reviewers who gave negative scores is the high computing cost associated with PAIR. This concern is valid, but I believe that future academic and engineering development can address these issues of computing cost and efficiency effectively. I encourage the author to incorporate the discussions and comments from the reviewers into the paper. Additionally, it would be beneficial to improve the proposed method to reduce its computing cost in further study.

Reviews: review: paper_summary: paper_summary: The paper introduces the Pre-denoising Augmented Image Retrieval (PAIR) model, aimed at defending against adversarial patch attacks in image retrieval systems. The approach involves dynamically reconstructing images based on their semantic content to purify adversarial patches while maintaining the semantic integrity of the images. The authors also propose a novel training strategy incorporating a semantic discriminator to enhance the model's performance. Experimental results demonstrate that PAIR significantly reduces the success rate of state-of-the-art patch attack methods and shows competitive results against other forms of attacks.
strength_weakness: strength_weakness: (1)	The PAIR model introduces a novel method for defending against patch attacks by reconstructing images based on their semantic content.
(2)	The experimental results are robust, showing that PAIR reduces the success rate of patch attacks to below 5%, which is a notable improvement over existing methods.
(3)	PAIR's preprocessing denoising module is theoretically compatible with any downstream image retrieval model, making it a versatile solution that can be applied broadly across different systems.
(4)	The authors provide source code to prompt reproducibility.
limitations: limitations: (1)	The implementation of the PAIR model involves a sophisticated generative network and a semantic discriminator. This complexity might pose challenges for practical deployment in real-world systems.
(2)	While the paper presents strong results against specific patch attack methods, it would be beneficial to see a broader evaluation against a wider range of adversarial patch attack methods to fully establish the model's robustness.
(3)	As shown in Table 1, the proposed method performs slightly worse than LGS under non-adaptive attack.
(4)	The performance of the proposed method on clean images is comparatively lower than the baselines and the undefended model.
paper_summary: paper_summary: The paper proposed the Pre-denosing Augmented Image Retrieval (PAIR) model for defending image retrieval systems against adversarial patch attacks. The method is based on the idea of masked autoencoders that randomly masks out and reconstructs the entire images in order to remove the adversarial patches. To improve the reconstruction performance, the paper proposed to include the SSIM loss as an additional regularization term for training the generator. Experiment results show that the proposed method is highly effective in defending two state-of-the-art attack methods.
strength_weakness: strength_weakness: 1. The paper is in general well written and easy to follow. 
2. The proposed method is general and can be applied to any image retrieval system. 
3. The paper conducted extensive evaluations on both content-based and text-based image retrieval systems with both adaptive and nonadaptive attacks. The proposed method demonstrates superior performance compared to the baselines.
limitations: limitations: 1. The novelty of the proposed method seems incremental compared to MAEDefense [22]. It seems the main difference is the added SSIM regularization term for training the generator.
2. The random masking scheme might cause information loss that cannot be recovered by the generator. 
3. The performance of PatchZero and SAC seems to be too low. Since the attack patch format is set as a QR code, it should be very easy to be detected by the patch detectors. Have the authors retrained the patch detectors on the dataset?
4. For the text-based image retrieval scenarios, it seems the defense needs to reconstruct every image in the database, which can be computationally expensive.
5. How is the reconstruction quality? Can the authors provide some visualization of before and after defense images?
paper_summary: paper_summary: The paper introduces the PAIR model to defend image retrieval systems against patch attacks. PAIR dynamically reconstructs images based on their content, thwarting patch attacks while maintaining semantic integrity. It incorporates a semantic discriminator in training to enhance semantic understanding and reconstruction accuracy. Experimental results demonstrate PAIR's effectiveness.
strength_weakness: strength_weakness: - The Mask-reconstruct approach is a certain level of innovation.
- The method is straightforward and easy to implement, with thorough experimentation.
- The code has been released.
limitations: limitations: -	In line 300, it is mentioned that "Previous work such as SAC and PatchZero used a patch detector to locate and mask attack patches. This is highly dependent on the performance of the patch detector." This statement similarly applies to this work, as the PAIR method is also highly dependent on the performance of the generator. If the generator is not sufficiently good, the reconstructed image quality will suffer, potentially resulting in significant structural differences from the original image.

-	There is a substantial risk associated with masking the original image and then reconstructing it. If the image contains text or human faces, for example, it would be challenging to recover details after masking, which could significantly impact the image database negatively.

-	Experiments lacking efficiency comparison. If the reconstruction process consumes a considerable amount of resources, it may not be an ideal defense method.

Please update the TTH citation to reference [1].

[1] Hu, Fan, Aozhu Chen, and Xirong Li. "Towards Making a Trojan-Horse Attack on Text-to-Image Retrieval." ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
paper_summary: paper_summary: This paper proposes the Pre-denosing Augmented Image Retrieval (PAIR) model, a new approach designed to protect image retrieval systems against adversarial patch attacks. Experiments show that PAIR significantly outperforms existing defense methods. It effectively reduces the success rate of two state-of-the-art patch attack methods to below 5%, achieving a 14% improvement over current leading methods
strength_weakness: strength_weakness: (1) PAIR is the first defense model against patch attacks in content-based and text-based image retrieval. It is insensitive to variations in the shape, form, and position of adversarial patches
(2) PAIR is highly effective in defending against two state-of-the-art attack methods. It reduces the success rate of attacks to less than 5%. Furthermore, PAIR also shows competitive results in defending against other forms of attack, such as global perturbation attacks.
(3) The writing of this paper is well and can be easy to follow.
limitations: limitations: (1) The novelty of this paper is limited. There are too many studies using the generative model to defend the adversarial attacks and backdoor attacks of neural networks. The proposed method is straightforward.
(2) Why only consider the adversarial patch? What is the performance of the proposed method against original adversarial attacks of the Image Retrieval Model?
(3) The proposed method needs to train a generative model. Whether using pre-trained generative models such as stable diffusion can also effectively against adversarial attacks