SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\alpha$  is a hyper-parameter to balance the two losses. Given a perturbed image  $\mathbf{x}_{\mathrm{ptb}}$ , the solution to the optimization problem in Eq. (4.1) is the purified image  $\mathbf{x}_{\mathrm{pur}}$ . We use projected gradient descent to solve the optimization problem efficiently. In practice, we initialize  $\mathbf{x}_{\mathrm{pur}}$  by  $\mathbf{x}_{\mathrm{ptb}}$  plus an extra small Gaussian perturbation similar as in Madry et al. (2018). A summary of our proposed method is shown in Algorithm 1 in the Appendix.
In this section, we conduct comprehensive experiments to demonstrate the effectiveness of the proposed method and verify its capability to disable imperceptible perturbations introduced by existing protection methods. Specifically, we evaluate against two protection methods on the specific tasks for which these methods were proposed: style mimicking and malicious editing of images.
Style Mimicking We conduct all the style mimicking experiments on the WikiArt dataset (Saleh and Elgammal, 2015). For GLAZE<sup>1</sup> (Shan et al., 2023), we essentially followed the settings of the
<sup>1</sup>Experiments reproduced based on the 3rd version of the GLAZE paper on Arxiv (11 Apr 2023).
original paper and reproduced the code: given a victim artist  $V$ , we randomly select 124 of their works. We use ViT+GPT2 model to generate a title for each artwork and attach the victim artist  $V$ 's name to the generated title as our prompt  $y$ . Next, we randomly draw 24 works from these 124 art pieces as the style imitation training set, and the remaining 100 as the test set. We fine-tune a pre-trained LDM on the training set, and then generate new artworks mimicking the style of the victim artist by the prompt  $y$ .
Malicious Editing of Images For malicious editing of images, we utilize the code and settings made public by Photoguard (Salman et al., 2023). The purpose of malicious editing is to modify the background of a photo following the given malicious prompt. We conduct our experiments on the Helen (Le et al., 2012) dataset for this task. We selected the 80 images with the smallest face-to-image ratio from the Helen dataset and their corresponding face masks. Subsequently, we use Photoguard to generate protective noise for these images. Then, we attempt to modify the images using a latent diffusion model. We generate noise using the most powerful Diffusion attack-L2 method in Photoguard.
Baselines To show the effectiveness of IMPRESS is obtained through our consistency-based loss design, we also consider three other simple post-processing baselines: JPEG compression of the image, adding Gaussian noise to the image, and resizing the image. For JPEG compression, we set the quality factor as  $15\%$ . For adding Gaussian noise, we set the mean as 0 and the variance as 0.15. For the image resizing method, we first resize the  $512 \times 512$  image to  $256 \times 256$ , then resize it back to  $512 \times 512$ .
5.2 Experimental Results on Style Mimicking
Table 1: Comparison of style mimicking performances when using fine-tuning data from clean images, GLAZE-protected images and post-processed protected images by our method and baselines. A higher accuracy suggests that the generated image style is close to the artist to be mimicked.
Metric	Clean	Protected	Post-processed
JPG	Noise	Resize	Ours
CLIP classifier Acc	90.8 ± 3.2	42.5 ± 8.3	64.9 ± 7.6	47.9 ± 7.0	66.6 ± 3.4	87.0 ± 6.7
Diffusion classifier Acc	95.4 ± 3.1	42.9 ± 9.5	67.6 ± 9.0	40.9 ± 9.8	68.0 ± 9.5	82.3 ± 8.3
In Table 1, we present the experimental results for the task of Style Mimicking. To conduct the experiment, we fine-tune the pre-trained LDMs using clean data, GLAZE (Shan et al., 2023) protected data, and the post-processed data by our method as well as other baselines. We then use those fine-tuned models to generate new images mimicking the style of a certain artist and evaluate whether the generated image can successfully mimic the style of the artist. We consider two metrics for evaluating the generated image styles by constructing two image style classifier:
1. the CLIP classifier accuracy. Following the GLAZE paper (Shan et al., 2023), we use the pre-trained CLIP model (Radford et al., 2021) to build a style classifier to measure whether the generated images belong to the genre of victim artist  $V$ . We treat the genre classification provided in the WikiArt dataset as the ground truth label, and use the intersection of the 27 historical genres and 13 digital genres from WikiArt, as candidate labels for the CLIP model.
2. the Diffusion classifier accuracy. We consider another classifier that builds directly upon diffusion models. The Diffusion Classifier (Li et al., 2023) is a Zero-Shot image classification method, which uses conditional density estimates of images in pre-trained Diffusion models to construct Zero-Shot image classifiers. In our experiments, we used the same set of candidate labels (39 art genres) to build a diffusion classifier for style classification.
For each generated image, we use both the CLIP classifier and Diffusion classifier to evaluate whether the style mimicking succeeds (whether the victim artist  $V$ 's style falls into the top-3 classification results). Typically, all protection methods aim to lower the classifier accuracy as much as possible to prevent style mimicking from happening if the data usage is unauthorized. As can be seen from Table 1, the images generated using clean data samples achieve an average accuracy of  $90.8\%$  on the CLIP classifier and  $95.4\%$  on the Diffusion classifier. While the generated images using the protected data
Figure 4: The experimental results of IMPRESS. Left: 4 groups of samples in Style Mimicking, each group of samples is generated by Stable diffusion when using clean images, GLAZE (Shan et al., 2023) protected images, and our IMPRESS purified images for model fine-tuning. Right: 4 groups of samples in Malicious Editing of Images, each group of samples is obtained by editing the clean image, PhotoGuard (Salman et al., 2023) protected image, and our IMPRESS purified image.
samples certainly lower the accuracy to  $\sim 42\%$ , after our purified process, the generated images can again achieve an average accuracy of  $87.0\%$  and  $82.3\%$  on the CLIP classifier and Diffusion classifier respectively, close to the case of clean data samples. By contrast, all the other post-processing baselines only achieved limited improved accuracy, clearly outperformed by our IMPRESS method. We also present some sample results in Figure 4. From Figure 4 (left), we can clearly observe that although the GLAZE method can successfully mislead the LDM when directly used, after using IMPRESS to purify the protected image, we can still exploit the remaining information.
5.3 Experimental Results on Malicious Editing
Table 2: Comparison of malicious editing performances with input data from clean images, Photoguard-protected images and post-processed protected images by our method and baselines. The higher score indicates better alignment with the clean images.
Metric	Protected	Post-processed
JPG	Noise	Resize	Ours
SSIM	0.41 ± 0.06	0.52 ± 0.07	0.18 ± 0.05	0.49 ± 0.08	0.49 ± 0.08
PSNR	14.24 ± 1.57	15.51 ± 1.69	11.06 ± 0.94	14.94 ± 1.59	15.61 ± 2.03
VIF-p	0.11 ± 0.03	0.15 ± 0.04	0.06 ± 0.02	0.14 ± 0.05	0.16 ± 0.04
For the Malicious Editing of Images task, we use a pre-trained LDM to edit images following the guidance of certain prompts. Again we consider input images from clean images, images protected by Photoguard (Salman et al., 2023), and the post-processed images by our method as well as other baselines. We followed the same settings as in the Photoguard paper (Salman et al., 2023), using SSIM (Wang et al., 2004), PSNR, and VIF-p (Sheikh and Bovik, 2006) as image similarity algorithms to measure the fidelity of the generated images. We calculate similarity scores between the edited
Style Mimicking, Artist: Claude Monet
JPG Compressed
Resize
IMPRESS
Image Editing, Prompt: A person in an airplane
JPG Compressed
Resize
Figure 5: Comparison of different post-processing methods on style mimicking and image editing tasks.
clean images and the edited protected images, as well as the post-processed images. Higher scores indicate results that are closer to the edited clean images.
According to Table 2, our proposed IMPRESS method achieved the best performance on the PSNR and VIF-p metrics, scoring 15.61 and 0.16, respectively, suggesting the edited images are still close to true images. We also obtained a score of 0.49 on the SSIM metric, which is the second-highest score. Note that in malicious editing, even simple image transformation methods can achieve relatively good results on those comparison-based metrics, which has also been pointed out in Salman et al. (2023) and Sandoval-Segura et al. (2023). This phenomenon confirms our point of view: existing image protection technologies are fragile. In Figure 4 (right), we show some real examples that while directing applying PhotoGuard can certainly cause the edited images less realistic, after being purified by our IMPRESS, the malicious editing would still go smoothly with no obvious artifact affecting the image fidelity.
In Figure 5, we also show some comparison with other post-processing baselines on both style mimicking and image editing. We can observe that IMPRESS largely outperforms other baselines for style mimicking while on image editing, all baselines can achieve decent performances in removing the perturbations. This is also consistent with our previous quantitative results, suggesting the imperceptible perturbations on the image editing task is significantly more fragile.
5.4 Adaptive Image Protection with Consistency-based Losses
In this section, we explore whether it is possible to leverage our consistency-based losses for building even better protection methods that could survive our IMPRESS method and successfully prevent unauthorized data usage in diffusion models. Specifically, since we know that keeping  $\mathbf{x}$  as consistent as possible with  $\mathcal{D}(\mathcal{E}(\mathbf{x}))$  is important, we add this regularization term to the optimization function of the existing protection method design. Take GLAZE as an example and we have the following optimization problem:
$$
\min _ {\delta} \| \mathcal {E} (\Omega (\mathbf {x}, T)), \mathcal {E} (\mathbf {x} + \boldsymbol {\delta}) \| _ {2} ^ {2} + \lambda \cdot \max  (\mathrm {L P I P S} (\mathbf {x}, \mathbf {x} + \boldsymbol {\delta}) - \Delta_ {L}, 0) + \beta \cdot | | (\mathbf {x} + \boldsymbol {\delta}) - \mathcal {D} (\mathcal {E} (\mathbf {x} + \boldsymbol {\delta})) | | _ {2} ^ {2},
$$
The coefficient of the added consistency regularization term is denoted as  $\beta$ . A meticulous tuning of the weight of the adaptive loss term has been carried out, as demonstrated in Table 3. We have modulated the weight of the adaptive loss term extensively, allowing it to comprise approximately  $10\%$  to  $95\%$  of the total loss. Our observations reveal that a minuscule proportion of the adaptive loss term in the entire loss leads the experimental results to mirror those of conventional image protection techniques, whereas an excessive proportion adversely impacts the performance of image protection.
The most pronounced effect of the adaptive protection technology is observable when  $\beta = 40$ , and we show the experimental results in Table 4 and 5 based on that. However, even under this condition, IMPRESS manages to attain around  $80\%$  accuracy by successfully mitigating the adaptive protective noise. Our experiments indicate that the adaptive design does not genuinely offer robust protection, since IMPRESS can still adeptly eliminate the protective noise, allowing the successful execution of style mimicking tasks, it is conjectured that attaining an optimal equilibrium among the three objective terms may pose a significant challenge.
To enhance the balance of competing objectives, we have incorporated the Orthogonal Projected Gradient Descent (O-PGD) approach by Gowal et al. (2020) into our methodology, as delineated
Table 3: CLIP classifier Acc for different weight of the adaptive loss term on style mimicking tasks
Training data	β = 1	β = 10	β = 40	β = 100	β = 1000
Protected	42.5	41.0	40.8	47.1	77.1
IMPRESS	85.6	84.6	81.9	85.7	82.1
Table 4: Experimental Results of Adaptive Protection on Style Mimicking Tasks
Metric	Clean	Protect Without Adapt	With Adapt
Protected	Purified	Protected	Purified
CLIP classifier Acc	90.8 ± 3.2	42.5 ± 8.3	87.0 ± 6.7	40.8 ± 9.1	81.9 ± 5.5
Diffusion classifier Acc	95.4 ± 3.1	42.9 ± 9.5	82.3 ± 8.3	44.1 ± 9.3	84.0 ± 6.6
Table 5: Experimental Results of Adaptive Protection on Malicious Editing Tasks
Metric	Protect Without Adapt	With Adapt
Protected	Purified	Protected	Purified
SSIM	0.41 ± 0.06	0.49 ± 0.08	0.42 ± 0.04	0.52 ± 0.05
PSNR	14.24 ± 1.57	15.61 ± 2.03	14.54 ± 1.74	15.90 ± 1.23
VIF-p	0.11 ± 0.03	0.16 ± 0.04	0.14 ± 0.04	0.16 ± 0.03
Table 6: Experimental results of adaptive defense after using O-pgd. In parentheses are the results of the adaptive defense experiments reported in the paper (without the use of O-pgd)
Metric	Clean	Protect Without Adapt	With O-pgd Adapt
Protected	Purified	Protected	Purified
CLIP classifier Acc	90.8	42.5	87.0	48.0 (40.8)	83.9 (81.9)
Diffusion classifier Acc	95.4	42.9	82.3	47.8 (44.1)	84.(84.0)
in Table 6 in the pdf. However, it's discernible that the utilization of O-PGD doesn't substantially bolster the efficacy of the adaptive method.
Based on our experimental observations, we hypothesize that the subpar performance of the adaptive method stems primarily from two intertwined complexities: 1) The introduced supplementary loss term complicates the optimization of the loss function. For GLAZE method, the loss function of the adaptive method has three components. Optimizing three different loss components is typically challenging, and striking a balance among these components remains an intricate task. 2) The loss term employed for the adaptive method might clash with the inherent loss term of image protection techniques. The purpose of image protection technologies is to induce semantical differences between generated images and original ones. Our image purification approach aims to make the image maintains its semantic properties (consistency after reconstruction), potentially leading to underlying optimization conflicts. These results suggest that designing adaptive image protection techniques specifically for IMPRESS might inherently be challenging.
In this work, proposed IMPRESS, a unified platform to evaluate the effectiveness of imperceptible perturbations as a protective measure. We demonstrated that it is difficult for the current imperceptible perturbations-based protection methods to prevent certain images from being correctly learned or processed by diffusion models as those imperceptible perturbations could be potentially removed with consistency-based losses. The proposed IMPRESS platform offers a comprehensive evaluation on several contemporary protection methods and can be used as an evaluation platform for future protection methods.
Omri Avrahami, Ohad Fried, and Dani Lischinski. 2022. Blended latent diffusion. arXiv preprint arXiv:2206.02779 (2022).
Andrew Brock, Jeff Donahue, and Karen Simonyan. 2019. Large Scale GAN Training for High Fidelity Natural Image Synthesis. In International Conference on Learning Representations (ICLR).
Aurelie Bugeau and Marcelo Bertalmio. 2009. Combining Texture Synthesis and Diffusion for Image Inpainting.. In VISAPP 2009-Proceedings of the Fourth International Conference on Computer Vision Theory and Applications. 26-33.
Valeriia Cherepanova, Micah Goldblum, Harrison Foley, Shiyuan Duan, John Dickerson, Gavin Taylor, and Tom Goldstein. 2021. Lowkey: Leveraging adversarial attacks to protect social media users from facial recognition. arXiv preprint arXiv:2101.07922 (2021).
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. 2023. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).
Prafulla Dhariwal and Alexander Nichol. 2021. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems 34 (2021), 8780-8794.
Pranav Dixit. 2023. Meet The Three Artists Behind A Landmark Lawsuit Against AI Art Generators. *BuzzFeedNews*.
Benj Edwards. 2022. Artists stage mass protest against AI-generated artwork on ArtStation. Ars Technica.
Shaopeng Fu, Fengxiang He, Yang Liu, Li Shen, and Dacheng Tao. 2021. Robust unlearnable examples: Protecting data privacy against adversarial learning. In International Conference on Learning Representations.
Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. 2023. Erasing Concepts from Diffusion Models. arXiv preprint arXiv:2303.07345 (2023).
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In neural information processing systems (NeurIPS).
Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy Mann, and Pushmeet Kohli. 2020. Uncovering the limits of adversarial training against norm-bounded adversarial examples. arXiv preprint arXiv:2010.03593 (2020).
Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 33 (2020), 6840-6851.
Jonathan Ho, Chitwan Sahara, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. 2022. Cascaded Diffusion Models for High Fidelity Image Generation. J. Mach. Learn. Res. 23, 47 (2022), 1-33.
Hanxun Huang, Xingjun Ma, Sarah Monazam Erfani, James Bailey, and Yisen Wang. 2021. Unlearnable Examples: Making Personal Data Unexploitable. In ICLR.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-image translation with conditional adversarial networks. In conference on computer vision and pattern recognition (CVPR).
Joseph Saveri Law Firm LLP. 2023. Class Action Filed Against Stability AI, Midjourney, and DeviantArt for DMCA Violations, Right of Publicity Violations, Unlawful Competition, Breach of TOS. https://cybernews.com/news/artists-unite-in-legal-battle-against-ai/.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600-612.
Julia Wolleb, Robin Sandkuhler, Florentin Bieder, and Philippe C Cattin. 2022. The swiss army knife for image-to-image translation: Multi-task diffusion models. arXiv preprint arXiv:2204.02641 (2022).
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. 2017. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE international conference on computer vision. 5907-5915.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Computer Vision and Pattern Recognition (CVPR).
Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. 2022. Egsde: Unpaired image-to-image translation via energy-guided stochastic differential equations. arXiv preprint arXiv:2207.06635 (2022).
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In international conference on computer vision(ICCV).
A Algorithm of IMPRESS
The following outlines the algorithmic process of our proposed IMPRESS method for the removal of protective noise.
Algorithm 1 IMPRESS
Input: Image encoder  $\mathcal{E}$ , image decoder  $\mathcal{D}$ ; hyperparameters  $\alpha, \eta, \Delta_L$ .
1: Initialize  $\mathbf{x}_{\mathrm{pur}} = \mathbf{x}_{\mathrm{ptb}} + \mathcal{N}(\mu, \sigma^2\mathbf{I})$
2: for iter = 1, 2, ..., N do
3:  $\mathcal{L}_{\mathrm{sim}} = ||\mathbf{x}_{\mathrm{pur}} - \mathcal{D}(\mathcal{E}(\mathbf{x}_{\mathrm{pur}}))||_2^2$
4:  $\mathcal{L}_{\mathrm{lpips}} = \max (\mathrm{LPIPS}(\mathbf{x}_{\mathrm{pur}},\mathbf{x}_{\mathrm{ptb}}) - \Delta_L,0)$
5:  $\mathcal{L} = \mathcal{L}_{\mathrm{sim}} + \alpha \mathcal{L}_{\mathrm{lipips}}$
6:  $\mathbf{x}_{\mathrm{pur}}\gets \mathbf{x}_{\mathrm{pur}} - \eta \nabla_{\mathbf{x}}\mathcal{L}$
7:  $\mathbf{x}_{\mathrm{pur}} = \mathrm{Clip}(\mathbf{x}_{\mathrm{pur}},\min = -1,\max = 1)$
8: end for
9: Return  $\mathbf{x}_{\mathrm{pur}}$
B Detail of Experiment
In this section, we have provided a detailed introduction to our experimental setup. The random seed for all experiments is set to 0, and our experiment code is publicly available in https: //github.com/AAAAAAsuka/Impress.
Style Mimicking Since we do not have access to unreleased artworks, we conduct all the style mimicking experiments on the WikiArt dataset (Saleh and Elgammal, 2015). The WikiArt dataset contains paintings from 195 famous artists throughout history. The dataset consists of 42,129 training images and 10,628 testing images. WikiArt categorizes all artists into 27 different genres (e.g., Impressionism, Cubism). we remove artists from the WikiArt dataset that the CLIP model cannot accurately classify (classification accuracy less than  $80\%$ ). Then, we select nine artists with the most artworks among the remaining artists, eight of which are used for testing the model's performance, and the remaining artist serves as the validation set for adjusting hyperparameters.
For GLAZE (Shan et al., 2023), we basically follow the original paper's setting: given a victim artist  $V$ , we randomly select 124 of their works. Then, we use the ViT+GPT2 model to generate a title for each work and attach the victim artist  $V$  's name to the generated title as the prompt  $y$ . Next, we randomly draw 24 works from these 124 art pieces as the style imitation training set, and the remaining 100 as the test set. Afterward, we use the stable-diffusion-2-1-base model released on Hugging Face as the initial pre-trained model weights and fine-tune it on the training set. After obtaining the fine-tuned LDM, we generate imitation works in the style of the victim artist using the prompts from the training set. Finally, we selected 8 artists and randomly extracted 24 pieces for fine-tuning the diffusion model and 100 pieces for validation, totaling 800 evaluation samples.
For fine-tuning the diffusion model, we use the fine-tuning script released with the model, training for 500 steps on the training set with a learning rate of  $5 \times 10^{-6}$ . For GLAZE, we choose a perturbation budget of  $p = 0.05$ , regularization coefficient  $\alpha = 30$ , and train for 500 steps using the Adam optimizer with a learning rate of  $10^{-2}$ . For our method, we set the perturbation budget to  $p = 0.1$ , regularization coefficient  $\alpha = 0.1$ , and train for 3,000 steps using the Adam optimizer with a learning rate of  $10^{-2}$ .
Malicious Editing of Images We use the Helen (Le et al., 2012) dataset for the experiments in the Malicious Editing of Images task. Helen is an annotated facial image dataset, consisting of 2,330 images sourced from Flickr. The annotations include facial landmarks and masks of facial features such as eyes, nose, etc. The dataset also includes masks for the entire face.
For malicious editing of images, We utilize the code and settings made public by Photoguard(Salman et al., 2023) for image protection. The purpose of the Malicious Editing of Images task is to modify the background of a photo. Hence, we selected the 80 images with the smallest face-to-image ratio from the Helen dataset and their corresponding face masks (the sample size of Photoguard experiment
is 60). Subsequently, we use Photoguard to generate protective noise for these images. Then, we attempt to modify the images using a diffusion model with the prompt "A person in an airplane".