Unveiling Structural Memorization: Structural Membership Inference Attack for Text-to-Image Diffusion Models

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 04:44:32

Abstract: With the rapid advancements of large-scale text-to-image diffusion models, various practical applications have emerged, bringing significant convenience to society. However, model developers may misuse the unauthorized data to train diffusion models. These data are at risk of being memorized by the models, thus potentially violating citizens' privacy rights. Therefore, in order to judge whether a specific image is utilized as a member of a model's training set, Membership Inference Attack (MIA) is proposed to serve as a tool for privacy protection. Current MIA methods predominantly utilize pixel-wise comparisons as distinguishing clues, considering the pixel-level memorization characteristic of diffusion models. However, it is practically impossible for text-to-image models to memorize all the pixel-level information in massive training sets. Therefore, we move to the more advanced structure-level memorization. Observations on the diffusion process show that the structures of members are better preserved compared to those of nonmembers, indicating that diffusion models possess the capability to remember the structures of member images from training sets. Drawing on these insights, we propose a simple yet effective MIA method tailored for text-to-image diffusion models. Extensive experimental results validate the efficacy of our approach. Compared to current pixel-level baselines, our approach not only achieves state-of-the-art performance but also demonstrates remarkable robustness against various distortions.

Main paper: 
 introduction: 
 In recent years, large models, especially diffusion models [14, 42, 44] have shown superior generative performance and found extensive application across various fields. Moreover, the advent of the text-to-image diffusion models [30] has facilitated the creation of high-quality, diverse text-conditional images. These models have significantly propelled the advancements of Artificial Intelligence Generated Content (AIGC). Nevertheless, the wide adoption of large models has raised various legal and ethical concerns, notably copyright issues [1] , consent [8, 10] and ethics [17, 27] . One of the pressing concerns is the unauthorized use of images for training models. This not only risks compromising the privacy of image owners but also poses copyright infringements, as models can realistically replicate copyrighted artworks based on training data. This is attributed to models' capacity for memorization, which means models can remember certain elements or even reproduce almost identical images from their training datasets. Under such circumstances, Membership Inference Attack (MIA) [16] serves as an approach to tackle the issue. Given a specific unauthorized image, the goal of MIA is to determine whether it is a member of the training set of a target model. The core of MIA is to ingeniously exploit the models' memorization of members to distinguish them from non-members.Recently, numerous Membership Inference Attack (MIA) methodologies [7, 19, 24] have been introduced for diffusion models. These methodologies, which rely on pixel-wise noise comparison, are designed to assess models' verbatim memorization of member images. However, we argue that it is practically impossible for largescale text-to-image models to memorize all the pixel information, given that their training sets usually contain billions of images. For instance, the Stable Diffusion-v1-1 is trained on the LAION2Ben dataset, which contains around 2.32 billion text-image pairs. Hence, we attempt to capture more advanced memorization capabilities of large text-to-image diffusion models, specifically at the structure-level. To investigate the structure-level memorization, we first examine how a specific image is corrupted during the unidirectional diffusion process for better comprehension of image structural variations, and then explore whether this correlates with models' memorization.As illustrated in Figure 1 , we iteratively employ noise to corrupt a specific image throughout the diffusion process. We then select various pairs of noisy images and compute the residuals between each pair. These residuals capture the change in image's corrupted parts. Our key observation is that diffusion models tend to corrupt the detailed features within the image in the initial diffusion stages, whereas the image structure is mostly preserved. Following this, the corruption extends to the overall structure of the image in the later diffusion stages. For instance, in Figure 1 , the model primarily focuses on the detailed patterns of the hat in the very early stages. As the diffusion progresses, it then begins to address the structural aspects of the cat's fur. As for textual prompts in the text-to-image models, they primarily influence the overall structure and context of images in the later stages, while having minimal impacts in the early phases. Based on these findings, we delve deeper into the differences in structural corruption between members and nonmembers. We reveal that the structures of members are better preserved than those of nonmembers in the initial diffusion stages, as the diffusion models have memorized the structures of the members during training phases.In light of the aforementioned observations, we introduce a straightforward yet effective MIA approach for text-to-image diffusion models by comparing the structural similarity between the original image and its corrupted version. Overall, the merits of our approach mainly contain three aspects: 1) Structural difference between members and nonmembers reveals the diffusion models' memorization at the skeletal level, which is preferred by large models. 2) Comparing differences at the image level is more robust to various distortions, particularly additional noise, than methods that rely on noise comparison. 3) Our method exhibits robustness to textual prompts, rendering it highly effective for membership inference tasks on images which lack training textual prompts in real-world scenarios.We conduct a series of comprehensive experiments on both the Latent Diffusion Model and the Stable Diffusion under varying image resolutions. These experiments demonstrate the superior performance of our proposed method. Furthermore, we evaluate the robustness of our method under a range of practical distortions. Our findings confirm the resilience of our method. In addition, we examine the effect of diverse textual inputs on the efficacy of our method, as we can not obtain the ground-truth texts of images in training. Our results confirm that our method's performance is robust to changes in textual inputs, providing valuable insights to the practical application of MIA.We summarize the contributions of this paper as follows:• Instead of pixel-level memorization, we delve into the advanced memorization capabilities of large diffusion models at the structure-level. Furthermore, we investigate the differences in the preservation of image structures between members and nonmembers during the diffusion process. • Drawing upon our findings, we propose a straightforward yet effective MIA method for text-to-image diffusion models by comparing the structural difference, which is more robust to various distortions.• We further verify that our method exhibits robustness to variations in textual prompts, enabling its application to images lacking training textual prompts in real-world scenarios. • Experimental results show that our method substantially outperforms existing MIA methods for text-to-image diffusion models, demonstrating its effectiveness. 
related work 2.1 membership inference attack: 
 As proposed by Shokri [37] , Membership Inference Attack (MIA) aims to infer whether a specific sample is a member of a target model's training set. MIA is categorized into two main tasks: whitebox attack and black-box attack. White-box attack [25, 29] presumes access to the internal structure and parameters of the target model, enabling a comprehensive analysis of the model's vulnerabilities. Conversely, black-box attack [32, 38, 41] operates solely through the model's observable inputs and outputs, posing a challenging yet more realistic scenario. Primarily, MIA is specifically targeted at classification models [5, 23, 34, 45] . Subsequently, with the rapid development of generative models, an increasing number of MIA methods have begun to explore the vulnerabilities of such models, including VAE [18] and GAN [9] . For instance, LOGAN [11] is the first to adopt MIA to GAN in both white-box and black-box settings. It utilizes the outputs from the discriminator for inference in white-box scenario, while training a shadow GAN model in black-box scenario. Hilprecht et al. [12] proposes the Monte Carlo score and reconstruction loss, which can be used for attacking VAE. GAN-Leaks [3] also uses the Monte Carlo score for attacking GAN in black-box scenario. MIA for diffusion models. Recently, several MIA methods targeting diffusion models have emerged. The Naive Loss method [24] and PIA [19] both use the training loss of diffusion models as a metric for membership inference, specifically by comparing the added noise with the predicted noise. The key difference is that Naive Loss method employs random Gaussian noise, whereas PIA utilizes the diffusion model's output at time t=0 as the noise. SecMI [7] compares the distance between two adjacent noisy images, which are generated through the diffusion process and the denoising process respectively. Nevertheless, these methods all rely on pixel-wise noise prediction, which are suboptimal in larger models and are vulnerable to real-world perturbations.Starting from Denoising Diffusion Probabilistic Model (DDPM) [14, 39] , generative diffusion models have gained significant attention in recent times and achieved remarkable breakthrough across diverse applications [6, 22, 28, 31, 33] . The training goal of diffusion models is to learn the reverse denoising process of gradually transforming Gaussian noise into signal. Score-based generative models train a neural network to forecast the score function, enabling the generation of samples through Langevin Dynamics [42] [43] [44] . The sampling process can either be a Markov process like DDPM, or a non-Markov process, such as DDIM [40] . Non-Markov process like DDIM can be used to accelerate the generating process.Except for unconditional generation from pure noise, diffusion models have also been explored for conditional generation, such as text-guided image generation. The text-to-image model [30] incorporates an image encoder-decoder framework to efficiently conduct the diffusion and denoising process within a latent space. The encoder compresses the input sample into a latent representation, while the decoder reconstructs the latent sample back to pixel space. Classifier guidance [6] and classifier-free guidance [13] are both proposed for high-quality image generation conditioned on various textual prompts.Although diffusion models have demonstrated superior generation performance, elucidating the generation process poses significant challenges. Until now, several researches have tried to explore and analyze the generation process. Choi et al [4] have conducted experiments on measuring the LPIPS distance of two different images under various time steps. They conclude that diffusion models learn coarse features and structures when the Signal-to-Noise Ratio (SNR) is low, whereas they learn more subtle and imperceptible features as the SNR becomes higher. Based on their observations, Wang et al. [46] design an encoder to provide comparatively strong conditions for the diffusion model when the SNR is below 5𝑒 -2 in the super-resolution image generation task. Likewise, Kwon et al. [20] also verify that modification of the generation process in the early denoising stage can achieve larger high-level semantic changes. Furthermore, Park et al. [26] conduct exponential sampling to carry out an analysis of the generation process. They conclude that in the early denoising stage, the diffusion models establish spatial information representing semantic structure, and then widen to the regional details of the elements in the later stage. 
method: 
 Given an image 𝑥 0 , our goal is to infer whether 𝑥 0 belongs to the training set of a diffusion model 𝜖 𝜃 . Current methods mainly leverage pixel-level memorization. We argue that for large-scale model, its memory mechanism is beyond pixel-level to structure-level. To demonstrate this, we first explore the structural changes throughout the diffusion process. We find that the structural information is largely maintained in the initial steps, and the members' structures are better preserved as the diffusion models have seen the structures of members during the training process (Section 3.2). Based on this observation, we design a structure-level MIA for text-toimage diffusion models (Section 3.3). The overview of our proposed method is shown in Figure 3 .Text-to-Image Diffusion Models. Distinct from other traditional generative models, diffusion models contain two processes: the diffusion (forward) process and the denoising (backward) process. During the diffusion process, diffusion models iteratively introduce Gaussian noise to the original image 𝑥 0 with a total steps of T:EQUATIONwhere:EQUATIONand the variance schedule 𝛽 1 , ..., 𝛽 𝑇 is predefined. As t approaches T, 𝛽 𝑡 becomes closer to 1.During the denoising process, diffusion models generate image through multiple denoising steps starting from Gaussian noise:EQUATIONwhere:EQUATIONand Σ 𝜃 (𝑥 𝑡 , 𝑡) is a constant depending on 𝛽 𝑡 , 𝜇 𝜃 (𝑥 𝑡 , 𝑡) is predicted by a neural network 𝜖 𝜃 as:EQUATIONUnder this formulation, in text-to-image diffusion models, we use classifier-free guidance [15] to guide the image generation by textual prompts y. The degree of text influence is controlled by adopting Eq.6 and adjusting the unconditional guidance scale 𝛾.EQUATIONDDIM Inversion. To expedite the denoising process and ensure a unique output, deterministic DDIM sampling [40] has been introduced, thereby enabling a skip-step strategy. Then for the diffusion process, a simple inversion technique, named DDIM inversion, has been suggested for the DDIM sampling. Such inversion process in Eq.7 provides a deterministic transformation between an input image and its corrupted version.EQUATIONWe also give more mathematical details in Supplementary Materials.To better capture the structure-level memorization of diffusion model, we first explore the changes in structure information throughout the diffusion process. Current arts [4, 20, 26, 46] show that during image generation, diffusion models focus more on imperceptible details when the noise levels are minimal, while concentrating on high-level context when faced with high noise levels. Similarly, but more carefully, we especially focus on the changes in structural information of both members and non-members throughout the unidirectional diffusion process. We leverage the structural similarity (SSIM) [47] as a metric. During the diffusion process, the original image 𝑥 0 is gradually corrupted by noise. A lower SSIM between 𝑥 0 and its corrupted version 𝑥 𝑡 indicates greater structural loss. We first explore the decrease rate (𝑣) of SSIM throughout the whole diffusion process for both members and nonmembers:EQUATIONFigure 2 (a) depicts the average decrease rate over 500 members and 500 nonmembers. It can be noted that the rate of decrease in SSIM between original images and its corrupted version is observed to initially increase and then decrease. More significantly, the decrease rates for members and nonmembers exhibit distinct behaviors. Nonmembers exhibit a higher rate of decrease when the diffusion timestep t ranges from 0 to approximately 100. This suggests that, for images that have been exposed to the diffusion models during training, their structures are more apt to be maintained in the early diffusion steps compared to images that are not included in the training set. However, as the images are further corrupted, the structural information is diluted by noise. The rate of decrease in structural similarity for members is even greater than that for non-members.Given the difference in decrease rate among members and nonmembers, we further assess the average SSIM difference (△𝑆𝑆𝐼𝑀) between the member set 𝐷 𝑚 and the hold-out set 𝐷 ℎ :△𝑆𝑆𝐼 𝑀 (𝑡) = 1 |𝑋 𝑚 | ∑︁ 𝑥 0 ∈𝑋 𝑚 𝑆𝑆𝐼𝑀 (𝑥 0 , 𝑥 𝑡 ) - 1 |𝑋 ℎ | ∑︁ 𝑥 0 ∈𝑋 ℎ 𝑆𝑆𝐼𝑀 (𝑥 0 , 𝑥 𝑡 ) (9)where 𝑋 𝑚 ∼𝐷 𝑚 , 𝑋 ℎ ∼𝐷 ℎ . Figure 2 (b) depicts the average SSIM difference over 500 members and 500 nonmembers. The structural similarity for the member set is larger than that for the hold-out set in the first 800 diffusion steps. Besides, the difference in structural similarity between the member set and the hold-out set gradually increases during the first 100 diffusion steps, reaching a maximum at around step 100, which serves as an important clue for dividing member set images and hold-out set images. These findings offer a foundation for our proposed straightforward MIA strategy.Following the intuition above, we introduce a simple yet effective membership inference attack method for text-to-diffusion models, centered on the structure similarity between the original image and its corrupted version. As shown in Figure 3 , we input an image 𝑥 0 into the encoder of the text-to-image diffusion model, thereby obtaining its latent representation 𝑧 0 . We also adopt the BLIP [21] model to extract a caption from image as textual prompt, since in practical applications, it is difficult to obtain the training-time texts corresponding to the images. Then we follow Eq. 7 to perform DDIM inversion to 𝑧 0 in the latent space, and get the corrupted latent 𝑧 𝑡 . Subsequently, we utilize the decoder of the text-to-image diffusion model to transform 𝑧 𝑡 back to the pixel space and get 𝑥 𝑡 . The ingenious application of the encoder and decoder in the text-to-image model enables imagelevel comparison, facilitating the extraction of intricate structures without noise interference. By computing the structural similarity (SSIM) between 𝑥 0 and 𝑥 𝑡 , we obtain a membership score for 𝑥 0 and predict its membership as the following:EQUATIONThis indicates that we consider an image is a member of the training set of the target model 𝜃 if SSIM(𝑥 0 , 𝑥 𝑡 ) is larger than a threshold 𝜏. 
experiments 4.1 experimental setup: 
 Target Models and Datasets. We utilize two prominent text-toimage diffusion models: the Latent Diffusion Model and the Stable Diffusion-v1-1, trained on the LAION-400M [36] and LAION2Ben [35] datasets, respectively. We conduct experiments on the two models without further fine-tuning or other modifications. For the datasets, the LAION-400M dataset comprises 400 million textimage pairs, while LAION2B-en, a subset of LAION-5B, contains approximately 2.32 billion English text-image pairs. These datasets are crawled from the Internet which are general and diversified. Additionally, we employ the COCO2017-Val dataset, which includes 5,000 images and is commonly adopted for model evaluation. Implementation Details. For the two target models, we both use the 5000 images in COCO2017-Val as the hold-out set. As for member set selection, we randomly sample 5000 images from the LAION-400M dataset as the member set for the Latent Diffusion Model; and we also randomly sample 5000 images from the LAION2B-en dataset as the member set for the Stable-Diffusion-v1-1. Our experiments are conducted across two image resolutions: 256x256 pixels and 512x512 pixels. Besides, we adopt DDIM inversion (Eq. 7) with an interval of 50 and incorporate noise addition twice during the forward diffusion process. Evaluation Metrics. In order to evaluate the performance of our proposed method, we adopt the widely used metrics [16] : Attack Success Rate (ASR), Area-Under-the-ROC-curve (AUC), Precision and Recall. We also follow the metrics used in [2] , including the True Positive Rate (TPR) when the False Positive Rate (FPR) is 1% (TPR@1%), and the True Positive Rate when the False Positive Rate is 0.1% (TPR@0.1%). (More details about experimental setups can be found in Supplementary Materials) We compare our method with three current MIA methods for diffusion models, including PIA [19] , SecMI [7] , and Naive Loss [24] . We leave the details of baselines in the Supplementary Materials.Evaluation on Latent Diffusion Model. Table 1 shows the results on the Latent Diffusion Model. Compared to baselines, our method exhibits remarkable performance enhancements, particularly for images with resolution 512, where it surpasses all baselines in AUC, ASR, Precision, and Recall metrics. Notably, it achieves a 14.1% increase in AUC and a 12.2% increase in ASR compared to the next best method. For images with resolution 256, our method still outperforms baselines in AUC, ASR, and Recall, albeit with a marginal decrease in Precision. The ROC curve and log-scaled ROC curve is depicted in Figure 4 . We also consider the TPR at very low FPR, i.e. 1% and 0.1% FPR, as shown in Table 3 . Our method consistently outperforms in all assessments, underscoring its superiority in MIA performance. Particularly, its effectiveness significantly increases for images with resolution 512. This reveals large-scale model's structure-level memorization and highlights the potential of our method for more precise MIA on high-resolution images.Evaluation on Stable Diffusion. Table 2 shows the results on the Stable-Diffusion-v1-1. Our method significantly exceeds baselines in AUC, ASR, and Recall. For images with resolution 512, it shows a 15.4% improvement in AUC and a 13.2% improvement in ASR over the nearest competitor. For images with resolution 256, our approach maintains a 7.3% higher AUC and a 5.5% higher ASR than the second-best method, despite a slight 1.4% reduction in Precision. Besides, the ROC curve and log-scaled ROC curve is depicted in Figure 5 . The TPR at 1% FPR and 0.1% FPR is illustrated in Table 4 . The results consistently demonstrate our method's ability to Total Timestep. To evaluate the impact of the hyper-parameter total diffusion timestep T, we vary T from 50 to 800, with a fixed interval (𝑡 𝑖 =50). For instance, setting T=200 involves adding noise from t=0 to t=200 in 50-step increments, totaling 4 query times. Experiments are conducted using the Latent Diffusion Model on images with resolutions 512 and 256. Results are shown in Table 5 . AUC and ASR metrics remain stable between T=50 and T=200, then begin to decrease from T=300, continuing to drop with further increases in T. Notably, at T=800, AUC falls below 50%. The outcomes align with our findings outlined in Section 3.2. With total diffusion timesteps under 300, the model maintains the structural integrity of member images more effectively, distinguishing them from non-member images. As T increases over 300, noise accumulation adversely affects the structures of both member and non-member images, thus reducing the attack effectiveness. Interval. To investigate the influence of the hyper-parameter interval 𝑡 𝑖 , we fixed the total timestep at 100 and varied 𝑡 𝑖 from 1 to 100. Using the Latent Diffusion Model, we conducted experiments on images with resolutions 512 and 256. As demonstrated in Table 6, there is minimal variation in AUC and ASR across different 𝑡 𝑖 settings, possibly due to our method's reliance on a deterministic diffusion process that eliminates random noise in each step. Thus, changes in 𝑡 𝑖 do not significantly affect the image's structural information. Nonetheless, a higher 𝑡 𝑖 value within the fixed total timestep implies more queries and higher computational costs. Therefore, we opt for 𝑡 𝑖 =50 as a practical compromise. TPR@1%↑ TPR@0.1%↑ TPR@1%↑ TPR@0.1%↑ TPR@1%↑ TPR@0.1%↑ TPR@1%↑ TPR@0. In real-world scenarios, images undergo various distortions, like noise and brightness fluctuations, during transmission. Additionally, augmentation techniques are often applied to modify training data for large-scale diffusion models, leading to the discrepancies between training images and their originals. This necessitates the robustness of our method to such variations. We evaluate our method's robustness using the Latent Diffusion Model on images with resolution 512. Four degradation techniques are applied to images:• Additional noise. Salt and Pepper Noise, which randomly corrupts 10% of the pixels in each image, is added to images. • Rotation. Images are rotated by 10 degrees counterclockwise around the geometric midpoint. • Saturation. The saturation levels of images are adjusted, either increased or decreased by 50%, with equal probability. • Brightness. The saturation levels of images are altered, either increased or decreased by 50%, with equal probability.Results are shown in Table 7 and Table 8 . It is evident that our methods achieve the highest results in ASR, AUC and TPR at 1% FPR across all four types of distortions. Notably, our structure-level approach exhibits exceptional resilience against additional noise, whereas other baseline methods experience a significant decline in performance. This is attributed to their reliance on noise-level comparison, which renders them vulnerable to such disturbances. Collectively, these experimental results underscore the superior stability and robust nature of our methods in effectively handling diverse distortions.All the baseline MIA methods for diffusion models involve both the forward diffusion process for noise introduction, and the backward denoising process for noise prediction. On the contrary, our method only leverages the forward diffusion process. We argue that during the initial diffusion process, as the structures of nonmember images are more severely corrupted than those of members, the structural differences between members and nonmembers have widened significantly. Conversely, the denoising process, which acts as the inverse of the diffusion process, reconstructs both the corrupted member images and nonmember images to their original states, which leads to the reduction in the image structural differences between members and nonmembers.To validate this findings, we utilize the comparison of the original images and their backward reconstructed states for MIA, and make a contrast with our method. We conducted experiments using the Latent Diffusion Model on images with resolutions 512 and 256. The results are illustrated in Table 9 . We observe that employing backward reconstruction results in a decrease in AUC and ASR by roughly 3%, regardless of the image resolutions. The TPR at 1% and 0.1% FPR also decrease to a large extent when using backward reconstruction. This reveals the superiority of our method in utilizing the unidirectional diffusion process for MIA, compared to other bidirectional methods.To evaluate the influence of texts on our method' performance, we delve into the impact of the unconditional guidance scale 𝛾 using the Latent Diffusion Model on images with resolutions 512 and 256. We use classifier-free guidance to guide the image generation by textual prompts. The degree of textual influence is controlled by adopting Eq. 6 and adjusting 𝛾. Specifically, setting 𝛾 to 0 renders the model entirely unconditional, while a setting of 1 makes it fully conditional, guided solely by text, forming the basis for our experiments. As 𝛾 increases, the influence of textual information is increasingly pronounced. Here we vary 𝛾 from 0 to 5. Results are shown in Table 10 . All four metrics remain virtually unchanged across varying scale values, suggesting that textual information has minimal impact on structural similarity during the initial diffusion stage. The models preserve image structures well, irrespective of the presence of textual information when the noise levels are minimal.To further explain this result, we conduct the reconstruction experiments, where we corrupt a image in the diffusion process to a certain timestep T, and then restore it in the denoising process. We compare the structural similarities between original images and their reconstructions under three conditions: captions from the BLIP model, empty prompts, and unrelated texts. One of the results is shown in Figure 6 (a) (b) (c). Notably, at T=200 (the noise level is low), reconstructions across all types of prompts are similar. However, as T increases to 500 and 800 (the noise levels are high), variations in reconstruction outcomes become pronounced. This indicates that textual impact on image structure is minimal at low noise levels, where models prioritize detailed features. Conversely, when faced with higher noise levels, where models focus more on overall structure, unrelated texts significantly influence the reconstruction results, underscoring the guiding role of textual information.We also plot a trend curve depicting how structural similarity between the original and reconstructed images changes with the total diffusion step T. As shown in Figure 7 , when T is below 300, structural similarity remains consistent across different prompts. However, as T increases, structural similarity experiences the sharpest declines with an unrelated prompt. These results suggest that the text impact on our method is minimal, since we only assess structural similarity in the initial diffusion phase where the noise levels are minimal. 
conclusion: 
 In this paper, we explore the structure-level memorization of largescale text-to-image diffusion models. We primarily investigate the corruption of images structures throughout the diffusion process. We further demonstrate that the structures of member images in training set are better preserved than those of nonmembers in the initial diffusion stages, since models can memorize member images' structures during training. Drawing on these insights, we introduce a novel Membership Inference Attack (MIA) method for text-toimage diffusion models to judge whether an unauthorized image is utilized for training a diffusion model. Our proposed method is to assess models' structure-level memorization. We evaluate our method on state-of-art text-to-image diffusion models, e.g., the Latent Diffusion Model and the Stable Diffusion. Experimental results show that our method achieves higher ASR, AUC, TPR @ 1% FPR and TPR @ 0.1% FPR than all baselines. Besides, our method also exhibits greater robustness against diverse distortions and maintains efficacy across different textual prompts, underscoring its applicability in more real-world contexts. 
 
 Appendix:  


Decision: This paper focuses on the membership inference attack task for diffusion-based generative models. Instead of using pixel-level information, the authors explore the role of structural information change during the diffusion procedure and propose using SSIM for this task. The method is clear and somewhat straightforward, but it was tested on four models (two of which are added in the supplementary materials), and the overall effectiveness is demonstrated. 

After rebuttal, two reviewers provide their final ratings. One of them raised the rating from 3 to 4 and the other chose to keep the rating of 3 due to the unsolved issues. By reading this submission, the supplementary materials, the five reviews, and the rebuttal text, I think the overall contribution of this paper could benefit the development of research in this task and the remaining issues are not very hard to fix.

Reviews: review: paper_summary: paper_summary: Membership Inference Attack (MIA) is a hot topic for generative models, focusing on detecting whether the specific data sample is used in the model training. Compared with the traditional pixel-level MIA, this paper considers structure-level memorization: the structures of members are better preserved compared to those of nonmembers.
strength_weakness: strength_weakness: 1. This paper matches the scope of ACM, and it will have a large group of audience since the security and privacy issues for generative models (especially for the multimedia domain) are quite important.
2. The authors well present the method, which is very easy to understand.
limitations: limitations: 1. The idea of calculating structural similarity is quite straightforward.
2. The experiments are not adequate. The authors are expected to use more generative models (small size to large size), not limited to two models.
3.  Table 1 and Table 2 show that the proposed method cannot always achieve the best performance.
paper_summary: paper_summary: This paper investigates the structural memorization capabilities of large-scale text-to-image diffusion models and proposes a novel Membership Inference Attack (MIA) method based on structural similarity. Unlike traditional pixel-level approaches, the paper leverages structural similarity (SSIM) to distinguish between training set members and nonmembers. The experiments demonstrate the effectiveness and robustness of the proposed method across various conditions, including different image resolutions and distortions.
strength_weakness: strength_weakness: 1.	The paper is well-structured, with a logical flow from the problem statement to the proposed solution and experimental validation, making it easy for readers to follow.
2.	The introduction of a structure-level membership inference attack, leveraging structural similarity (SSIM) rather than pixel-level noise comparison, represents an advancement in the field.
limitations: limitations: 1. Clarification on Data Sampling in Section 3.2: The authors use 500 members and 500 nonmembers in Section 3.2. It should be clarified whether these two sets are sampled from the same distribution. Specifically, the authors should discuss whether the rate of decrease for these two groups would be more similar if they were sampled from the same distribution compared to being sampled from different distributions.

2. Lack of Discussion on Threshold $\tau$: The paper lacks a discussion on how the threshold $\tau$ is determined. The authors should include experiments that show how the threshold $\tau$ is set under different model architectures or datasets from different distributions, and whether there is a significant variation in the threshold value.

3. Limited Data Diversity: The experiments are primarily conducted on the LAION-400M and LAION2B-en datasets, which may not fully capture the method's performance across more diverse and specialized datasets such as medical or remote sensing images.

4. The font size in Figures 4 and 5 should be increased for better readability.
paper_summary: paper_summary: The paper proposes a member inference attack for text-to-image diffusion models. Instead of pixel-level memorization, this paper studies the advanced memorization capabilities of large diffusion models at the structure-level, and the differences in the preservation of image
structures between members and non-members during the diffusion process. 
Compared with existing advanced baselines, the proposed method is more robust to various image distortions and maintains efficacy across different textual prompts.
strength_weakness: strength_weakness: 1、The paper proposes a novel perspective to study the member inference attacks from text-to-image diffusion models.
2、The ingenious application of the encoder and decoder in the text-to-image model enables
image-level comparison. The MIA attack method based on structural similarity is proposed,which is robust to image distortion.
3、Experimental results demonstrated the effectiveness of the proposed method.
limitations: limitations: 1、The changes in structure information during the diffusion process have been studied by many
previous works [4, 20, 26, 46], thus this idea is not raised for the first time.
2、There are a large number of details that are not clearly explained. For example, there is no detailed description on how to set the threshold in Eq. (10).
3、The paper applies BLIP to obtain textual prompts, but it is not shown in Figure 3. Furthermore, there is no detailed explanation on how to utilize the textual prompts obtained through BLIP.
4、This paper does not provide source code to verify the feasibility of the method used in the paper.
paper_summary: paper_summary: This paper explores the advanced memorization capabilities of large diffusion models at the structure-level, investigating how these models preserve image structures differently between members (images used in the training set) and nonmembers during the diffusion process. Based on these findings, the paper proposes a Membership Inference Attack (MIA) method specifically tailored for text-to-image diffusion models, aiming to enhance privacy protection while maintaining robust performance.
strength_weakness: strength_weakness: S1.The paper is well organized and the conducts extensive experiments.

S2. This paper is overall well written. The problem is well formulated and clearly explained.
limitations: limitations: L1. The paper mentions the Latent Diffusion Model and the Stable Diffusion-v1-1, which are trained on the LAION-400M and LAION2Ben datasets, respectively, in Lines 454-456. Is it possible to conduct experiments with both models on both datasets?

L2. The structured MIA method may require significant computational resources to process the diffusion and denoising of images. It would be beneficial to include an analysis of the model's complexity to better understand its resource demands and efficiency.

L3. It would be great to open the code and dataset.
paper_summary: paper_summary: This paper addresses the infeasibility of pixel-level Membership Inference Attack (MIA) methods on large-scale models for determining whether a certain unauthorized image has been used in training. It proposes a novel MIA method for text-to-image diffusion model, which explores the structural-level memory of large diffusion models by studying the differences between member and non-member image structures during the diffusion process. By utilizing this structural-level memory, it achieves superior performance while also guaranteeing enhanced robustness.
strength_weakness: strength_weakness: (1)This method overcomes the limitations of traditional pixel-level MIA approaches, which rely on comparing pixel noise directly, when applied to large-scale models. By utilizing more advanced structural-level memory for MIA, it enables the attainment of excellent performance even within large-scale models.
(2)This method is more robust in dealing with various distortions of the image. Robustness experiments are conducted through four degradation techniques: adding additional noise, image rotation, modifying saturation, and modifying brightness. The experimental results show that the method advantages, especially when dealing with additional noise.
limitations: limitations: (1)The theoretical basis for the structural memory of diffusion models is insufficient when applied to MIA. The paper observes through experiments that during the diffusion process, the structural similarity of members will be greater than that of non-members. However, the paper does not provide a theoretical proof of the validity of this conclusion when applied to MIA. Simply relying on experimental observations may not be enough to fully prove the reliability and universality of this method in practical applications.
(2)The difference in structural similarity between members and non-members during the diffusion process lacks practical consideration. The difference in Structural Similarity Index (SSIM) peaks early in the diffusion process (around 0.25), and for Membership Inference Attacks (MIA) this difference needs to be large enough to reliably distinguish training set members from non-members. However, whether the difference of 0.25 is enough to achieve high-accuracy classification requires further analysis combined with a specific classification threshold τ.
(3)The paper overlooks an introduction to the stable diffusion model. While the experiments revolve around both the latent diffusion model and the stable diffusion model, the related work section only introduces concepts and principles pertaining to the latent diffusion model, completely omitting any discussion of the stable diffusion model.
(4)There are several issues present in the experimental setup of this paper. Firstly, the experimental methodology lacks clarity regarding the fundamental principles behind resolution selection. It seems inadequate to draw conclusions about the superiority of high-resolution images for MIA solely based on the general trend of resolution 512 outperforming resolution 256. Moreover, in real-world scenarios, resolution 512 may not be considered high resolution. Therefore, it remains uncertain whether higher or lower resolutions would impact structural memory and MIA effectiveness. Secondly, the experiment lacks a baseline for comparison. Simply comparing it to three methods does not sufficiently demonstrate its advantages. Thirdly, in the robustness experiment involving saturation modification, significant deviations from the optimal value are observed in the experimental results of this method when the false positive rate is 0.1%. What could be the underlying reason for this disparity?