Image-level Memorization Detection via Inversion-based Inference Perturbation
Authors: Yue Jiang
Haokun Lin
Yang Bai
Bo Peng
Zhili Liu
Yueming Lyu
Yong Yang
Xingzheng
Jing Dong
Venue: ICLR 2025 Poster (2025)
Keywords: Text-to-image diffusion model
data memorization detection
DDIM Inversion
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: we propose a simple yet effective image-level memorization detection method, namely Inversion-based Inference Perturbation (IIP).

Abstract: Recent studies have discovered that widely used text-to-image diffusion models can replicate training samples during image generation, a phenomenon known as memorization. Existing detection methods primarily focus on identifying memorized prompts. However, in real-world scenarios, image owners may need to verify whether their proprietary or personal images have been memorized by the model, even in the absence of paired prompts or related metadata. We refer to this challenge as image-level memorization detection, where current methods relying on original prompts fall short. In this work, we uncover two characteristics of memorized images after perturbing the inference procedure: lower similarity of the original images and larger magnitudes of TCNP.
Building on these insights, we propose Inversion-based Inference Perturbation (IIP), a new framework for image-level memorization detection. Our approach uses unconditional DDIM inversion to derive latent codes that contain core semantic information of original images and optimizes random prompt embeddings to introduce effective perturbation. Memorized images exhibit distinct characteristics within the proposed pipeline, providing a robust basis for detection. To support this task, we construct a comprehensive setup for the image-level memorization detection, carefully curating datasets to simulate realistic memorization scenarios. Using this setup, we evaluate our IIP framework across three different memorization settings, demonstrating its state-of-the-art performance in identifying memorized images in various settings, even in the presence of data augmentation attacks.

Published as a conference paper at ICLR 2025
IMAGE-LEVEL
MEMORIZATION DETECTION VIA
INVERSION-BASER INFERENCE PERTURBATION
Yue Jiang 1.21
Haokun L2n3 Yang Bai
Bo Peng'
Zhili Liu
Yueming Lyu
Yong Yang Xing Zheng'
Jing Dongl
NLPR, MAIS, Institute of Automation, Chinese Academy of Sciences
2 School of Artificial Intelligence, UCAS City University of Hong Kong
4 Tencent Security Platform Department HKKSS 6 Nanjing University
fyue, jiang, hackun. wwwpeeripacc.ccc.......... aaaesiiiac.....
coolcyangetencent com jdongenipr.. an.
ABSTRACT
Recent studies have discovered that widely used text-to-image diffusion models can replicate training samples during image generation, phenomenon known as memorization. Existing detection methods primarily focus identifying mem- orized prompts. However, real-world scenarios, image owners may need to verify whether their proprietary or personal images have been memorized by the model, even in the absence of paired prompts or related metadata. We refer to this challenge as image-level memorization detection, where current methods re- lying on original prompts fall short. In this work, we uncover two characteristics of memorized images after perturbing the inference procedure: lower similarity of the original images and larger magnitudes of TCNP. Building on these in- sights, we propose inversion-based Inference Perturbation (IIP), new framework for image-level memorization detection. Our approach uses unconditional DDIM inversion to derive latent codes that contain core semantic information of origi- nal images and optimizes random prompt embeddings to introduce effective per- turbation. Memorized images exhibit distinct characteristics within the proposed pipeline, providing a robust basis for detection. To support this task, we construct a comprehensive setup for the image-level memorization detection, carefully cu- rating datasets to simulate realistic memorization scenarios. Using this setup, we evaluate our IIP framework across three different memorization settings, demon- strating its state-of- tate-offthheart performance in identifying memorized images in var- ious settings, even in the presence of data augmentation attacks. Our code and datasets are available at httpt::///// //t::://gtttbb.tt wtaob...oo//e com joel lemelllejiinggnT
htttps://github.cco
INTRODUCTION
In recent years, dext-to-image diffusion models such as Stable Diffusion (SD) (Rombach et al. 2022) and DALL-E (Ramesh et al. 2022 have achieved remarkable advancements, showcasing impressive generation fidelity and diversity when conditioned on given prompts. However, these models have also been found to reproduce partial complete elements of their training images, a phenomenon broadly known as memorization (Schuhmann et al.) 2022; Carlini et al., 2023 (Somepalii et al.) 2023a Existing memorization detecting methods primarily focus on identifying whether specific prompts lead to memorized outputs (Wen et al.) 2024 Ren et al. (024) (Fig. (Top)). The proposed magnitude of text-conditional noise prediction (MTCNP) (Wen et 2024) can effectively detect
memorized prompts during the generation procedure (Fig. (Top)).
However, real-world scenarios, users may encounter situations where they need to verify whether a specific image has been memorized, either in partial or whole, by widely used text-to-image dif- fusion models. For instance, an image owner might wish to confirm whether their proprietary or personal content has been embedded into the model's memory, potentially violating copyright or
Equal contribution. Corresponding authors.

---

Published as a conference paper at ICLR 2025
privacy rights. Unlike existing upproaches that rely textual prompts that generate the memorized images, this scenario requires a direct, image-level assessment, as shown in Fig. I (bottom). Such a setting assumes that only the image itself is accessible, without information about the original paired prompts or associated metadata, making existing detection methods for memorized prompts (Wen) et al. 2024 Ren et al. 2024) struggle to address this challenge. Ma et al. (2024) analyzes the de- lection of memorized images in diffusion models and identifies three conditions for memorization. Nevertheless, they focus solely on exact duplication with a relatively small subset, which limits its applicability in real-world scenarios.
In this paper, we reveal two characteristics of memorized images, which provide the basis for our image-level memorization detection. We discover that memorized images exhibit lower similarity of the original images and maintain larger magnitudes of text-conditional noise predictions (MTCNP) (Wen et al. 2024) after perturbation of the generation procedure.
Based on these findings, we propose a novel image-level memorization detection frame- work IIP free of original prompts. We are motivated to obtain latent code with pri- mary semantic information of original images and employ perturbed inference procedure subsequently (as illustrated in Fig. hot- tom)) This perturbation aims derive both lower similarity of the original images and larger magnitudes of TCNP for memorized images, starkly contrasting with that of non- memorized images. Specifically, we employ unconditional DDIM Inversion (Song et al. 2020) at smaller timesteps to obtain semantic latent codes and perturb the inference proce- dure with optimized random prompts, which effectively derive significant characteristics for image-level memorization detection.
Memoried Prompis Detection
Giver Limmdo Supuenss Wimmas Poricnde
Stude Dillace
these given presett)
senooph art flomiie trindinng arthheliee."
Images
Traae Gallery
T31 Difaii
Saillo Dill
Image-evel Memoelzation Desection
Figure 1: Differences between memorized prompts detection and image-level memorization detection.
We distinguish three different settings of memorization for comprehensive setup, including orized vs non-memorized training images, memorized training vs non-training images (non- memorized), and the generated memorized non-memorized images. Furthermore, we simu- late a variety of real-world memorization scenarios for practical applicability, including memo- rized images with varying degrees of replication. Extensive experiments and analysis demonstrate our method consistently achieves state-of-the-ar performance across various datasets and different open-source text-to-image diffusion models.
We summarize our contribution as follows:
We formalize image-level memorization detection task for text-to-image diffusion models across various datasets with a comprchensive setup, which assists inspecting the security of training images against memorization risks.
We uncover two characteristics of memorized images under inference perturbation with random prompts. Based on these findings, we propose a novel Inversion-basee Inference Perturbation (IIP) framework to effectively detect memorized images without accessing original prompts.
Extensive experiments demonstrate that our method achieves state-of-the-ee performance in detecting memorized images in various settings. Additionally, our IIP framework shows strong robustness against data augmentation attacks.
2 MOTIVATION
2.1 PRELIMINARY
Diffusion Model. Diffusion models are probabilistic models (Ho et al. 2020) designed to learn the data distribution p(x) by progressively denoising noises. In the text-conditioned latent diffusion
2

---

Published as a conference paper at ICLR 2025
Memorized Prompes Detection
Thhe Ni girett Sustners Woman Pesfiart
Similarit Comparison
Magnitale
Ao Aiuit
WW Pruits
Jot Climit
No C Amail
Stakk Diflaskon
Woman
High Simibartyy Manmurizoa
digitel comeept of old trodin in fiórida swamys trendingg one artstario
Suabke Diflusio
Low Siilerty: Non metooreed
Thasshp
Free of origieed promptts
Image-level level Memocizatio Desection (IIP)
Magnitade of TCPP
Sinilarity Memortadd
Bus Worlin
DIUUN
DOUM
BUSI
Images
Null-teet
cole
Perturbetionn
Perturbal
Training Non-traisinng
DOUM
DOUN
Inveniken
Sampling
High Similarity: Nam momeeiieei
Figure 2: Memorized prompt detection methods (top) vs. our image-level memorization detection method IIP (bottom). Unlike previous methods, our approach focuses on identifying memorized images from model without requiring access to the original prompts. Detection is achieved by evaluating the similarity of the original image and magnitude of TCNP values (Wen et al. 2024).
models, such as Stable Diffusion (Ramesh et al. 2022 the model learns the conditional latent
distributions p(2|c) using denoising autoencoder (2(2,, t, c), where denotes the conditioned text
prompts. Based on image-text pairs, the conditional LDM Sohl-Dickstein et al. (015)) : Ho et al.
2020) is trained as follows:
LLDM BE(s)et.e (,
After training, the diffusion model generates images by employing classifier-free guidance (Ho & Salimans 2022):
E(2, co(ze, Canu)) ) + sy 4, (p) ea ca Cnmu)),
Here, 2, represents the latent code at timestep i, and 8. denotes the guidance scale. The backbone eg, derived from DDPM (Ronneberger al., 2013 Ho et al., 2020), is used for noise prediction, where Cp refers to the given prompt and Caull represents the null-text prompt.
Unconditional DDIM Inversion. During the inference stage of diffusion models, the predicted noise undergoes further sampling to generate latent codes for previous timesteps. A commonly used method for this is DDIM sampling (Song et al. 2020), which follows a determined process for generating these latent codes. DDIM Inversion reverses the DDIM sampling process (from to T') as follows:
a++l
(z.
/1 ar (o( VI 0:41 ,0(zz,t,c)
Due to its deterministic nature, DDIM Inversion enables the retrieval of latent codes for given image. Specifically, when using a null-text prompt Caull, referred to as unconditional DDIM In- version, the inverted 2f reconstruct the original image with minimal error during the denoising
process (Song et al. 2020 Mokady et al. 2023
Magnitude of text-conditional noise prediction. Memorized prompts consistently guide the gen- eration toward fixed results. regardless of the initialization. indicating significant text guidance dur- ing the denoising process. /Wen et al. (2024) find that these prompts exhibit larger values of the latter term in Eq. 2 across range of timesteps 7, which are referred to as the magnitude of text-conditional

---

Published as a conference paper at ICLR 2025
Natos Lood
Jovo C Amxits Business Wombin
A TORM Businels Woman
Co Smuile
Buenc Wordn
Business
Business
Women
Woraat
Block mSAAIO wasooulleed Loand
1000
900
Anloca
SILO
2040
Original
Proept
Pertuzb Pronge
(e.9.
"ata"!
Time-atep
Figure 3: Perturbed results during different inference timesteps for memorized and non non-memorized prompts. The magnitude of TCNN for perturbations is present on the right.
noise prediction (TCNP).
1
|e,le,, cp) co(2 t,, Gnadt)l2.
(4)
T
2.2 PERTURBATED ANALYSIS.
We conduct preliminary experiments to un- cover clues for identifying memorized images. Our approach primarily involves perturbed in- ference explore potential features. Specifi- cally, we remove original prompts at different timesteps and replace them with meaningless ones (e.g., "ata") during the subsequent genera- tion period. Interestingly, we discover two key characteristics: after permutation operations, 1) the similarity of images changes a distinct pattern, and 2) the MTCNP remains larger for memorized images.
Density
30
20
10
0,00
0.25
0.50
0.75
10'
4, Distance
Magnitude TCNP
1.0
Similarity
1.0
Magnitude TCNP
True Positive Rate
Memorized Images Exhibit Lower Similar-
ity. We apply perturbation during the genera- tion period, and the results of memorized and non-memorized images are shown in Fig. 3 (left). It can be observed that when the orig- inal prompts are replaced after step 600, the perturbed results display clear textual discrep- ancy of the original images. In contrast, non- memorized images show minimal discrepancy compared to their original images. Although in the latter inference stage, prompts reflect subtle information in most cases such as texture and color for diffusion models (Zhang et al.] [003b)] Hertz et al., 2022). memorized image-text pairs would reveal greater discrepancy of the original results when prompts are exchanged during inference.
0.5
0.5
skp (AD-
Agp KESAALL--9))
(AD-C (.95)
Sap EA (99)
stp AAG-0ADIC 0.95)
Agg 406-DAALC (93)
sup (AU- (199)
sup 210--(ALC--R))
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
False Positive Rate
Figure 4: KDE (top) and ROC (bottom) curves of different metrics for perturbations across dif- ferent timesteps. Results are evaluated using 2000 images generated from memorized and non- memorized prompts following Wen et al. (2024).
Memorized Images Maintain Larger MTCNP. We calculate the magnitude of TCNP before and after inference perturbation, with the results shown in Fig. (right). As confirmed by Wen al. (204)) original memorized prompts exhibit a significantly larger Magnitude of TCNP compared to non-memorized prompts. Our experiments extend this finding and reveal that even after applying meaningless prompt instead of the original prompt in the perturbed inference period, the magnitude TC for memorized images remains consistently larger. This phenomenon suggests that larger MTCNP does not solely depend on the original prompts, in the latter inference stage, the primary semantic information of an image obtained, which also results in larger MTCNP even without original prompts.
Further Discussion. To validate these characteristics, we further apply permutation on 2000 im- ages generated from 500 memorized and non-memorized prompts. The KDE plots of similarity
4

---

Published as a conference paper at ICLR 2025
Fanide
Nagnitatt
C
by
Natan Parturbing
Magnittude
Encoder
Fi >
2
Eg Disbnnce
Noo Pimits
Voee Pimil's
Business
Business Womann
Womta
DDIM
DDIM Sampling
Null-teet
Original Image
Latrtt code Zty
X50
Figure 5: The overview of our IIP. We first employ unconditional DDIM Inversion to obtain a latent code with primary semantic information of the original image, and perturb the subsequent inference stage with optimized non mem--mmrezed prompt embedding. Our method allows memorized images to exhibit lower similarity of the original images, accompanied by larger magnitudes of TCNP.
degree and MTCNP shown in Fig. 4 (top), with the L1 score used as the similarity measure in the latent space. Our findings indicate that memorized results exhibit lower similarity compared to non-memorized results, while consistently maintaining larger MTCNP values. In addition, apply these metrics to detect memorization in 2000 images, with the corresponding ROC plots presented in Fig. (bottom). Both metries effectively detect memorization, with AUC scores exceeding 0.9.
Based on these results, we are motivated to explore the use of latent codes that retain primary seman- tic information of the images in combination with random prompts, similar to the perturbed process starting from smaller timesteps. This procedure is likely to yield both lower of original images and larger MTCNP values for memorized images after the subsequent inference stage, which helps to effectively detect memorized images without original prompts.
METHOD
In this section, we provide detailed introduction IIP. First, we employ unconditional DDIM Inversion to obtain latent codes containing primary semantic information from the original images. Next, introduce optimized random prompt embeddings that are far from memorized during the subsequent inference stage. The perturbation is reinforced by minimizing the magnitude of TCNP to yield less memorized embeddings. thereby enhancing the sigmificant characteristies for detec- tion. The entire pipeline enables effective detection of memorized images free of original prompts, through evaluations of both the similarity between the reconstructed and original images the latent space and the magnitude of TCNP during the inference stage.
3.1 DERIVE SEMANTIC LATENT CODE
We aim to obtain the latent code containing primary semantic and structural information of the original images, which leads to significant characteristics for memorization detection. DDIM In- version (Song al. 2020 is known for its ability to reverse the sampling process and derive latent codes of original images with varying prompts. In this work, we employ unconditional DDIM Inver- sion, which demonstrates the capability nearly fully reconstruct the original images. Specifically, given the original image is Co. the latent code 21, at timestep L is obtained as follows:
Olt,
(2t, V 01,-1 co(2,, Crult) + / - oz, c0(2), -1, t, Crutt, (5)
2b,
a11-1
where Guull is the null-text prompt, co is the noise predictor (Ho et al. 2020 Rombach al. 2022 And 2,, is processed from 2g E, (zo), and E; is the image encoder (Kingma 2013) of the Diffu- sion Model (Rombach et al. 2022
Moreover, to obtain more semantic information of the original images, we employ the inverted latent codes at smaller timestep I,, which shortens the inverted steps and also reduces inversion error.
5

---

Published as a conference paper at ICLR 2025
3.2 OPTIMIZE CON-MEMORIZED PROMPT PERTURBATION
To derive significant characteristics for memorization detection, our goal is to perturb the subsequent inference started from the obtained latent code =t, with prompts far from memorized. Given the random prompt is ca and subsequent inference stage starts from Zt,, the magnitude of TCNP is calculated follows:
(6) To obtain the prompts far from memorized, we optimize the embedding of ca by minimizing the magnitude of TCNP across range of timesteps (from ł to ). To avoid straightforward optimiza- tion reductions final magnitude and obtain an embedding of ca far from memorized, we add noises to obtain an approximate latent code z/, and constrain the embedding within a certain distance from the null-text embedding. The embedding of ca is optimized as follows:
(,, - C,
Led =
Ad
|Koo(e), t, ca) - co(e t, Gradi)l2 + de ([Ea(ca)) - Ee(Cnaut)le
where Eg is the text encoder (Kingma 2013 of the Diffusion Model (Rombach et al. 2022), and
Xa. Ar are the Syyee-parametent
After optimization, we start inference using the latent code 2;, from timestep f, along with the optimized prompt embedding ča. The subsequent latent codes are predicted following DDIM sam- pling (Song et al. 2020):
01-1 -a--VI- vI- co(2i,,,E)) v1- aq
Finally, we can effectively detect memorized images without original prompts using both metries of similarity to the original images and the magnitude of TCNP. Specifically, we find that L1 distance in the latent space demonstrates greater effectiveness in our experiments, as is sensitive to even subtle image discrepancies. The two metrics are calculated as follows:
IIP sim
20.
20.1
(9)
fy [|eo(2t L, ea(ze, l, Cauat)
IIP mtcnp
(10)
where and Eg are the latent codes of the original image and the generated image of our IIP.
4 EXPERIMENT
4.1 EXPERIMENTAL SETUP
Datasets. We evaluate method across three dataset types for comprehensive analysis in various settings: 1) memorized and non-memorized training images, 2) memorized training images and non-training images, and 3) generated images from memorized and non-memorized prompts. All settings in the main paper are based on Stable Diffusion v1.4 (Rombach et al. 2022) and results on other versions of Stable Diffusion are provided the appendix. For memorized training images, follow Webster 2023) and collect 152 memorized images from the training set of SD v1.4. These images have an SSCD similarity score (Pizzi et al. 2022) exceeding 0.7, which strongly indicates duplication (Somepalli et al. 2023alb). For non-memorized training images, we collect images from the member sets LATON-mi and filter those with an SSCD similarity score lower than 0.15, retaining 1200 images. Since LAION-mi (Dubiñski al. 2024) extract part of the training set and non-member sets for Stable Diffusion v1.4, we select 1200 non-member images from LAION-mi as memorizer non-training images. For generated images, follow the setting of Wen et al. 2024), selecting 500 memorized prompts from Webster (2023) for Stable Diffusion v1.4 and 500 non-memorized prompts from various sources, including LAION (Schuhmann al. (022). COCO
[Lnn et al. 2014 Lexica.ar[] and randomly generated strings. Each prompt generates 4, 8, and 16
Ffusi Proompp

---

Published as a conference paper at ICLR 2025
images, resulting in total of 2K, 4K, and 8K generated images for both types of prompts. We approximate that images generated from memorized prompts are highly memorized, albeit with slight differences. The following Table I
presents the details of each category.
Datset
Type
Number
Source
Filter
Memorized Training
Memorization
152
Training set SDv1.4.
SSCD >0.7 SSCD c0.15
Non memorized Training
Non Memorization
1200
Memher set of LAION mi
Non-Training
Mon-Memorization
1200
Non-member set of LAION-mi
Memorized Generated
Memorization
2K RK/4K/8K
Memorized prompts
Non-menorized Generated
Non- -om-Mmorrizatinn
2K EX/4K/88 SK
Non-memorized prompts
Table Detailed illustration of all datasets.
Existing Baselines. While existing methods explicitly claim to detect memorization on image- level without paired prompts, we consider two types of baselines. First, methods that identify mem- orization by assessing the similarity between the original and reconstructed images using null-text prompts. For this, we select DDIM Inversion| | image inpainting [ and image-to saage-to-image -imag generation] ] as representative approaches. Specifically, for inpainting, we use 16 16 grid mask that matches the original image size (Wu et al., (024a). Second, membership inference attack methods serve as potential baselines, as they might detect memorization through stronger membership clues. We use two tatte-ff-hhe-rt methods, SecMI (Duan et al. 2023) and PIA (Kong et al. (023), for comparison. All baselines, as well as our method, are implemented based on Stable Diffusion v1.4.
Evaluation Metries. We calculate the Area Under the ROC Curve (AUC), and the True Positive Rate the False Positive Rate of 1 1 (TPR @ 1%FPR) as metrics for all three datasets to evaluate the effectiveness of different methods. Additionally, because of the significant difference in the number of memorized and non-memorized images in both the training and non-training datasets, compute the Area Under the PR Curve (AUC-PR) for these datasets. We compare the Accuracy (Acc.) for the generated datasets, which contain a balanced number of both types of images.
Implementation Details. All baselines employ the same DDIM sampling (Song et al. 2020). with inference steps and guidance scales consistently set to 50 and 7.5, respectively. Importantly, none of the methods access the original prompts for detection. In our experiment, we set I = 20 and
ti 10, and hyperparameters Aa= = 1.0, de = 1.0. All experiments are conducted on a single A A100.
4.2 EXPERIMENTAL RESULTS
Mem (Train) / Non-mem (Train)
Mem (Train) / Non-mem (Non-train)
Method
AUC
AUC-PR
@PR@1SFPR
AUC
AUC-PR
TPR ISFP
DDIM Inversion
0.938
0.699
0.388
0.943
0.689
0.395
Inpaint
0.671
0,153
0.007
0.628
0.139
0.007
Emage-to-imant
0.721
0,170
0.000
0.798
0.227
0.000
SecMI
0.584
0.130
0.000
0.557
0.124
0,030
PIA
0.507
0.104
0.000
0.536
0.112
0.000
IIP
0.999
0.995
0.987
1.000
0.998
0.993
0.995
0.962
0.842
0.997
0.976
0.875
Table 2: Quantitative results for different methods of detecting memorized images across different
datasets. The best and second-best results are highlighted in bold and underline.
4.2.1 QUALITATIVE COMPARISON
Since SecMI and PLA primarily depend the model's intermediate outputs and do not generate images, we conduct the qualitative comparison with reconstruction based methods: DDIM Inver- sion, image inpainting, and image-to-image. Results are illustrated in Fig. Notably, these baseline methods reconstruct images similar to original images for both memorized and non-memorized images, hardly determining memorization. In contrast, our method reveals significant textual in- consistencies in the reconstruction of memorized images, particularly in texture and certain colors,
stable
diffusion/ing/ime
ne

---

Published as a conference paper at ICLR 2025
Non-mem
Non-man
Non-nek
Mem
(Train)
Mom (Gen)
(Train)
Mionntrain)
(Gan)
TOKTO
POP
CULTURE
TOKYO
POP
mith
TOKYO POP
GULTUR
TOKYO
POP
CULTUPE
TOKYO POP CULTUPE
Figure 6: Qualitative comparisons with baseline methods under different settings. Our IIP demon- strates the most distinguishable results between memorized and non non-memorized images (red boxes).
Method
AUC
Acc.
TPR@19 % FPR
42K
#4K
#8K
AVG
42K
#4K
ASK
AVG
42K
#4K
#8K
AVG
DDIM Inversion
0.739
0.741
0.733
0.738
0.681
0.677
0.674
0.677
0,062
0.062
0,075
0.066
Inpaint
0,919
0.921
0,919
0.920
0.879
0.875
0.871
0.875
0.621
0.640
0.623
0.628
Image-tn-nne imagee
0.934
0.933
0.892
0.887
0.890
0.722
0.719
0.703
SecMI
0.631
0.630
0.629
0.630
0.614
0.610
0.608
0.611
0,061
0.049
0,057
0.056
0.710
0.711
0.710
0.653
0.649
0.651
0.651
0.074
0.096
IIP
0.964
0.965
0.963
0.964
0.935
0.939
0.936
0.937
0.866
0.867
0.866
0.866
0.980
0.978
0.977
0.978
0.956
0.958
0.957
0.957
0.920
0.920
0.921
0.920
Table 3: Quantitative comparisons with baseline methods across varying sizes of generated datasets. Key results are highlighted according to Tab.
as highlighted in red boxes. In comparison, non memorized images closely resemble the originals. Overall, our method performs the best in distinguishing memorized images qualitatively.
4.2.2 QUANTITATIVE COMPARISON
To comprehensively evaluate the effectiveness of different methods, conducted quantitative periments across various datasets (Tab. and B3. We utilized the L, distance to assess the similarity, which generally outperforms L2 distance for most methods (see (A L2 results). In addition to the similarity metric, our method can also employ the magnitude of TCNP as another metric.
We present the results for detecting memorized images from non-memorized training images and non-training images in Tab. Among baseline methods DDIM Inversion is the most effective and achieves AUC values exceeding 0.9. However, its TPR @ F&PP values are low, similar im- age inpainting and image-to-image, indicating in a relatively high miss rate for memorized images, SecMI and PIA, exhibit limited effectiveness, yielding the lowest AUC and TPR @ FPPP values. This may stem from that cues of memorization are less prominent for these methods. In contrast other methods, our approach demonstrates nearly perfect discriminative performance, achieving AUC values exceeding 0.99, PR-AUC values greater than 0.96 for both metrics, and TPR @ F%F values surpassing 0.98 for the similarity metric.
For detecting memorization on generated datasets (Tab. 3. we observe that image inpainting and image-to-image methods show notably improved performance, particularly in TPR@ % FPR, with the best results exceeding 0.7. SecMI and PIA also show slight improvements. This may be at- tributed to the generated images being potentially influenced by the model, which may activate in- termediate outputs. However, DDIM Inversion diminishes its effectiveness, as both generated mem- orized and non-memorized images are more easily recoverable after inversion. Nonetheless, our method consistently demonstrates superior performance, with both AUC and Acc. exceeding 0.9 for
8

---

Published as a conference paper at ICLR 2025
Method
Mem (Train)/ / Non-mem (Train)
Mem (Train) Non-mem (Non-train)
Mem (Gen) / Non-mem (Gen)
AUC
AUC-PR
AUC
AUC-PR
TPRØ I%FPR
AUC
Acc.
1PR@19HR
DDIM Inversion
0.847
0.303
0.046
0.865
0.345
0.046
0.792
0.732
0.084
Inpaint
0.901
0.495
0.092
0.891
0.494
0.092
0.618
0.397
0.035
Image-to-inuge
0.562
0.114
0.510
0.103
0.642
0.609
0.043
SocMI
0.525
0.110
0.000
0.513
0.107
0,000
0.534
0.523
0.019
0.508
0.107
0.000
0.504
0.104
0.000
0.621
0.586
0.041
TIP sin
0.99%
0.959
0.822
0.998
0.984
0.914
0.892
0.819
0.452
IPP mtxp
0.823
0.662
0.480
0.838
0.699
0.539
0.918
0.844
0.636
Table 4: Quantitative comparisons with baseline methods across different datasets under attacks.
two metrics, and the results of MTCNP showing TPR @ 1% FPR surpassing 0.9. Interestingly, we observe that the similarity metric is more effective in detecting memorized training images, while the MTCNP performs better on the generated dataset. This discrepancy may stem from model con- sistency, as generated images tend to evoke memorization characteristics more readily and are more easily reconstructed.
4.2.3 PERFORMANCE OF DIFFERENT METHODS UNDER DATA AUGMENTATION ATTACKS
We conduct experiments on various methods under different data augmentation attacks, including random flip, rotation, and color changes, widely used data augmentation techniques for training or image preprocessing, to evaluate the robustness of different methods for detection. The results in Table show that most methods cxhibit reduction in performance. For detecting memorization from non-memorized training and non-training datasets, only DDIM Inversion, image inpainting, and our approach retain effectiveness, with AUC values exceeding 0.6. However, these methods ex- hibit TPR @ %FPR values lower than 0.1, and their AUC-PR scores are also below 0.5, indicating significant probability of false positives. In contrast, our method, based the similarity metric, demonstrates the best overall performance and exhibits superior robustness against data augmenta- tion attacks, Additionally, for detecting memorized images from generated images, DDIM Inversion shows good robustness in terms of AUC but with the TPR@ @ 1FFPR value falling below 0.1. Other methods all suffer significant performance declines. However, our approach maintains the best per- formance, with the MTCNP metric demonstrating enhanced robustness for generated images. The two metrics complement each other, resulting in our method exhibiting the most stable and robust performance against data augmentation attacks.
Compoaaen
Methed
(Train) Non-men (Train)
AIX-PR
ALC
(Gen)
Apc
1.U
Similerty
1.0
Magritude of TCMP
Sin.
0.338
0.358
with NPP
Sim.
0.901
0936
0.966
0336
0.:61
0.058
Thue Postiive Rate
Full Moddi
wil
Siz.
Sim.
MICCP
0.900
0,77
8.999
0.995
0.37
8.955
0362
0.77
0367
6.42
0664
D158
6:64
0.50
0319
0335
0.956
0.536
0:920
0.666
0.730
0.5
002 (M09-3 JALC- 821)
0.5
488-)44- 1800
Table 5: Ablation studies of different components in our IIP. The N.P.P. denotes the memorized prompt perturbation module and the S.L.C. denotes semantic latent code.
AR0OAACC 169)
M081AIX 6.9%
0.0
0.0
0.5
False Positive Rate
Figure Effects of perturbations.
4.3 ABLATION STUDY
We conducted the ablation study both the training and generated datasets, and the results are present in Tab. 5 We observe that with the introduction semantic latent codes, the performance of our method improved across both datasets. Furthermore, the memorizd prompt perturbation module not only facilitates the availability of the MTCNP metric but also improves the effectiveness of the similarity metric. However, due insufficient semantic information to activate memorization clues, the TPR @1 1FPR values for both the similarity and MTCNP metrics remain relatively low. Whereas, the full model exhibits the best performance.
We also analyze our performance with initializations at different timesteps on the training dataset in Fig. 7 The similarity metric demonstrates better performance in detecting memorized training images. Both the two metrics exhibit improvements from step 1000 to slep 400. Specifically, the similarity metric can nearly identify all memorized images starting after step 600, while the magni- tude of TCNP shows a slight decrease in the AUC value starting at step 200. This may be attributed to minor distribution differences between the training and generated memorized images.

---

Published as a conference paper at ICLR 2025
5 RELATED WORK
5.1
TEXT-TO-IMAGE DIFFUSION MODELS
Recently, pre-trained models have achieved great success in real-world applications (Lin et al.
2024b Wu et al., 2024b Lin et al.] 2024a, Zhang et al. 2024, Xiong et al.) 2024 Chen et al.
2025 Especially, with the development of large image-text paired datasets (Schuhmann et al.
2022
2021 and diffusion models (Ho et al., 2020 Song et al., 2020;
Sohl-Dickstein et al. 2015 text-
conditioned generation based on diffusion models has sparked significant interest text-conditioned
generation for various image-generation tasks (Hertz et al. 2022, Zhang et al., 2023c Brooks et al.) 2023). Stable Diffusion, one of the most widely used diffusion models, demonstrates impressive fidelity and diversity. SD has been integrated into many downstream models like ControlNet (Zhang) ct al. 2023a) and DreamBooth (Ruiz et al. 2023). driving advancements in generative tasks.
5.2
MEMORIZATION IN DIFFUSION MODELS
Carlini et al. 2023); Somepalli et al. 2023aa b) have revealed the widely employed text-to-image diffusion model, Stable Diffusion (Rombach et al. 2022 and Imagen (Saharia et al. 2022
produce training data during generation, which includes not only exact duplications but also general compositions from the training samples (Webster 2023 Carlini al. (2023) extract memorized image-fext pairs by generating a great number of images with various prompts and searching for the most similar training images. /Wesster (2023) propose white and black attacks to extract mem- orized image-text pairs by the amount of noise modification in one-step synthesis and the edge consistency in generated images. Recently research has focused on detecting memorization from prompts. [Wen et al.] (2024) propose that the magnitude of predicted noise and effectively detect memorized prompts. The clue of MTCNP has been a significant source of inspiration for our work. Ren et al. (2024) find that the distribution of cross-attention exhibits great differences between mem- orized and non-memorized prompts. Ma et al. (2024) pioneered the analysis of detecting memo- rized images in diffusion models without original prompts and proposed an adaptive algorithm for this task. However, their work focuses solely on exact memorization with a limited number of memorized images. Differing from the works mentioned above, we uncover two characteristics for image-level memorization detection and formalize the task across various datasets and real-world scenarios with comprehensive experimental setup, enhancing practical applicability.
Membership inference attack (Shokri et al. 2017) can effectively distinguish between training and non-training images but struggles to differentiate between memorized and non-memorized training
images, as both are classified simply as training data. For diffusion models, MMtssuooto n et al. (2023):
Hu Pang (2023) focus on unconditional diffusion models that do not require prompt access. Car-
lini et al. (2023) propose a query query-based method using diffusion losses, while Kong et al. (2023)
employ DDIM Inversion as the ground-truth trajectory infer memberships. Dubiński et al. (022))
introduce a dataset for membership inference attacks on Stable Diffusion and propose an attack tar-
geting process modification in diffusion models. Matsumoto et al. (2023): Hu & Pang (2023) focus on unconditional diffusion models that do not require prompt access. Unlike these membership in-
ference attack methods, our approach focuses on determining whether an image is memorized by the model instead of used for training, without accessing the original prompts.
CONCLUSION
In this paper, we first reveal two key characteristics of memorized images under inference pertur- bation during the original generation period: 1) lower similarity the original images, and 2) larger magnitudes of TCNP. Based on these observations, we propose a novel image-level memorization detection framework IIP. We are motivated to derive significant characteristics. Specifically, we ob- tain the semantic latent code of original images using unconditional DDIM Inversion and optimize the non-memorized prompt embedding for perturbation in the subsequent inference stage. After perturbation, the memorized images would exhibit lower similarity to the original images and larger magnitude of TCNP, which contrasts with the non-memone non-memorized images. These distinct char- acteristics facilitate effective detection. Extensive experiments demonstrate the state-of-the off-the-an art performance of our method across different datasets in identifying both training and generated mem- orized images, even under data augmentation attacks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work supported by the National Natural Science Foundation of China (NSFC) under Grants No. 62272460, Beijing Natural Science Foundation under Grant No. 4232037and the Special Fund for Key Program of Science and Technology of Jiangsu Province under Grant No. BG2024042.
REFERENCES
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. CVPR, pp. 18392-184022 2023.
Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 5253-5270, 2023.
Siran Chen, Yuxiao Luo, Yue Ma, Yu Qiao, and Yali Wang. H-mba: Hierarchical mamba adaptation for multi-modal video understanding in autonomous driving. arXiv preprint #rxiv:2501.04302, 2025.
Jinhao Duan, Fei Kong, Shiqi Wang, Xiaoshuang Shi, and Kaidi Xu. Are diffusion models vulner- able to membership inference attacks? In International Conference on Machine Learning, pp. PMLR, 2023.
Jan Dubiński, Antoni Kowalczuk, Stanislaw Pawlak, Przemyslaw Rokita, Tomasz Trzciński, and Pawel Morawiecki. Towards more realistic membership inference attacks on large diffusion mod- els. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 4860-4869, 2024.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. toomptto-prompt
Prompt-to-p prom--romppp image editing with cross attention control. arXiv preprint (TXi::2208.01626, 2022.
Jonathan Ho and Tim Salimans. arXiv:2207. 12598, 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Hailong Hu and Jun Pang. Loss and likelihood based membership inference of diffusion models. In International Conference on Information Security, pp- 121-141. Springer, 2023.
Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.61 14,661111101 14,1110000 2013.
Fei Kong, Jinhao Duan, RuiPeng Ma, Hengtao Shen, Xiaofeng Zhu, Xiaoshuang Shi, and Kaidi Xu. An efficient membership inference attack for the diffusion model by proximal initialization. arXiv preprint ar 2r0vv22305. 18355, 2023.
Haokun Lin, Haoli Bai, Zhili Liu, Lu Hou, Muyi Sun, Linqi Song, Ying Wei, and Zhenan Sun. Mope-clip: Structured pruning for efficient vision-language models with module-wise pruning error metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 27370-27380, 2024a.
Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song. Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quan- tized llms. In The Tiihth-eighth Annual Conference on Neural Information Processing Systems, 2024b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Dision-ECCY 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.

---

Published as a conference paper at ICLR 2025
Zhe Ma, Xuhong Zhang, Qingming Li, Tianyu Du, Wenzhi Chen. Zonghui Wang. and Shouling Ji. Could it be generated? towards practical analysis of memorization text-to-image diffusion models. arXiv preprint arXiv:2405.05846 2024.
Tomoya Matsumoto, Takayuki Miura, and Naoto Yanai. Membership inference attacks against diffusion models. in 2023 ieee security and privacy workshops (spw). 2023.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6038-6047, 2023.
Ed Pizzi, Sreya Dutta Roy, Sugosh Nagavara Ravindra, Priya Goyal, and Matthijs Douze. self- supervised descriptor for image copy detection. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 14532-14542, 2022.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 2rXXi:2204.061255 1(2):3, 2022.
Jie Ren, Yaxin Li, Shenglai Zen, Han Xu, Lingjuan Lyu, Yue Xing, and Jiliang Tang. Unveiling and mitigating memorization in text-to-image diffusion models through cross attention. arXiv preprint XwX.::2243.3105222 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical image computing and assister-assisted intervention- MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceed- ings, part III 18, pp. 234-241, Springer, 2015.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to- -omag diffusion models for subject-driven generation. In CVPR, pp. 22500-22510, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- fion processing systems, 35:36479-36494, 2022.
Christoph
Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02111 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b; An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278- 22222227222994 2022.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. 2017 IEEE symposium on security and privacy (SP), pp. 3-. IEEE, 2017.
Jascha Sohl-Dickse Dickstein. Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing, pp. 2256-2265. PMLR, 2015.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art or digital forgery? investigating data replication in diffusion models. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 6048-6058. 2023a.
12

---

Published as a conference paper at ICLR 2025
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Under- standing and mitigating copying in diffusion models. Advances in Neural Information Processing Systems, 36:47783-47803 2023b.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500, 2020.
Ryan Webster. A reproducible extraction of training images from diffusion models. arXiv preprint AxXiv:3305008944 2023.
Yuxin Wen, Yuchen Liu, Chen Chen, and Lingjuan Lyu. Detecting, explaining, and mitigating memorization in diffusion models. In The Twelfth International Conference Learning Repre- sentations, 2024.
Xiaoyu Wu. Yang Hua, Chumeng Liang, Jiaru Zhang, Hao Wang. Tao Song, and Haibing Guan. Cgi-dm: Digital copyright authentication for diffusion models contrasting gradient inversion. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10812- 10821, IEEE Computer Society, 2024a.
Yichen Wu, Long-Kai Huang, Renzhen Wang, Deyu Meng, and Ying Wei. Meta continual learn- ing revisited: Implicitly enhancing online hessian approximation via variance reduction. In The Twelfth International Conference on Learning Representations, 2024b.
Jing Xiong, Gongye Liu, Lun Huang, Chengyue Wu, Taiqiang Wu, Yao Mu, Yuan Yao, Hui Shen, Zhongwei Wan, Jinfa Huang, al. Autoregressive models in vision: A survey. arXiv preprint arXiv:2411.05902, 2024.
Lvmin Zhang. Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings the pp. 3836-3847, 2023a.
the IEEF/CVE International Conference on Computer Vision,
Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. Plug- and-play: An efficient post-training pruning method for large language models. In The Twelfth International Conference Learning Representations, 2024.
Yuxin Zhang, Weiming Dong, Fan Tang, Nisha Huang, Haibin Huang, Chongyang Ma. Tong-Yee Lee. Oliver Deussen, and Changsheng Xu. Prospect: Expanded conditioning for the personaliza- tion of attribute-waare image generation. arXiv preprint arXiv:2305. 16225, 2023b.
Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma, Weiming Dong, and Chang- sheng Xu. Inversion-based style transfer with diffusion models. In CVPR, pp. 10146-10156. 2023c.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A. ADDTIONAL EXPERIMENTAL RESULTS ON SDV1.4. 1.4.
In this section, we provide additional experimental results on SDv1.4.
Mor (Tmin)) Noteeman (Tnin)
Men (Triin) / Nor-mem (Non-iziin)
Mem (Ckn)) Net-maan (Gm)
2.1
1.13
Taxe Riicive Rete
Tow Poodiov Jane
8.0
BOOM
DD3
kyeas
Imagetringg
DLth
0.0
0.9
300
8:50
A75
0.59
025
1.00
Talse Posiiive Rate
False Positive Rate
False Positive Kate
Figure AI: ROC curves of different methods for detecting memorized images from non-memorized training images, non-training images, and non-memorized generated images on SDv1.4.
Mom (Tnin) Non-mam (Tnin)
Mon (Tnint) Non-men (Non-train)
Mer (Cim)) Non-mam (Gen)
1.30
1000
9.75
True Poeilive Raie
Troa Pretive
True Poelive
0.54
BBM
1:3
FBA bwifl
bwhil
Swhft
4.0
6:30
6435
6.30
Prsttive
Falio
Faluo Proitive Rate
Figure A2: ROC curves of different methods on SDv1.4 under data augmentation attacks for detect- ing memorized images across different datasets as Fig. AI
Results ROC Curves. Fig. A and Fig. A2 present the ROC curves across different datasets and settings on SDv1.4. As shown in Fig. A we can observe that our method achieves near-perfect dis- crimination in detecting memorized training images from non memomizede training and non- non-training images, with the similarity metric demonstrating excellent effectiveness. For detecting memoriza- tion in generated images, the MTCNP metric cxhibits relatively strong performance. Specifically, the similarity scores more effective for identifying memorized training images, while MTCNP proves better for detecting memorization in generated images, which aligns with our findings in the paper. Furthermore, as shown in Fig. A2, our similarity-based metric demonstrates strong robustness against data augmentation attacks, even though the performance of MTCNP degrades considerably. While after data augmentation attacks on generated images, our method still achieves the best per- formance, with MTCNP maintaining superior effectiveness. The complementary strengths of these two metries contribute to the overall robustness and effectiveness of our approach across various settings and data augmentation attacks.
Results of Ly similarity distance. We further evaluate the effectiveness of various reconstruction- based methods using the L2 distance, with results presented in Tab. AT and Tab. A2 Our observa- tions indicate that, except for image inpainting, all other methods exhibit a decline in performance when compared to I,, scores. Notably, our method demonstrates significant drop in TPR @ FPR
14

---

Published as a conference paper at ICLR 2025
values. This decline may be attributed to the more pronounced modifications to the image content introduced by the image inpainting method, whereas our approach primarily relies on subtle texture variations, rendering the L1 distance more effective metric. Despite the slight performance duction observed under the I distance, our method continues to outperform others and maintains state-of-the-ar performance.
Mem (Train) Non-mem (Train)
Mem (Train) / Non-mem (Non-train)
Mem (Gen) / Non-mem (Gen)
Method
AUC
AUC-PR
TPR@ I%FPR
AUC
AUC-PR
T ISFPR
AUC
Acc.
@ I&FPR
DDIM Inversion
0.847
0.303
0.046
0.865
0.345
0.046
0.690
0.648
0.028
Inpaint
0.901
0.495
0.092
0.891
0.494
0.092
0.924
0.885
0.644
Image-to-image
0.114
0.000
0.103
0.000
0.928
0.890
0.715
IP Siza
0.996
0.959
0.822
0.998
0,984
0.914
0.960
0.919
0.815
Table A1: Quantitative results using L2 distance to evaluate similarity for reconstruction-bant
methods on SDv1.4. The best and second-best results are highlighted in bold and underline.
Mem (Train) / Non-mem (Train)
Mem (Train) Non-mem (Non-train)
Mem (Gen) / Non-mem (Gen)
Method
AUC
AUC-PR
TPR@ ISFPR
AUC
AUC:PR
TPR@ISFF
AUC
Acc.
TPR I%FPR
DDIM Inversion
0.298
0.053
0.346
0.059
0.788
0.723
0.088
Inpaint
0,718
0.176
0.007
0.685
0.160
0.013
0.630
0.603
0,042
Emage-to-inann
0.539
0.109
0.000
0.512
0.103
0.000
0.616
0.588
0.041
IIP sm
0.994
0,945
0.776
0.995
0.945
0.855
0.867
0.794
0.341
Table A2: Quantitative results using L2 distance to evaluate similarity under data augmentation at-
tacks for reconstruction-base methods on SDv1.4. Key results are highlighted according Tab, A.
Addtional qualitative comparisons across different datasets. We present additional qualitative results on SDv1.4 Fig. A3 where significant similarity inconsistencies are highlighted within red boxes. The results indicate that our method exhibits more pronounced distinction in similarity inconsistencies between memorized and non- non-meorrized images.
(Train)
Men (Gen)
(Tzain)
(Non-train)
(Con)
Inarge Leago wopliniainttttoooommm 5OXM Inveraion
Ox1 Inages
Figure A3: Addtional qualitative comparisons with baselines across different datasets on SDv1.4.
Our method consistently achieves the best distinguishable results for memorization detection.
15

---

Published as a conference paper at ICLR 2025
A.2 QUANTITATIVE RESULTS ON OTHER VERSIONS OF STABLE DIFFUSION.
Datasets. To further assess the effectiveness of our method across different text-to-image diffusion models, particularly in detecting memorized training images, we conduct qualitative experiments on the training and non-training datasets for SDv1.5 and SDv2.1. For memorized training images, we follow Webster (2023) and collect 190 and 78 memorized images for SDv1.5 and SDv2.1, respec- tively. We obtain training images that have an SSCD similarity score (Pizzi et al.
2022) of generated results exceeding 0.7, strongly indicating duplication (Somepalli al. 2023a b). Furthermore, since the extracted training and non-member sets from LAION-mi (Dubiński al. 2024) for SDv1.4 also apply to SDv1.3 and SDv2.1 we select 1200 non- oonmmeoriized training images from the member sets of LAION-mi with SSCD similarity score below 0.15 of the memorized images, along with 1200 non-member images from LAION-mi as non-training images for these two Stable Diffusion models.
Quantitative results and performance under data augmentation attacks. We present the sults for detecting memorized training images from non-memorized training and non-training im- ages in Tab/A3 and A4b evaluated using our similarity metric. As shown, our method achieves AUC-PR values exceeding 0.96 and TPR @ 1%FPR above 0.87 across both models and settings, demonstrating our superior performance across different stable diffusion models. Furthermore, we analyze the performance under data augmentation attacks for SDv1.5, with results shown in Tab While baseline methods show significant accuracy degradation after attacks, our method maintains near-consistent performance, highlighting its exceptional robustness.
Mem(Train) Non-mem(Train)
Mem(Train) Non-menooooonnnaainn N
Method
AUC
AUC-PR
TPR@ 1%FPR
AUC
AUC-PR
@PR@ I 1FPR
DDIM Inversion
0.876
0.689
0.474
0.882
0.697
0.505
Inpaint
0.572
0.145
0.005
0.515
0.131
0.000
Image-to-image
0.786
0.271
0.016
0.858
0.354
0.021
SecMI
0.508
0.141
0.011
0.546
0.160
0,016
PIA
0.594
0.176
0.021
0.639
0.212
0.042
IIP sim 0.986
0.967
0.874 0.784
0.988
0.973
0.895
IIP
micnp 0.967
0.924
0.970
0.934
0.826
Table A3: Quantitative results for different methods of detecting training memorized images across different datasets on SDv1.5.
Mem(Train) Non-mem(Train)
Mem(Train)/ (oanTTrriinn/oonmmmmCoooooottnnn
Method
AUC
AUC-PR
TPR@ 1%FPR
AUC
AUC-PR
@ I 1%FPR
DDIM Inversion
0.883
0.307
0.064
0.892
0.320
0.128
Inpaint
0.651
0.107
0.000
0.612
0.099
0.000
Image-to-image
0.713
0.099
0.000
0.784
0.134
0.000
SecMI
0.552
0.078
0.000
0.533
0.074
0.000
PIA
0.538
0.117
0.077
0.564
0.126
0.077
IIP sim 0.998
0.979
0.974 0.321
0.998
0.985
0.987
IIP
micnp 0.825
0.390
0.852
0.486
0.346
Table A4: Quantitative results for different methods of detecting training memorized images across different datasets on S.v2.1.
16

---

Published as a conference paper at ICLR 2025
Mem (on-mem((Triin)
Mem(Train) (ooiri/n/
Method
AUC
AUC-PR
TPR@ 1%FPR
AUC
AUC-PR
@PR@ 1%FPR
DDIM Inversion
0.745
0.282
0.032
0.756
0.300
0.032
Inpaint
0.811
0.410
0.053
0.793
0.410
0.053
Image-to-image
0.532
0.136
0.016
0.586
0.155
0.016
SecMI
0.520
0.133
0.000
0.534
0.138
0,000
PIA
0.567
0.152
0.000
0.582
0.161
0,005
IIP sim 0.983
0.957
0.853
0.986
0.969
0.884
IIP mtenp
0.919
0.781
0.532
0.928
0.812
0.611
Table AS: Quantitative performance of different methods under data augmentation attacks across different datasets on SDv1.5.
A.3 EXPERIMENTS IN REAL-WORD WORLD SCENARIOS.
In practical situations, memorization varies in different degrees beyond exact duplication of the training images (Carlini et al.) 2023; Somepalli et al.) 2023a b). For example, partial replica- tion (Somepalli et al. 2023a), where generated images incorporate specific elements fragments of existing images, is a common phenomenon in real-world scenarios and presents substantial privacy and copyright risks. To evaluate our performance in real-world scenarios, expand the experiments to include a broader range of memorization situations. Specifically, we apply SSCD similarity score thresholds of 0.4 and 0.5, which indicate high pixel-Ievel replication Somepalli et al.) (2023b). These thresholds enabled the identification of a spectrum of memorization cases, from full duplication partial replication. The numbers of memorized images extracted under different thresholds for different models are shown in Tab. A6
Number of Memorized Image
Model
Filter
743
$Dv1.4
SSCD) 0.5
1626
SDv1.4
SSCD 0.4
601
SDv2.1
SSCD > 0.5
1676
SDv2.1
SSCD 0.4
Table A6: The numbers of memorized images for different models and SSCD thresholds.
Mem(Train) Now--Nooo--maaaaa
Mem(Train) Non-memooonottrrrnn
Method
AUC
AUC-PR
TPR@ 1%PR
AUC
AUC-PR
TPR@ 1FPR
DDIM Inversion
0.839
0.776
0.292
0.836
0.776
0.312
Inpaint
0.604
0.418
0.008
0.552
0.391
0.005
Image-to-iman image
0.743
0.522
0.005
0.813
0.611
0.005
SecMI
0.510
0.415
0.019
0.516
0.421
0.016
PIA
0.543
0.424
0.020
0.582
0.473
0.034
IIP sim
0.984
0.978
0.774
0.988
0.984
0.812
IIP mnten
0.904
0.884
0.416
0.911
0.896
0.450
Table A7: Comparison of different methods on SDv1.4 with SSCD score(>>0.5) filtered memorized
images.
Experimental results are presented in Tables A8
A9 and A10 corresponding to different SSCD thresholds across different stable diffusion models. In all experimental settings, the proposed IIP method consistently outperforms the baseline approaches, demonstrating superior performance. Specifically, both AUC and AUC-PR values exceed 0.9 across all configurations, highlighting the
A7
17

---

Published as a conference paper at ICLR 2025
Mem(Train) Nonammm(((rran)))
Mem(Train) Non-mmemomooonttrann
Method
AUC
AUC-PR
@PR@ 1%FPR
AUC
AUC-PR
TPR@ 1%FPR
DDIM Inversion
0.805
0.792
0.179
0.802
0.792
0.202
Inpaint
0.596
0.538
0.010
0.541
0.509
0.009
Image-to-image
0.740
0.633
0.003
0.809
0.710
0,003
SecMI
0.510
0.591
0.005
0.537
0.629
0.010
PIA
0.538
0.603
0.010
0.577
0.648
0.017
IIP sim
0.981
0.985
0.689
0.987
0.990
0.750
IIP mtenp
0.850
0.895
0.260
0.860
0.905
0.287
Table A8: Comparison of different methods on SDv1.4 with SSCD score(>>0.4) filtered memorized images.
Mem(Train) Non-
Mem(Train) (ain(Train)/Nonnece NNo--memmmooo-trriin
Method
AUC
AUC-PR
TPR @ 1%FPR
AUC
AUC-PR
TPR@ 1%FPR
DDIM Inversion
0.768
0.599
0.077
0.779
0.611
0.101
Inpaint
0.599
0.402
0.012
0.555
0.377
0,012
Image-to-image
0.731
0.474
0.008
0.800
0.562
0.013
SecMI
0.582
0.440
0.045
0.559
0.415
0.020
PIA
0.505
0.384
0.042
0.530
0.414
0.045
IIP sim
0.955
0.935
0.624
0.956
0.943
0.737
IIP micnp
0.742
0.665
0.191
0.764
0.704
0.276
Table A9: Comparison of different methods on SDv2.1 with SSCD score(>:055) filtered memorized
images.
Mem(Train) (Tommi)
Mem(Train)/Non-no NNon-memmNoon-traiin
Method
AUC
AUC-PR
TPR@ 1%FPR
AUC
AUC-PR
TPR@ 1%FPR
DDIM Inversion
0.730
0.680
0.044
0.743
0.693
0.054
Inpaint
0.543
0.528
0.010
0.507
0.503
0.003
Image-to-image
0.747
0.657
0.009
0.813
0.732
0.015
SecMI
0.553
0.642
0.028
0.529
0.622
0.016
PIA
0.516
0.613
0.021
0.542
0.642
0.024
IIP sim
0.930
0.925
0.489
0.932
0.931
0.598
IIP micep 0.704
0.781
0.116
0.730
0.808
0.177
Table A10: Comparison of different methods on SDv2.1 with SSCD score(>>0.)) filtered memorized images.
effectiveness of method. These results further substantiate the performance of IIP in real-world scenarios where partial image replication may occur, making it particularly suitable for detecting broader memorization in text-to-image diffusion models.
A.4 COMPARISON WITH DETECTING MEMORIZED PROMPTS (WN ET AL. 2024).
We further clarify the difference between our inversion based inference perturbation (IIP) and mem- orization mitigation techniques (Wen et al.) 2024), which propose MTCNP as an effective metric detect memorized prompts. We make several new observations that extend this finding.
18

---

Published as a conference paper at ICLR 2025
For memorization detection, Wen et al. (2024) finds memorized prompts derive larger MTCNP. We extend the observation that even when perturbed prompts (random or meaningless) are added during the inference process, the phenomenon of larger MTCNP still persists. This suggests that the MTCNP not solely reliant on the original prompt, but also on some features related to the image itself, which we detailedly discussed Sec. 2.2
Besides, while Wen et al. (2024) minimizes the magnitude of text-conditional predictions to perturb the original prompt embedding the initial time step during generation. Our method first obtains the latent code by unconditional DDIM Inversion in latter timesteps and employs random and meaning- less prompt for inference, the prompt embedding optimized cooperates with the latent code obtained from the inversion. We specify our optimization differences as follows:
Different perturbing stages. We perturb the text embedding in the latter stage of inference (after timestep 30 the experiments), while Wen al. (2024) perturb the text embedding at the initial timestep.
The minimization is processed with different textual prompts and latent codes. Our method uses random and meaningless prompts, cooperating with the latent code obtained from DDIM Inversion. In contrast, [Wen et al. (2024) employs the original prompts and random gaussian latent code.
To further demonstrate the differences, we conducted experiments using the techniques in Wen et al. (2024: (a) perturbing text embedding in the initial timestep and (b) minimizing the magnitude of text-conditional predictions with original prompt and random gaussian latent code for the image- level memorization task.
Method
AUC
AUC-PR
TPR @ F%FP
(a) perturb embedding (Wen et al. 2024)
0.673
0.540
0.040
(b) minimize the MTCNP Wen et al. 2024)
0.665
0.580
0.067
sima
0.984
0.978
0.774
0.904
0.884
0,416
mtcnp
Table A11: Qualitative results of techniques in
(Wen et al. 2024 and IIP.
The experimental results demonstrate the different performance between our method and Wen et al. (2024). Our pipeline is proposed effectively solve image-level memorization detection tasks.
Furthermore, the evaluation metric similarity between the original and perturbed images mainly depends on the inference perturbation pipeline and the optimization objective alternative. MTCNP is one possible optimization objective in our current implementation, IIP can also employ objectives that do not rely on MTCNP. We employ the proposed similarity distance metric. Specifically, we replace the MTCNP- loss function with similarity-based follows:
t
Led tı
1
Ad
Σ t
12s - Eel + de |[Er(ca) - Ec(cnuu)ll.s
(11)
-
tp
where we exchang the MTCNP (160(24, t, ca c0(2 t, l1 in the original loss function 7 to similarity distance | - Zz/-
The experimental results are shown Tab. A12 Results demonstrate that both optimization ob- jectives perform comparably, indicating that the optimization objective is alternative and similarity computation is not heavily dependent on MTCNP.
A.5 LIMITATIONS AND DISCUSSIONS.
Limitations. While method demonstrates superior performance across various datasets, with different metrics performing optimally for different datasets, there are limitations under data aug- mentation attacks for generated datasets. Specifically, when faced with generated images with data
19

---

Published as a conference paper at ICLR 2025
Memorized
IPP Results of Different Optimization Objectives
Mem(Traing Non-mem((Tram)
Mem(Train) Non-mmemooon-trainn
Images
AUC
AUC-PR
TPR@1SFT
AUC
AUC-PR
TPR01XFPR
MICNP
0.984
0.978
0.774
0.988
SSCD 0.5
Sim. Distance
0.981
0,975
0.777
0.983
0.977
0,802
SDv14
SSCD 0.4
MTCNP
Sim. Distance
0.975
0.811
0.982
0.689
0.676
0.987
0.978
0.984
0.696
SSCD 0.5
MTCNP
0.955
0.935
0.624
0,956
0.943
0.737
Sim. Distance
0.956
0.940
0,700
0,959
0.946
0.730
SDv2.1
MTCNP
0.930
0.925
0.489
0.932
0.931
0,598
SSCD 0.4
Table A12: Quantitative performance of different optimization objectives for IIP
augmentation attacks, our method may not perform as effectively. The robustness in detecting mem- orization from these defense-cnhanced generated images remains area in need of further improve- ment. Additionally, although our work primarily focuses on image-level memorization detection, dememorization during generation represents an important aspect that could further mitigate the tention of user content in the model's memory. This issue has not been addressed in the current study, but intend to investigate it in future research.
Ethics and Broader Impacts. In recent years, the rise of generative models has increasingly high- lighted issues related copyright and data security. Memorization, duplication, poses significant threats to both model owners and users. Our method offers a straight- forward and effective solution for users and regulators to verify whether images have been mem- orized by the model, in the absence of original prompts. Notably, our method demonstrates near- perfect detection capabilities for memorized training images and exhibits high robustness against data augmentation attacks, providing effective tool for identifying data lcaks and copyright threats from the perspective of memorization.
form strong training data
20


Reviews:
[{"review_id": "SIJumB4U91", "reviewer": "Reviewer_8Z99", "summary": "This paper investigates the memorization limitation of the Stable Diffusion model to help protect the training data’s privacy. The paper proposes a new task: image-level memorization detection, which is different from existing works that detect memorization during inference. Then, based on two insights that memorized images under perturbed inference have a notable similarity discrepancy and a large magnitude of text-conditional noise prediction, the paper proposes IIP framework that uses unconditional DDIM inversion to derive latent noises for the images and conducting perturbations. The paper also construct a setup for this new task and demonstrated better performance than baselines.", "questions": "Please see the weaknesses regarding motivation and novelty.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This work tackles an important and practical problem: diffusion models can memorize their training data, potentially leading to privacy concerns. These issues are thoroughly discussed and effectively motivated in the paper.\n2. The paper is well-presented, clearly structured, and easy to follow. \n3. This paper further proposes a new task of image-level memorization detection and a correspondingly designed method for this new task.", "weaknesses": "1. The motivation for the proposed new image-level memorization detection task is based on a misunderstanding of the related work. Specifically, in lines 46-47, the paper claims that “However, these approaches rely heavily on access to the original prompts, which is often impractical.” Since all baselines and this paper experiment on the text-to-image Stable Diffusion model, using text prompts is practical. Also, the baseline methods do not need an organized prompt list that contains triggering prompts since they can obtain detection signals (such as TCNP in [1]) during the inference process. They can then detect the potential memorized images being generated within only one inference step and with good accuracy. Such lines of method are actually more practical than the proposed image-level memorization detection task, as the memorization is detected and halted from the source (way before the image is even generated). Thus, the proposed task is of limited practical significance. I would suggest the authors consider investigating unconditional diffusion models (where there is no text prompt) and see if the proposed method works.\n2. The finding that large TCNP correlates with memorization is not novel. I understand that the paper differs in performing DDIM inversion to the clean image, however, it is actually the identical finding of [1] that a noise can present large TCNP even after the first step of DDIM denoising.\n\n[1] Detecting, Explaining, and Mitigating Memorization in Diffusion Models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ao4COXbliP", "reviewer": "Reviewer_JFcJ", "summary": "The paper presents a new method for identifying whether an image was part of the diffusion model training set, without access to the prompt used for generation. As far as the authors know, they are the first to identify this task. To this end - inference under prompt perturbations is examined for memorized / not memorized samples, with the aid of DDIM's inversion capabilities (specifically unconditional inversion to avoid dependency on the prompt). Under this process, they find two key properties that differentiate between memorized and non-memorized samples. Building on their insights, they propose a Inversion-based Inference Perturbation (IIP) -  a novel method for the task at hand and out-perform the competitors on an extensive test suite.", "questions": "Please refer to Weaknesses above.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The results present SOTA performance with respect to the defined task and the evaluated existing methods.\n\n2. The experiments are extensive: Evaluate both success, sensitivity, and soundness.\n\n3. The writing is good and articulate.\n\n4. Novelty of the method: The authors propose for the first time to experiment with inference with perturbed prompts for the task at hand.", "weaknesses": "1. Novelty of the task: Please have a look at [1], [2] - these work engage in membership inference without using prompts, as far as I know. Could the authors clarify how their proposed task of image-level memorization detection differs from membership inference, particularly in light of the cited works that perform membership inference without prompts?\n\n[1] Matsumoto, T., Miura, T., & Yanai, N. (2023, May). Membership inference attacks against diffusion models. In 2023 IEEE Security and Privacy Workshops (SPW) (pp. 77-83). IEEE.\n‏\n\n[2] Hu, H., & Pang, J. (2023, November). Loss and Likelihood Based Membership Inference of Diffusion Models. In International Conference on Information Security (pp. 121-141). Cham: Springer Nature Switzerland.\n\n2. Comparison to other methods: Could the authors add comparisons to relevant methods that do not require prompt access (e.g. the methods mentioned above)?\n\n3. Additional experiments with a different stable diffusion model would substantiate the proposed method's advantage.\n\n4. Requiring inversion of the model is quite restrictive, considering how quickly generative technology changes, and how rare it is for generative models to enable such inversions. Can the authors think of ways adapt their findings to non-invertible scenarios?\n\n5. Minor Weaknesses: \na. In all related figures - change \"ori\" to the full word \"original\".\nb. The expression \" images exhibit greater similarity discrepancy \" in the introduction is rather confusing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CUwPaqcS9y", "reviewer": "Reviewer_dGYC", "summary": "The authors propose an algorithm named IIP to detect memorized images generated by LDMs. The algorithm features two observations during DDIM generation with perturbed prompts (1) a notable similarity discrepancy (2) a large magnitude of text-conditional noise prediction.", "questions": "Please see the weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) The observed phenomenon seems to be universal and statistically notable.\n\n(2) Within the experiment setup of this paper, the algorithm shows promising performance.\n\n(3) The presented figures are illustrative and depictive.", "weaknesses": "**Major**\n\n(1) The biggest concern I have about this paper is its high overlap with existing work [1], making its technical and empirical contribution weak. I shall support my claim with the following facts: (a) The metric TCNP/MTCNP is proposed by [1], which also applies the metric to memorization detection. (b) Perturbing the textual embedding during DDIM generation is already used by [1] to mitigate memorization. (c) minimizing the magnitude of text-conditional prediction is already used by [1] to achieve the perturbed prompt embedding. It wasn't until I carefully re-read [1] that I noticed so much overlap between the proposed algorithm and an existing work. I believe the authors fail to give enough credit to the existing work, as all this necessary information is not shown in the submitted manuscript.\n\n(2) The authors tend to use big words like \"SOTA\" and \"pioneer\". However, I am not fully convinced by the importance of the \"image-level detection\" setting proposed by the authors. If we can detect memorization as early as in the generation phase like [1], why would we bother doing the \"image-level detection\"?\n\n[1] IMAGE-LEVEL MEMORIZATION DETECTION VIA INVERSION-BASED INFERENCE PERTURBATION; https://openreview.net/pdf?id=84n3UwkH7b\n\n**Minor**\n\n(1) Only SD v1.4 is considered, which is not enough because the performance of the proposed algorithm might be highly affected by the architecture, the training data, and the sampling configuration of the LDMs.\n\n(2) Missing period in line 181 \"during the subsequent generation process [period here] Interestingly...\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tjvey3IyKP", "reviewer": "Reviewer_8dpK", "summary": "The paper introduces a method for image-level memorization detection. It is motivated by an analysis showing two key characteristics of memorized images under perturbed inference: similarity discrepancy and a large magnitude of text-conditional noise prediction. The key idea of the method seems to be using an unconditional DDIM inversion to derive latent codes and optimizing non-memorized prompt embeddings for effective perturbation. The method is evaluated on a number of datasets, showing strong ability to detect memorized images.", "questions": "* To what extent are the perturbed prompts related to the original prompts? Can we really say we do not use the original prompts in any way? E.g. if we use something derived from them\n* How expensive is it to run the method for an image to test if it is memorized?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* The results are strong and the method seems reliable in detecting memorized images\n* There is an analysis section to motivate the approach and explain why it may work\n* The paper is generally well-written and easy to read", "weaknesses": "* The prompts appear to be an input to the optimization procedure that generates the optimized prompt embedding, so even if the prompts are not directly paired with the images, we may not be able to say the method does not rely on prompts (as may seem from some parts of the paper). This can be discussed more to explain to what extent we do not need access to the original prompts.\n* All settings are based on Stable Diffusion v1.4: while this is a common model for image generation, one would like to see if the same method works across different models. There are currently various image generation models available, so it would be good to try the method on more - e.g. two more - to see if this method generalizes more broadly.\n\nMinor:\n* Ori prompt would be easier to understand if it was written as original prompt or orig. prompt", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)