No Caption, No Problem: Caption-Free Membership Inference via Model-Fitted Embeddings
Authors: Joonsung Jeon
Woo Jae Kim
Suhyeon Ha
Sooel Son
Sung-eui Yoon
Venue: ICLR 2026 Poster (2026)
Keywords: Membership Inference
Data Privacy in Generative Models
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Latent diffusion models have achieved remarkable success in high-fidelity text-to-image generation, but their tendency to memorize training data raises critical privacy and intellectual property concerns. Membership inference attacks (MIAs) provide a principled way to audit such memorization by determining whether a given sample was included in training. However, existing approaches assume access to ground-truth captions. This assumption fails in realistic scenarios where only images are available and their textual annotations remain undisclosed, rendering prior methods ineffective when substituted with vision-language model (VLM) captions. In this work, we propose MoFit , a caption-free MIA framework that constructs synthetic conditioning inputs that are explicitly overfitted to the target model's generative manifold. Given a query image, MoFit proceeds in two stages: (i) model-fitted surrogate optimization, where a perturbation applied to the image is optimized to construct a surrogate in regions of the model’s unconditional prior learned from member samples, and (ii) surrogate-driven embedding extraction, where a model-fitted embedding is derived from the surrogate and then used as a mismatched condition for the query image. This embedding amplifies conditional loss responses for member samples while leaving hold-outs relatively less affected, thereby enhancing separability in the absence of ground-truth captions. Our comprehensive experiments across multiple datasets and diffusion models demonstrate that MoFit consistently outperforms prior VLM-conditioned baselines and achieves performance competitive with caption-dependent methods.

Published as a conference paper at ICLR 2026
No CAPTION, No PROBLEM: CAPTION-FREE MEM- BERSHIP INFERENCE VIA MODEL--ITTEL EMBED- DINGS
Joonsung Jeon, Woo Jae Kim, Suhyeon Ha, Sooel Son" & Sung-Eui Yoon"
Korea Advanced Institute of Science and Technology (KAIST) (mikeraph, wkim97, suhyeon.. ha, si. son,
ac.kr
ABSTRACT
Latent diffusion models have achieved remarkable success high-fidelity text- to-image generation, but their tendency to memorize training data raises critical privacy and intellectual property concerns. Membership inference attacks (MIAs) provide principled way to audit such memorization by determining whether a given sample was included in training. However, existing approaches assume ac- cess to ground-truth captions. This assumption fails in realistic scenarios where only images are available and their textual annotations remain undisclosed, ren- dering prior methods ineffective when substituted with vision-language model (VLM) captions. In this work, we propose MOFIT, caption-free MIA frame- work that constructs synthetic conditioning inputs that explicitly overfitted the target model's s generative manifold. Given query image, MOFIT proceeds in two stages: (i) Model- surrogate optimization, where perturbation applied to the image is optimized to construct surrogate in regions of the model's 's un- conditional prior learned from member samples, and (ii) surrogate-driven embed- ding extraction, where a model-fitted embedding is derived from the surrogate and then used as a mismatched condition for the query image. This embedding ampli- fies conditional loss responses for member samples while leaving hold-outs rela- tively less affected, thereby enhancing separability in the absence of ground-truth captions. Our comprehensive experiments across multiple datasets and diffusion models demonstrate that MOFIT consistently outperforms prior VLM-conditiooed baselines and achieves performance competitive with caption-dependent methods. The code is available at hhtps//gihhubccom///onnnggggee
INTRODUCTION
Latent diffusion models (LDMs) (Rombach et al. 2022) have advanced image-generation capabil- ities of generative models and broadened their applications various tasks, such as photorealistic facial synthesis (Ergasti et al. 2024), medical CT image generation (Molino et al.) (225). and pro- tein structure generation (Fu et al.] 2024). However, there exist growing concerns and evidence that diffusion models can memorize and reproduce high-fidelity training images, posing serious threats Webster 2023] Carlini et al. 2023).
to training data privacy (Somepalli et al. 2023]
Membership inference attacks (MIA) have emerged as a standard empirical approach to assess the risk training-data exposure machine learning models (Shokri et al.) 2017). MIAs are designed to decide whether a given query sample is used in training the target model, providing concrete metrics for auditing memorization and detecting privacy leakage. Recent studies have adapted MIAs to LDMs. exploiting signals such differences in conditional training loss, reconstruction error, or denoising consistency to distinguish member samples from non-members (Carlini et al. 2023
Matsumoto et al. 2023 Duan et al. 2023 Fu et
2023 Zhai et al. 2024).
Existing MIA studies on text-to-image LDMs assume access to image- image-caption pairs; the ground- truth caption for query image is available for inferring its membership. We contend that this assumption is often impractical for auditors. For example, an artist who suspects a generated image
"Co-corresponding authors

---

Published as a conference paper at ICLR 2026
replicates their work typically lacks access to the training captions used by released target model. Moreover, training-set set provenance is frequently undisclosed on public generative- platforms
In this paper, we demonstrate that performing effective MIAs in the caption-free setting is challeng- ing: replacing ground -ruth captions with VLM-generated approximations substantially degrades the performance of ofte-of-the-ari MIA approaches based on CLiD (Zhai et al., 2024) (Sec. 3.3)
To address this challenge, we present our novel finding on a systematic difference in how member and non-member (i.e., hold-out) samples respond to mismatched conditioning in their denoising process. Member samples whose captions were used during training exhibit high sensitivity in their conditional denoising loss under alternatiee or misaligned conditions; hold-out images are relatively less affected (Sec. 3.3). This difference in sensitivity provides an important signal to establish and boost separability between member and non-member groups in the caption-free free setting.
Motivated by this observation, we introduce MOFIT, a framework that constructs Model-Fitted embeddings tailored the generative manifold of the target model. Given query image, MOFIT (1) synthesizes surrogate input that aligns closely with the model's ''s unconditional prior, and (2) extracts an embedding from this surrogate, forming tightly coupled pair within the model's conditioning space. At inference time, conditioning the original query with this embedding leads to pronounced increase in conditional loss for member samples, while hold-out samples exhibit relatively minimal changes thereby enhancing separability in the absence of ground-truth captions.
We evaluate the effectiveness of MOFIT as an alternative to VLM-generated captions in the caption- free setting. Across three fine-tuned text-to-image diffusion models Pokemon, MS-COCO, and Flickr MoFIT consistently outperforms prior methods that rely on VLM-generated captions, achieving up to +25% ASR and +30-47% TPR @ 1%FP improvements. Notably, MS-COCO, MOFIT even surpasses prior methods with access to ground-truth captions, highlighting its strong discriminative power without textual supervision.
In summary, our contributions are as follows:
We introduce the first MIA framework tailored for performing effective membership infer- ence against LDMs in the caption-free setting, reflecting a practical adversary who lacks access to ground-truth captions.
We present a novel empirical insight: during the denoising process, member samples ex- hibit larger changes in conditional loss under alternative conditioning than hold-out sam- ples, providing an exploitable feature for separating members from non-members.
Building on this observation, we propose two-stage MIA: (1) synthesize caption embed- dings explicitly optimized to overfit the target LDM, and (2) exploit those embeddings to condition the original query, thereby exploiting members' selective sensitivity and boosting loss-based separation.
MOFIT outperforms prior methods conditioned on VLM-generated captions and achieves competitive performance even against ofate-of-the-ar MIAs using ground-truth captions.
2 RELATED STUDIES: MEMBERSHIP INFERENCE
membership inference attack (MIA) is a privacy attack in which an adversary seeks to determine whether specific data sample is included in the training dataset of a target model. While early studies targeted deep neural networks (Shokri et al. (017). recent research has extended MIAS generative models particularly diffusion models- due their strong generation fidelity and potential for memorization (Carlini et al. 2023).
Carlini al. 2023) first examined MIA in unconditional diffusion settings by leveraging multiple shadow models to statistically infer membership. Matsumoto et al. (2023) extended this by directly exploiting the training loss values specific timesteps. SecMI (Duan al. (023) improved tack performance by estimating posterior errors across diffusion trajectories, while PIA (Kong et al.)
2023) reduced the query cost by approximating ground- truuh trajectories from single intermediate latent. PFAMI (Fu et al.) 2023) introduced probabilistic fluctuation-basel metric to capture differ- ences generation behavior between member and non-member samples. Most recently, CLiD (Zhai
https://civitai.ccm
2

---

Published as a conference paper at ICLR 2026
achieved state-of- ahr performance multiple datasets by targeting membership inference in text-conditioned LDMs through discrepancies between conditional and unconditional denoising losses.
et al. 2024)
We emphasize that all prior MIA research on diffusion models has a common threat model which the adversary already has access the exact text captions corresponding query images. How- ever, such ground-truth captions are often inaccessible in practice. For instance, when testing the membership status of facial images of specific individuals, is unrealistic assume that the adver- sary already has the corresponding ground-truth captions to test their membership. In this paper, we adopt a more realistic threat model: a caption- free setting in which only the query image is available.
PRELIMINARY
3.1 LATENT DIFFUSION MODELS
The goal of latent diffusion models is learn parameterized reverse process that approximates a given data distribution. In the forward process, Gaussian noise e ~ (N T) is incrementally added the latent 2g across timesteps 1, ...... T, yielding sequence of progressively noisier latents. Each noisy latent computed as 20 vI cumulative noise schedule.
dùé where a; = II: as denotes the
The reserve process is learned by denoising model 60. typically U-Net, trained to predict the added noise at each timestep. For text-conditioned generation, the model is trained on image-caption pairs (I, e) by minimizing a conditional noise prediction objective:
Loudd = (2o.t.c-N((.n)) [||e es (zz, t, c)
(1)
where 0 denotes the denoising model parameters. enable scalable guidance during inference with- out requiring external classifier, the model is usually trained using classifier-free guidance (Ho
Salimans| 20222). Specifically, the condition e randomly replaced with null token embedding @null during training, allowing the model to learn both conditional and unconditional denoising:
Luncond Ezg,t.e~~N(0. [[lc co(2,, t, omm1))
At inference, the model iteratively denoises the latent under the conditioning input over time steps, and the final denoised latent decoded into an output image.
3.2 PROBLEM STATEMENT
We assume target LDM trained a dataset D of image-caption pairs (Ir, c), partitioned into two disjoint subsets: the member set DM and the hold-out (non-member) set DH- The adversary's ob- jective determine whether a query image x D is included in the training set DM- Specifically, the target denoising model Eg is trained on DM. Following prior work (Dubiński et al. (024)) D drawn from the same distribution as Das ensuring realistic and challenging evaluation setting for membership inference.
Unlike prior research (Kong et al. 2023 Fu al. 2023), we consider more practical and challeng- ing setting in which the adversary has access only to the query image but not to its ground-truth caption c. This assumption reflects real-world deployment scenarios where training annotations are often inaccessible. To address the absence of the ground-truth caption, the attacker is allowed to use an alternative condition (e.g., generated or inferred caption) place c.
Formally, define the membership inference attack of query image v as binary function:
M(x, {b.
[1, if DM 0 if E DH
(3)
3.3 OBSERVATIONS
To better understand the effect of missing ground-trus truth captions, we empirically evaluate member- ship inference when the captions replaced with externally generated alternatives (e.g., VLM
3

---

Published as a conference paper at ICLR 2026
(a) GT captions
(b) VLM captions
(C) MoFit
(d) Member
200
Nember
100
Hold out
Hold out
80
Hold cut
Deralty
150
Deraity
150
Denaity
60
E,05
0.35
100
100
40
Hold-out
50
20
0
-0.02
-0.01
0.00
0.01
-0.02
0.011
0.00
0.01
0.00
0.01 0.02 0.03 0.04 Fcond Cancod
Loard Luncond
Loard
Luncand
Figure 1: Distribution of membership scores under different condition types: (a) ground-truth cap- tions, (b) VI.M-generated captions, and (c) our model-fitted embeddings. In (d), Loand values of member samples increase under condition substitution to VIM, whereas hold-out samples remain relatively stable in (e). Dotted lines denote Luncond, and all distributions are estimated using Gaussian kernel density estimation.
outputs). Motivated by CLiD ZZai et al. 2024). we contrast the distributional distinctness of the membership scores for Das and Drr under two attack scenarios: (i) ground-truth captions available, and (ii) ground-truth captions replaced by alternative captions.
Setup. CLiD ZZai et al. 2024) scores membership of a query image by the difference between conditional and unconditional noise-prediction losses:
LCLD = Loond Lumcomd = Er,, [|| c0 t, c)]l -r,c [[|c [llc-co(ae i, Coan) Gman) ||2]
(4)
where is the caption paired with image during training and Caull denotes the unconditional setting (i.e., no-text condition). To establish a caption-free setting where ground-truth captions are unavail- able, we replace the caption with an externally generated description using CLIP- ILIP-Interrogatof
a vision-language model (VLM), and substitute them into CLiD (Zhai et al. 2024) framework as conditioning input. For the target diffusion model, we adopt 3D-Pokemon I Stable Diffusion v1-4 model 9 tune-tneed on the Pokémon dataset (LambdaLabs, 2022). To ensure a fair comparison, we (0, when calculating Eq. and follow all other settings from the
reuse the same noise original CLiD framework.
Observations. We find clear degradation in MIA performance when VLM-generated captions è are used instead of ground-truth captions c, despite their apparent semantic alignment with the images. Under ground-truth conditioning, CLiD yields clearly separable distribution patterns of Loond Luncond for members and hold-out samples (Fig. [a)). However, conditioning on VLM outputs substantially reduces this separation and incurs largely overlapping score distributions (Fig. [b)). Quantitative evaluation results for this degradation are presented in Sec.
S.c. 5.2
We attribute the performance degradation to difference in sensitivity to conditioning be- tween members and hold-out samples. As Fig. d)) shows, member samples incur large increase in Lcond when conditioning replaced by VLM-generated captions; by contrast, hold- out samples (Fig. [T)) exhibit only a modest increase. Mcanwhile, Luncoud stays approxi- mately unchanged for both groups This asym- metric sensitivity produces systematic up- ward shift in the membership score (i.e., Leved Luncona) for members. Appendix A.3 provides results on additional datasets.
Metric
(d) Member
(e) Hold-out
Mcan Std (GT)
0,0253 I 0.0091
0.0433 + 0:0125
Mcan # Std (VLM)
0.0300 + 0.0104
0.0432 + 0.0125
KS Test Statistic
0.2284
0.0264
Divergence
0.7126
0.2796
Quantitative comparison of Leend distri- bution under ground-truth vs. VL.M captions for (d) member and (c) hold-out samples in Fig. [ Arrows indicate the direction of larger deviation.
Table 1:
The sensitivity of member and hold-out samples is further supported by the results in Tab. 1 where the mean absolute difference, Kolmogorov-Smimoo (KS) statistic, and Kullback-Leibler (KI.) di- vergence all indicate greater sensitivity of member samples compared to hold-out samples in re- sponse to changes in conditioning. Such behavior is intuitive: member samples, having been ex- plicitly exposed to ground-truth captions during training, exhibit increased sensitivity of Loond to

---

Published as a conference paper at ICLR 2026
(a) Model-fitted Surrogate Optimization
(c) Membership Inference
null
Diflusion Model rg
Minimize
Forward ((O.))
Eg puut))
Xo
8'
(b) - Embedding Extraction
Xa
Xg
VLM
wottchn nodfing
Text Encoder
Diffusion Model Eg
t.
70
20
21
Diflusion Model
Eg (z
t,
Minimize
Mopii
Forward with same t,4
Member Hold-out
Figure 2: Overview of our proposed method. (a) Given query image To. we first optimize per- turbation overfit to the learned representation from the model. (b) From resulting surrogate image + 8*, we extract model-fitted embedding o*, which is then used synthetic condition to amplify the disparity between member and hold-out samples in (c).
condition changes. In contrast, hold-out samples, absent from the training distribution, demonstrate less iundition-sensitiin behavior. Based on these findings, we summarize the following two key observations:
1. Member samples are highly sensitive to conditioning: Loond increases consistently when using alternative captions.
2. Hold-out samples are relatively less affected by conditioning variations: Loond exhibit only minimal changes across different captions.
Intuition We propose exploit the observed sensitivity difference improve membership infer- ence in the caption- free setting. Specifically, we generate conditioning embeddings that significantly increase Leand for member samples while inducing only minimal changes for hold-out samples.
Additionally, we observe that Luncond tends to be lower for member samples than for hold-out sam- ples. This is expected, as member samples are directly involved in minimizing Luncond during training (Ho & Salimans, 2022). Consequently, embeddings from MOFIT amplify the difference Loond Luncond (Eq. 4) for members via elevated Loond and relatively low Luncond while pro- ducing less amplification for hold-out samples, which exhibit only modest increases in Loond and maintain relatively high Lunconds thereby reinstating a reliable separability signal (see Fig. [c)).
4 METHODOLOGY
We propose MOFIT, caption-free ceebeeshhp-inferene framework that leverages Model-Fitred embedding to selectively increase the conditional loss Loond for member samples.
In caption-free setting, alternative other than relying VLMs -is recover the paired caption of a query image Zo by directly optimizing an embedding for the clean 2o with respect to Lcond (Eq. []. However, since the membership status of zo is unknown, the optimization aligns embeddings for both member and hold-out samples. Therefore, at inference time, such embeddings produce uniformly low Leond for both members and non-members, eroding the discriminative signal (see Sec. 1.3 for details).
To address this challenge, we first transform each query image into surrogate that strongly overfitted to the target model's learned distribution. Given a query image MOFIT first constructs a motel-fitted surrogate, Le., 2 = Zo + 5", where 8 tightly optimized perturbation such that z appears more coherent with the model's internal distribution when perceived by the target LDM. From this surrogate, we derive its paired embedding o² by minimizing the conditional loss Lcond (Eq. D. forming a overfitting pair (zūj Φ²). For membership inference, MOFIT then conditions the original query 2y with the model-fitted embedding Φ", i.e., (Zy, o*). Intuitively, given Zu, MoFIT constructs surrogate -ereedddid pair (Iz. σ²) that is not only tightly aligned in the model's conditioning space, but also deliberately overfitted to the target model's 's internal distribution. Thus, conditioning the original query on 6² elicits asymmetric sensitivity: member samples incurs

---

Published as a conference paper at ICLR 2026
pronounced Leond responses while hold-out samples exhibit relatively modest changes. An overview of MOFIT depicted in Fig.
We note that is not the exact counterpart of o"; this mismatch (æo, O*) in inference induces mis- alignment between the image and its conditioning. Accordingly, member samples exhibit heightened sensitivity and produce larger cond responses than when conditioned with VI.M-generated captions (Co. @VLM), as observed in Sec. 3.3
4.1 MIDEL-FITTED SURROGATE OPTIMIZATION
MoFIT constructs a surrogate image that is explicitly optimized to resemble training samples, thereby producing variant that is intensively adapted to the model's unconditional prior for a given query image. Concretely, it injects perturbation 5 into the query image Tu. i.e, z6 = 2o + 6. This surrogate image 2o is then forwarded to specific timestep t in the forward process using single sampled noise vector ~ N/0,, I),, i.e., zi = = 20 + vI drξ where zó denotes the latent encoding of the perturbed image x/6-
In the absence of captions, we use the null conditioning @mulll and optimize å to make the model's unconditional prediction match the sampled noise é Formally, solve:
8 arg min Lumcund arg min 4, [lle co t, omun))l
To promote strong convergence toward the model's learned manifold, we fix and & during op- timization, thereby stabilizing the direction of perturbation. This model-fitted surrogate image t zo + 8" then serves as the input for optimizing an embedding that is tightly coupled with the surrogate.
4.2 SURROGATE- DRRVIEN EMBEDDING EXTRACTION
Given the model-fitted surrogate zÿ. we aim to extract an embedding ø² that reflects the model's response to this model-aligned input. To this end, we treat Φ as an optimizable parameter and minimize the conditional denoising loss (Eq. I under the same noise é and timestep used in the
previous surrogate optimization stage:
= arg min Ezg,,, [llé c0 t, o)]]
(6)
We initialize é with the embedding of a VL-M-generated caption, which may serve as suitable starting point for the optimization. Consequently, d* becomes conditioning embedding optimized to best describe zy, and used as the condition the membership inference stage. Condition- ing the original image Ho o" creates a deliberate image-condition mismatch that elicits large increase in Loond for member samples, while hold-out samples exhibit only a modest change.
4.3 MEMBERSHIP INFERENCE WITH AMPLIFIED LCOND DISPARITY
Given the optimized embedding o² paired with the model-fitte surrogate zi. we perform member- ship inference by computing the discrepancy between conditional and unconditional losses on the original image Zg- Since o* intensively optimized to align with the model-fitted surrogate z under fixed t and é, but does not correspond to the original image 2o, both member and hold-out samples receive mismatched condition. However, only member samples sensitive to misaligned conditions respond with significantly elevated Leomd values under ø² whereas hold-out samples remain less affected by the mismatch. This behavior difference forms the basis of our inference signal.
Accordingly, we define our membership score as the difference between conditional and uncondi- tional denoising losses:
LMOFIT E.,.t. [l|č co(2,, t, o" 2 E.o.,. [lle - co(2,, t, (mmm))1
To further enhance the discriminative power of our attack, we incorporate auxiliary losses that have shown utility in prior work (Zhai et al., 2024). particular, we consider unconditional loss Luncood well the CLiD score based on VLM -egeerrret captions. The latter is computed follows:
LVLM
Ezo.
[[| c0 5t, dv Ezg.t. [l]ē oo (ze, L, (lm)))l
(8)

---

Published as a conference paper at ICLR 2026
Pokemon
MS-COCO
Flickr
Methods
Condition
ASR
AUC
TPR@ ISFPR
ASR
AUC
TPRØ 1%FPR
ASR
AUC
TPR IIFPR
CLii
GT
96.52
90.14
91.10
77.20
Loss
72.27
78.99
4.81
63.70
67.88
4.80
61.60
64.24
5.40
SecMI
78.51
86.
6.97
57.30
58.07
4.20
54,00
52.38
2,00
PIA
VLM
71.79
76.76
10.82
66.00
69.70
61.00
64.05
5.00
PFAMI
74.43
81.25
6.01
80,40
87.50
29.40
76.90
84.99
24.80
CLiD
77.55
83.43
19.23
80.90
86.53
50.80
79.00
85.16
40.60
MONTT
94.48
97.30
50.48
88.00
94.17
47.00
86.00
91.32
53.20
Table 2: Comparison of membership inference performance under the caption-free setting, where baseline methods conditioned using either ground-trui VLM-generated captions. Bold num- bers denote the best, and underlined numbers indicate the second-best results.
where @VLM denotes the embedding of VLM-generaan caption. We then formulate the final mem- bership decision rule as a weighted combination of the normalized losses, following the robust- scaling strategy introduced (Zhai et al. 2024). The corresponding membership prediction is computed as:
M(x, 1 [r R (EmoFi) + (1-7) .R (-Laux) r].
where Laxx E { (ancoodd LVTM). and the negation on Laux reflects the inverted loss dynamics of our score function, whereby member samples attain higher scores than hold-out samples. R(-) denotes the robust scaler defined R(w,)= (w, i)QQQR with w as the median and IQR as the interquartile range. The hyperparameter E [0, controls the balance between our proposed score and auxiliary loss, and is the decision threshold.
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
MOFIT & Baselines. We iteratively update the perturbation 5 along the gradient sign direction, employing a step size initialized 0.15 and linearly decayed in proportion to the iteration count throughout optimization. The resulting model-fitted surrogate is then used to extract an embedding via the Adam optimizer with a learning rate of 0.06. Throughout both optimization stages, the diffusion timestep is fixed at = 140, within a total schedule of 1000. Optimal hyperparameters are selected based on search results presented in Fig. 5 of Appendix A.T
We consider five prior methods - Loss-based inference (Matsumoto et al. (023). SecMI (Duan et al., 2023), PIA (Kong et al., 2023). ) PFAMI (Fu et al., 2023), and CLiD (Zhai al. 2024) as baselines, each conditioned on VLM--gnerated captions to simulate the caption-inaccesse inan-cesicceeeeee setting. For real- world datasets (i.e., MS-COCO and Flickr), we generate captions using BLIP-2 (Li et al. 2023). while for the stylized dataset (i.e., Pokemon), we employ CLIP- CLIP-Interrogator. Additional details are provided in Appendix A.I
Target Models. We evaluate method Stable Diffusion v1.4 fine-tuned three datasets Pokemon, MS-COCO (Lin et al. 2014], and Flickr (Young et al. 2014)) - as well as the pre-traines Stable Diffusion v model. However, noted in Dubinski et al. (2024) and further demonstrated in Tab, Tab. of Appendix I existing methods perform near chance level on the LAION-mi split a member/hold-out partition specifically constructed for Stable Diffusion due the model's 's strong generalization. To clearly assess the performance difference between VLM-captioned baselines and our approach, replace the original member with 431 verified memorized samples (Webster) 2023). while retaining the LAION- mi hold-out set for evaluation.
Evaluation Metrics.
We report Attack Success Rate (ASR), AUC, and True Positive Rate at 1% False Positive Rate (TPR@ @ 1%FPR) following the standard metrics used in our baselines. For MS- COCO and Flickr, we evaluate on 500 randomly sampled images from each of the member and hold-out sets, while all available images are used for the Pokemon dataset.
https::/hhugginngfface (atabledde ffust on-vlon-vvl-1 f foa.ealled--fuuu-oooooo Ffusi -n-

---

Published as a conference paper at ICLR 2026
5.2 EVALUATION ON TUNEE LDMS
We evaluate the membership inference performance of various methods on latent diffusion models (LDMs) fine-tuned with three distinct datasets Pokemon, MS-COCO, and Flickr under a prac- tical threat model where only query images are accessible to the auditor. In Tab. 2 we first report the performance of CLiD (Zhai et al. 2024) when access to ground-truth (GT) captions is granted, representing the upper- bound case (first row). In the caption-free setting, baseline methods may utilize captions generated by VLMs as alternatives (second row). However, this substitution leads to substantial drop across all evaluation metrics. CLiD's ASR on the Pokemon dataset decreases by nearly 29% when GT captions are replaced by VLM -LM-generated alternatives. This result under- scores a critical limitation: while VLMs can generate semantically relevant descriptions, they cannot replicate the exact ground-truth captions, thereby failing to recover the same conditioning effect.
MoFIT conditions each query image zo using its model-fitted embedding o', which is extracted from a surrogate image z0 specifically optimized to align with the model's learned distribution. Crucially, because 6 is highly tailored to 2q. conditioning the query To with 6² during inference results in pronounced misalignment. This effect amplifies membership-specifie responses, thereby improving inference accuracy. As reported in Tab. 2 MOFIT significantly outperforms prior meth- ods conditioned on VLM-generated captions across both ASR and AUC. Remarkably, it even sur- passes CLiD with ground-truth captions on the MS-COCO dataset, indicating that surrogate-base based misalignment can serve as a competitive alternative to original training captions.
5.3 EVALUATION ON STABLE DIFFUSION
Stable Diffusion v1.5
Methods
Condition
We assess MOFIT on the pre-trained Stable Diffusion v1.5 (SD v1.5) using the modified LAION-mi bench- mark (see Sec. 5.1 for details). As demonstrated in
ASR
ALC
T I%FPR
CLiD
GT
77.38
77.83
49.65
Loss
68.21
74.73
4.87
SecMI
55,10
$4,57
8.12
Tab. 3] prior methods suffer notable performance degra- dation when conditioned on VLM-generated captions. In contrast, MOFIT outperforms all VLM-conditioned baselines and even surpasses the GT-captioned CLiD in ASR. Although its AUC is slightly lower, it achieves the highest TPR @ %FPR, exceeding the second- result by over 20%, demonstrating strong discriminative power in high-precision regimes. We include additional experiments on SD v2.1 (Sec. A.9.2) with different text encoder and SD v3 with fundamentally different architecture (Sec. A.9.3).
PIA
VLM
63.92
6.03
PFAMI
72.85
78.15
19.26
CLiD
58.12
59.28
4.18
MOFI
4"
77.61
71.03
41.30
Table 3: Evaluation on SD v1.5.
5.4 UNDERSTANDING THE SEPARABILITY OF MOFIT
We attribute the performance gain of MOFIT to the distinct sensitivity characteristics between member and hold-out samples. Interestingly, this disparity is further amplified when conditioning on the model-fitted embedding o". As observed in the MS-COCO dataset (Fig. b,,, top)), mem- ber samples exhibit a pronounced increase in feomd, indicating strong sensitivity to the mis- aligned condition. In contrast, hold-out samples exhibit only a modest increase their Loond dis- tribution closely aligned with that under other conditions (Fig. bb, bottom)).
(b)
40
Member Hold-out
Member
30
GT
Density
20
6.1
0.2 0.5 0.4 0.5
0.6
Ours
Coos Carart
Euroor
10
Hold-out
GT
0.04 CMert
0.08 Luncond
Ours
Eeord
0.1
0.3 0.5
Cora & Funcre
Figure 3: (a) Membership score distributions when conditioned on the model-fitted embed- ding e². (b) Loand and Luncond for member and hold-out samples under varying conditions.
As a result, when computing the final discrep- ancy score LMOFIT (Eq. 7 in Fig. Baa), member samples predominantly exhibit positive values due to the pronounced gap between Leon and Luncon-- In contrast, hold-out samples yield scores near zero, suggesting minimal impact from the change in conditioning. Results of Pokemon and Flickr datasets are depicted in Appendix A.2
5.5 EFFECTIVENESS OF MODEL-FITTED SURROGATE 25
To further evaluate the effectiveness of Zo + 8*, we perform ablation by varying the input image used to extract the embedding e². Specifically, we compare four configurations: (i) the original query image xo corresponding to the alternative described in Sec. 4 (ii) the query image with
8

---

Published as a conference paper at ICLR 2026
Input
Condition
Pokemon
MS-COCO
Flickr
ASR
AUC
@ |SFPR
ASR
AUC
T FPR
ASR
AUC
T I&FPR
75.63
81.64 14
11.06
78.00
85.59
31.00
75.50
82.75
27.20
ino +
93.99
96.42
10.34
81.70
89.76
29.20
79.60
86.63
28.60
ao AsAx
81.79
34.00
75.00
82.32
28.00
MOFIT
94.48
97.30
50.48
88.00
94.17
47,00
86,00
91.32
53.20
Table 4: Quantitative comparison of MIA performance under input image variations.
random noise Zig + 6, (iii) an adversarial variant zo + ÓMAX optimized to maximize Eq. 5 and (iv) proposed surrogate 2a + 6", which minimizes Eq. For configuration (ii), we add uniformly sampled noise drawn from the range [-e,s] to the query image, and for each dataset, we sweep (0.1, .... 0.9). reporting the results at the best-performing noise level (0.5 for Pokemon, 0.8 for MS-COCO and 0.6 for Flickr). Importantly. for (iii,, while MOFIT constructs a model model-fitted fitted pair (rg. 6²) that is tightly coupled and mutually adapted to the target model, ÓMAX forces the query To to explicitly deviate from the model's 's learned distribution.
As shown in Tab. MOFIT outperforms all alternative input types, underscoring the importance of extracting &= from a surrogate zo that is carefully aligned with the model's learned represen- tation. This pairing forms a mutually adapted structure: zō is tailored to tightly conform to the model, and 6 captures this overfitted signal with high fidelity. While random noise ó offers modest discriminative signals in specific cases (c.g., Pokemon), its performance lacks consistency across different datasets. In contrast, MOFIT achieves stable and superior results in all evaluation met- rics and datasets, demonstrating robust generalization to variations in training data. Membership score distributions for each dataset are provided in Appendix A.4 and additional ablation studies are detailed in Appendix A.
5.6 DISCUSSIONS
5.6.1
OVERAITIIGG DEGREE BURROGATE--EBEEDIIG PAIRS
To assess whether the two-stage optimization in MoFIt effectively overfits both the surrogate im- age x² and the corresponding embedding Φ", we
20
20
Donsty
Demityy 5 tr
(a)
examine the distributions of Leond and Lumcond com- puted by the target model. Specifically, we compare these loss values for ground-truth image-caption pairs (I, c) and our model-fitted pairs (z).
0.04
0.12
0.20
0.04
0.12
0.20
Lncord
Member (x.4)
Hold-out c)
300
3000
Fig. a illustrates the loss distributions of (x, from the Pokemon dataset. In Fig. 4b, left), we
Densaty
200
Density
2000
100-
1000
present the Luncond values of x= where corresponds to the outcome of the surrogate optimization stage (Sec. 1.1 in MOFIT. Notably, the Luncond values of x² are significantly lower than those of x. indicating that the surrogate has been strongly overfitted the model's 's unconditional prior.
0.0125
0.0175 0.0225
0e-04
1e-03 2e-03
uuccnd
Coard
Member (x', Hold-out
Figure 4: Distributions of (left) Luncond and (right) Leond of member and hold-out pairs from (a) Pokemon dataset and (b) model- fitted pairs of MOFIT.
In Fig. lb, right), we observe further decrease in Loond following the embedding optimization stage (Sec. 4.2h. where the embedding 6² specifically tailored to pair with the surrogate x². Compared to the Leond distribution of (ar, e) in Fig. Ja. right), the loss values of (x*, ΦO)) are not only substan- tially lower but also more concentrated exhibiting significantly reduced variance. This suggests that Ø² is highly aligned with the surrogate x= on the model's 's learned manifold. This strong align- ment is expected to induce substantial mismatch when o² is applied to the original query image x, thereby amplifying the sensitivity of member samples during membership inference. Corresponding loss distributions for other datasets are provided in Appendix A.5
This uniform overfitting serves critical foundation: once the surrogate and its embedding is overfitted and tightly aligned, the embedding can more effectively separate members from hold- outs, evidenced by the improved performance of MOFIT in Tab.

---

Published as a conference paper at ICLR 2026
(a) Gaussian Blur
(b) JPEG Compression
(c) LoRA
Methods
Condition
ASR
AUC
TPR@ ISFPR
ASR
AUC
TPR 1%FPR
ASR
AUC
TPROISFR
CLii
GT
70.40
89.59
59.00
1.00
Loss
64,10
68.17
4.20
63.40
66.53
3.80
54.50
49.26
0.00
SecMI
60. 10
62.15
5.20
54,00
54.59
2.80
58,50
53.50
1,00
PIA
VLM
63.10
65.28
4.40
64,90
67.18
58.50
53.50
0.00
PFAMI
80.60
87.51
27.72
78.04
84.39
36.03
73.50
77.30
1.00
CLiD
81.00
87.17
44.60
78.80
85.21
39.20
59.00
53.11
1.00
MONIT
88.70
94.92
54,20
82.80
89.75
26.00
58.50
54.35
0.00
Table 5: Membership inference performance under fine-tuning with data augmentations: (a) Gaus- sian blur and (b) JPEG compression. (c) Fine-tuning with Low-Rank Adaptation (LoRA) (Hu et al. also shows potential as a defense.
2022
5.6.2 POTENTIAL DEFENSIVE METHON
Data Augmentation. To evaluate resilience against defender-side strategies, we fine-tune SD v1.4 on MS-COCO with augmentations: (a) Gaussian blur (3 x kernel, a E [0.1, 2.0) and (b) JPEG compression (quality = 60). MOFIT is then evaluated using embeddings from the non-augmented base model, simulating challenging setting where the attack is unaware of input-space changes during fine-tuning. Tab. a.b)) shows that all methods exhibit comparable or slightly degraded per- formance under augmentation. Two trends remain: basclines degrade notably when switching from GT to VL.Mggnnrraed captions; and (ii) MOFIT consistently outperforms all baselines under VLM captions, maintaining ASRs above 82.80% even with augmentation.
LoRA. We observe that Low-Rank Adaptation (LoRA) (Hu et al.) 2022), which updates only small set of additional parameters instead of the full U-Net, significantly degrades the performance of MoFIT and existing baselines. As shown in Tab. S(c). both MoFit and most baselines drop to rear-random performance when evaluated on 100 training samples from the LoRA-adapted SDv1. captions. We attribute this robustness to LoRA's minimal footprint, which retains most original weights and reduces memorization capacity (Amit
under both ground-truth and VLM-generatee
al. 2024). Additional insights into PFAMI's robustness are provided in Appendix A.7
5.6.3 LIMITATION: RUNTIME AND EARLY STOPPPING
One limitation of MOFIT may be the runtime, taking 7 to minutes per image optimize the surrogate and extract its embedding (see Appendix I.). Thus, we conduct early stopping strategy that terminates the optimization process once predefined loss threshold is reached. We evaluate 100 images each from the member and hold-out splits MS-COCO, and compare with the state- of-the-art baseline CLiDD performing 83.50% ASR and 87.37 AUC in the same setting. In Tab. of Appendix [A.8.2 MOFIT outperforms CLiD when the optimization is stopped at 0.08 during surrogate optimization or at 0,007 during embedding extraction, saving 336.64 and 75.93 seconds, respectively. This suggests that an adversary can strategically balance efficiency and effectiveness by choosing an appropriate early stopping criterion. Additional details are provided in Appendix A.8.2
CONCLUSION
We present MOFIT, a novel membership inference framework that operates under a practical caption-free setting. MoFIT enforces separability between member and hold-out samples through a model-fitted surrogate with its tightly optimized embedding, which amplify the conditional loss for member samples while keeping hold-out samples relatively stable. Extensive experiments across multiple benchmarks demonstrate that MOFIT substantially outperforms prior otate-of-the-art meth- ods conditioned VLM-generater captions and, in some cases, surpasses caption-dependent basclines. We believe this work broadens the scope membership inference in generative models and underscore the need for stronger safeguards against MIA attacks.
httts::/hugginngface co/
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We would like to thank the reviewers for their constructive comments and suggestions. This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2023-00208506) and the Institute of Information & Communica- tions Technology Planning Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2020-I1200 Penetration Security Testing of ML Model Vulnerabilities and Defense). Prof. Sung-Eu Eui Yoon and Prof. Sooel Son are co-corresponding authors.
ETHICS STATEMENT
This work does not involve human subjects, personally identifiable information, or sensitive user data. All experiments were conducted using publicly available datasets: MS-COCO, Flickr-8k, LAION-mi, Webster (2023), and released fine-tuned Pokemon diffusion model. Pokemon dataset has been taken down due to copyright issues, so used the dataset released by the authors of SecMI (Duan et al. 2023
Our proposed method, MOFIT, is intended to study and evaluate privacy vulnerabilities in generative diffusion models. While membership inference can reveal training data exposure, our findings are strictly aimed at understanding the privacy risks of large-scale generative models and informing the design of more robust, privacy-preserve
architectures.
We do not release any models, tools, datasets that could be directly used to exploit privacy vul- nerabilities in deployed systems. We follow responsible disclosure principles and emphasize that contributions for defensive and diagnostic purposes only. No personally identifiable data or private user content was used in this research.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we provide comprehensive implementation details of our proposed frame- work MOFIT in the main paper (Sec. (5.1) and include additional training configurations, hyper- parameters, and architectural descriptions in the appendix. Dataset used (MS-COCO, Flickr-8k, LAION-mi, Webster (2023). and the released SD-Pokemon model) are publicly available, and describe the dataset splits and preprocessing steps in Appendix A. For key components such as surrogate optimization, embedding extraction, and evaluation metries (e.g.. CLiD score), algorith- mic procedures are formalized in the main text, and corresponding ablation and sensitivity analyses are included Appendix A.
THE USE OF LLMS
The author(s) used ChatGPT for minor grammatical refinements of the manuscript. These modifi- cations have been manually reviewed and finalized by the author(s).
11

---

Published as a conference paper at ICLR 2026
REFERENCES
Guy Amit, Abigail Goldsteen, and Ariel Farkash. Sok: reducing the vulnerability of fine-tuned language models membership inference attacks. arXiv preprint arXiv:2403.0844 088481,,, 2024.
Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX security symposium (USENIX Security 23), pp. 5253-5270, 2023.
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al. Symbolic discovery of optimization algorithms. Advances in neural information processing systems, 36:49205 494333, 2023.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 28 2818-2829, 2023.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Moham- madreza Salchi, Niklas Muennighoff, Kyle Lo. Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for oftte-of-thearr rision-language models. In Proceedings of the Com- puter Vision and Pattern Recognition Conference, pp. 91-104, 2025.
Jinhao Duan, Fei Kong. Shiqi Wang, Xiaoshuang Shi, and Kaidi Xu. Are diffusion models vulner- able to membership inference attacks? In International Conference on Machine Learning, pp- 8717-8730. PMLR, 2023.
Jan Dubiáski, Antoni Kowalczuk, Stanislaw Pawlak, Przemyslaw Rokita, Tomasz Trzciński, and Pawel Morawiecki. Towards more realistic membership inference attacks on large diffusion mod- els. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision January 2024.
(WACV), pp.
Alex Ergasti, Claudio Ferrari, Tomaso Fontanini, Massimo Bertozzi, and Andrea Prati. Controllable face synthesis with semantic latent diffusion models. In International Conference on Pattern Recognition, pp. 337-352. Springer, 2024.
Cong Fu, Keqiang Yan, Limei Wang, Wing Yee Au, Michael Curtis McTThrow, Tao Komikado, Koji Maruhashi, Kanji Uchino, Xiaoning Qian, and Shuiwang Ji. A latent diffusion model for protein structure generation. In Learning on graphs conference, pp. 29-1. PMLR, 2024.
Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, and Tao Jiang. A proba- bilistic fluctuation based membership inference attack for diffusion models. arXiv preprint arXiv:2308. 12143, 2023.
David Hasler and Sabine E Suesstrunk. Measuring colorfulness in natural images. In Human vision and electronic imaging VIII. volume 5007, pp. 87-95. SPIE, 2003.
Jonathan Ho and Tim Salimans. arXiv:2207. 12598, 2022.
Classifier-free diffusion guidance. arXiv preprint
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3 2022.
Fei Kong. Jinhao Duan, RuiPeng Ma, Hengtao Shen, Xiaofeng Zhu, Xiaoshuang Shi, and Kaidi Xu. An efficient membership inference attack for the diffusion model by proximal initialization. arXiv preprint arXhv:2305.1835 2023.
LambdaLabs. Pokémon blip captions.
wtt.ssttaaaaatt////s.o Lambdal http:/aaaaaa/p/oooooaaaaaa. captions.. 2022. Accessed: 2025-09-04.
Jingwei Li, Jing Dong, Tianxing He, and Jingzhao Zhang. Towards black-box membership inference attack for diffusion models. arXiv preprint aoXiv:2405.2777, 2024.
12

---

Published as a conference paper at ICLR 2026
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-iman pre- training for unified lssounla understanding and generation. In International conference on PMLR, 2022.
machine learning, pp.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning. pp. 19730-19742 PMLR, 2023.
Tsung-V Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Zihao Luo, Xilie Xu, Feng Liu, Yun Sing Koh, Di Wang. and Jingfeng Zhang. Privacy-preserving low-rank adaptation against membership inference attacks for latent diffusion models. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 5883-5891, 2025.
Tomoya Matsumoto, Takayuki Miura, and Naoto Yanai. Membership inference attacks against diffusion models. In 2023 IEEE Security and Privacy Workshops (SPW), pp. 77-83. IEEE, 2023.
Daniele Molino, Camillo Maria Caruso, Filippo Ruffini, Paolo Soda, and Valerio Guarrasi. Text- to-ct generation via 3d latent diffusion model with contrastive vision-language pretraining. arXiv preprint drvi:22506. 2500::200000003333, 2025.
Obioma Pelka, Sven Koitka, Johannes Rückert, Felix Nensa, and Christoph M Friedrich. Radiology objects in context (roco): multimodal image dataset. In Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis: 7th Joint International Workshop. CVII-STENT 2018 and Third International Workshop, LABELS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings 3. pp. 180-189. Springer, 2018
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10694-10695, 2022.
Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. Orb: An efficient alternative to sift or surf. In 201/ International conference on computer vision, pp. 2564-2571. leee, 2011.
Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379 423, 1948.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut, Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556 2565, 2018
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. In 2017 IEEE symposium security and privacy (SP), pp. 3-18. IEEE, 2017.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art digital forgery? investigating data replication in diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6048-6058, 2023.
Ryan Webster. A reproducible extraction of training images from diffusion models. arXiv preprint arXiv:2305.086/0 2023.
Peter Young. Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the association for computational linguistics, 2:67-78, 2014.
Shengfang Zhai, Huanran Chen, Yinpeng Dong. Jiajun Li, Qingni Shen, Yansong Gao, Hang Su, and Yang Liu. Membership inference text-to-image diffusion models via conditional likelihood discrepancy, Advances in Neural Information Processing Systems, 11174122--7146,, 2024.
13

---

Published as a conference paper at ICLR 2026
APPENDIX CONTENTS
A.1
Implementation Details
A.2
MOFIT on Pokemon and Flickr
A.3
Response of member and hold-out samples according to the condition
A.4
Input image variations
A.5
Handling Unknown Memhership via Surrogate Overfitting
A.6
Results of baselines with ground-trune truth captions
A.7
Membership Inference against LoRA-Adapted LDMs
A.8
Runtime of MOFIT and Early Stopping Strategy
A.8.1
Optinizer-Specific VRAM Usage and Runtime
A.8.2
Early Stopping Strategy
A.9
Evaluation against Stable Diffusion v1.5, v2.1, and v3
A.9.1
Stable Diffusion v1.5
A.9.2
Stable Diffusion v2.1
A.9.3
Stable Diffusion v3
A.10
Evaluation against Stable Diffusion Fine-Tuned on Medical Dataset
A.11
Recent Vision-Language Models (VLMs)
A.12
Stability of MoFIT against Random Target Noise
A.13
Failure Cases of MOFIT
A.14
Limited Number of Training Samples
A.15
Comparison with Black-Box Method, ReDiffuse
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 IMPLEMENTATION DETAILS
We provide additional implementation details to improve the reproducibility and clarity of our perimental setup.
Optimization Details & Ablation Study. We iteratively optimize the perturbation ó in the direction of the gradient sign. Specifically, the perturbation is uniformly sample from the range [-0.3, 0.3] and is updated via following equation:
z =
sign( V Luncond),
(10)
where Luncond denotes the unconditional loss defined in Eq. 5 and xy represents the surrogate being optimized at iteration i. The step size ar is initialized to 0. 0.15 and proportionally decayed with the iteration count.
To justify the choice of a, we conduct hyperparameter analysis across multiple values: E {0.025, 0.05, 0.1, 0.2, 0.3]. We perform this analysis on the Pokemon dataset by randomly sampling 100 images from both the member and hold-out sets. As shown in Fig. [5] increasing generally improves attack performance up to a certain point, with both ASR and AUC peaking at a = 0.15. Based on this observation, we use 0.15 throughout all experiments.
The resulting model-fitted surrogate is then used to extract an embedding via the Adam optimizer with learning rate of 0.06. The number of optimization steps varies by dataset: 200 steps for Pokemon, 300 steps for MS-COCO and Flickr, and 1000 steps for Stable Diffusion v1.5. The increased iteration count for Stable Diffusion is due its strong generalization, which requires more steps to be overfitted.
For single query image, the overall runtime for its two-stage optimization process varies by dataset: approximately 7 min- utes for Pokemon, minutes for MS-COCO and Flickr, and minutes for Stable Diffusion v1.5, all measured on a sin- gle NVIDIA GeForce RTX 4090 GPU. Please refer to Sec. A.8.I] for more details.
ASR and AUC vs. Initial stea size
ASR and AUC.v.. Timestap
ASR
ASR
AUC
AUC
6300
0.30
6:200
0.500
Timestop
Figure 5: ASR and AUC for different initial step size and timestep
Both optimization procedures are performed at a fixed diffusion timestep of 140, within the full denoising schedule 7 1000. This choice is guided by abla- tion results presented in Fig. where ASR and AUC are evaluated across varying timesteps ranging from / = 50 to 700. The results indicate that performance peaks within the range [100, 200]. Accordingly, we adopt t = 140 for all experiments.
Membership Inference. For the auxiliary loss Laux in Eq. we use Luncond for Pokemon and Stable Diffusion v1.5, and use LVLM for MS-COCO and Flickr. The balancing hyperparameter is increased by 0.05 within the range [0, 1). Threshold is selected maximize ASR for each 7
Baselines. All baseline methods are evaluated using their default settings. For the Naive method Maasuuooto et al.] 2023), membership determined based on Loand- For SecMI (Duan et al. 2023) and PIA (Kong et al. (023). we adopt the SecMI-stat and the default PIA, respectively both rely on threshold-based inference without training a classification model. When ground-truth captions are available, CLiD (Zhai et al., 2024) proposes several caption-splitinng strategies (e.g., simple clipping, random noise, and word importance calculation). Among them, we adopt simple clipping. Unlike CLiD, MoFIr follows the settings of SecMI and PIA, assuming access to subset of member and hold-out samples for threshold calibration; hence, we do not train a shadow model to determine y T.
Target Models. We evaluate our method on a range of toxt-to-image diffusion models trained on diverse datasets. We begin with SD-Pokemon, introduced in Sec. 3.3 which fine-tunes Stable
15

---

For mem- Published as a conference paper at ICLR 2026
Pokemon
Flickr
(b)
100
Member
40
Member
Member GT VLM
80
Hold-aut
Hold out
Density
Density
30
60
40
0.04
Focoa Auword
0.12
0.20
02
0.3
Care Conond
0.4 0.5 D6
Epnord
10
Hold-out
VLN
CMOFR
Loord
Luccond
CMoF: = Ccond Luncond
0.02
8.
0.5
Durs
Corcord
Figure 6: Score distributions on the Pokemon and Flickr datasets. (a) LMORA (Eq. score of member and hold-out samples. (b) Lcond and Luncond under condition variations.
Diffusion v1.4 416/417 member/hold-out image-caption pairs for 15,000 steps. Using the same base model, we further fine-tune on 2,500/2,458 splits of MS-COCO and 2,500/2,500 splits of Flickr, each with 150,000 steps, following the experimental setups in prior studies (Duan al. 2023 Zhai
ct al.
2024 Kong et al. 2023
To evaluate our method on a widely-used large- scale model, we additionally consider the pre- trained Stable Diffusion v1.5 [ bership evaluation, we adopt the LAION-mi benchmark (Dubiński et al. 2024), which pro- vides curated member and hold-out splits for Stable Diffusion. However, as discussed in Sec.5.1 existing methods perform near chance level due the model's 's high generative capac- ity. We evaluate all baseline methods using 500 images each from the member and hold-out sets of LAION-mi. As demonstrated in Tab. 66 ASR of prior methods is almost the same as random guessing, even in the condition of ground-truth captions (Please refer to Appendix AA.9II evalua- tion of MoFIT). Accordingly, to enable a more discriminative evaluation. we substitute the member set with verified memorized images, identified using the reproduction methodology (Webster) 2023). This curated benchmark allows for more discernible comparison between VI.M-captioned baselines and our approach, enabling clearer assessment of each method's effectiveness. Among the 500 image URLs, we utilize the 431 images that were successfully downloaded.
LAION-mi
Methods
Condition
ASR
AUC
TPR@ 1%FPR
Loss
55.02
50.72
1.60
SecMI
53.55
53.34
2.90
PIA
PFAMI
52.75
51.67
0.75
CLiD
56.90
58.62
4.60
CLiD
VLM
53.80
$2.09
Table 6: Quantitative comparison on SDv1.5 for LAION-mi.
VLMs. Since BLIP-2 is designed to generate natural language descriptions real-world images, we use it to produce alternative captions for the MS-COCO, Flickr, LAION-mi, and Webster (2023) datasets. For the stylized Pokemon dataset, we instead employ ELIP-Interrogatorr, which is com- monly used to infer the underlying text prompts that may have been used to generate given syn- thetic image. This choice is further motivated by the fact that BLIIP (Li et al. 2022 was previously used caption the original Pokemon training set.
A.2 MOFIT ON POKEMON AND FLICKR
addition to Fig. we present the corresponding score distributions for the Pokemon and Flickr datasets in Fig. 6 For both datasets, the discrepancy scores in (a) exhibit clear separability between member and hold-out samples. Furthermore, in (b), the Leond values notably increase under the model-fitted embedding condition of MoFIT, while the Luncond values remain largely unchanged. As discussed in Sec. [5.2] this differential response plays key role restoring separability and accounts for the superior performance of MOFIT reported in Tab. 2
16

---

Published as a conference paper at ICLR 2026
Members
Hold-out
35
GT
30
40
VLM
Danaty
20
10
Carcne
0.01
0.03
0.05
0.07
0.01
0.03
0.05
0.07
Pokemon
Member
Hold-out
Member
Hold-out
17.5
20
VLM Cuncrr
Hord-out
Nember Curcont
Hard out
10.0
Carcen
Dauiity
10
7.5
u
2.3
0.0
0.03
0.07
0.11
0.15 0.19
0.03
0.07 0.11 0.15
0.19
0.03
0.07
0.11
0.15
0.19
0.03
0.07
0.11
0.15
0.19
MS-COCO
(c) Flickr
Figure 7: Responses of Loond and Lancand across multiple datasets under different conditioning schemes.
ASR MS-Coro
100
Netes aange
Noise range
Neioe range
Figure 8: ASR and AUC when embeddings are extracted from perturbed images zo + â under varying noise levels E.
A.3 RESPONSE OF MEMBER AND HOLD-OUT SAMPLES ACCORDING TO THE CONDITION
As observed in Sec. 3.3 member and hold-out samples exhibit differing response to condition changes. We further investigate this deviation by expanding both the dataset and the types of condi- tioning. In addition to the Pokemon dataset, we include MS-COCO and Flickr to examine whether similar patterns emerge. We also introduce a new type of conditioning: while VLM "LL--generated captions semantically describe the given image, we simulate non-descriptive conditioning by us- ing captions from the opposite group - i.e., member images are conditioned on captions from hold-out set, and vice versa.
Fig. 7 shows the distributional changes of Loond and Cuncond across all datasets. Member samples consistently exhibit increased Lcond values when transitioning from ground-truth to VLM-generated captions, with a further increase under non-descriptive captions (red lines). In contrast, hold-out samples remain less affected, even under condition of captions from the member set. These results reinforce the two key observations discussed in Sec. 3.3
A.4 INPUT IMAGE VARIATIONS
Noise level & for each dataset in Sec. 5.5 In the experiment described in Sec. 5.5 embeddings are extracted from perturbed query images of the form xy 8, where 8 is uniformly sampled from We sweep E {0.1, 0.2. ... 0.9} to identify the optimal noise magnitude for each dataset. Fig. g reports ASR and AUC values as functions of e, computed using 100 member and 100 hold-out
https://hugginggface staale ee ffust www.sioonol--/ttaaaaaaa
fi..uuuuuuuu Ffust -
17

---

Published as a conference paper at ICLR 2026
Xg 5
Xg ŌMAX
An + 6" (Ours)
Memiite 40M0-out
40
Memmeer Hilt-cue
80
Member
Hilbout
Pokemon
Deniity
Danuity
0.8
Donsty
8.8
40
0.4
Momber
20
Mobo
-0.17
-0.12
-0.07
-0.02
0.82 0.00
0.02 0,04 0.06
0.2 0.4 0.6 08 1.0 12
0.00 0.01 0.02 0.03 0.04
Monn Eaard
Eunoone
Crerre Coand Curcord
Cierra
Coane
Cyrcord
CxpeR Ccons Eurcord
Denetty
Demety
Denstty 88
MS-COCO
10
0.0
Luncond
Crore Coans Curcend
Coase Ruscard
Ceare
12
Hember vatsoot
2.0
Mamoo -o1d-out
Dengit
Densty
Dergety
Denstyy
Flickr
Namber Fulcoou
-0.55
-0.45
-0.35 -0.25
0.30
0.6 0.8 1.0
0.00 0.02 0.04 0.05 0.00
DMuRt Ecord Cuncond
CHaPA Coard
Curcond
Cueoe = Coaad Curcend
Cuooit
Geoed Cuncord
Figure 9: LMaRM score distribution according to the input image variations.
MS-COCO
Flickr
Dendty
Density
Densty 2
(a)
2
0.3
0.4
0.6
0.3
044
0.5
Duncond
Goond
<uncond
Foond
Member
1000
200
1500
(b)
Density 15
Denatty
500
750
Denstry
150
100
Denuity
1000
500
250
0.020 0.025
0.030
2e-03
3e-03
4e-03
0.014
0.022
0.030
1e-03
2e-03
3e-03
Funcond
Ccond
Cuncond
Gondd
Member
Hold-out (x")))
Figure 10: Distributions of Luncond (dotted lines) and Lound for member and hold-out samples from (a) MS-COCO and Flickr datasets, and (b) model-fitted pairs used MOFIT.
samples per dataset. Based on these results, we select = 0.5 for Pokemon, E = 0.8 for MS-COCO, and 0.6 for Flickr. The corresponding evaluation results the full datasets summarized in Tab.
To further support the superior results of MOFIT reported in Tab. present the LMOFIT score distributions under different input variations: the original image To. a randomly perturbed image To + 6, adversarial variant zi + ŌMAX. and the model-fitte surrogate 20 + 8* used by MoFIr. As shown in Fig. 9 conditioning on Ty + 8* yields the greatest separability, demonstrating the effectiveness of MOFIT in leveraging model-fitted surrogates.
A.5 HANDLING UNKNOWN MEMBERSHIP VIA SURROGATE OVERFITTING
To infer membership status when the original caption is unavailable. alternative approach other than VLM "LM-generated captions- is to directly optimize a text embedding for the clean query image x using the conditional loss Land (as discussed in Sec. 4 However, as proven in Tab. 4 this strategy
18

---

Published as a conference paper at ICLR 2026
Pokemon
MS-COCO
Flickr
Methods
Condition
ASR
AUC
TPR@ 1%FPR
ASR
AUC
TPRØ 1%FPR
ASR
AUC
TPR@ ISFPR
Loss
80.07
66.70
4.60
SecMI
81,99
89.53
6.49
59.20
60.47
5.00
55,00
54,66
2.40
PIA
GT
80,43
86.90
19.95
78.28
84,64
16.40
70.31
74.94
5.80
PFAMI
75.75
47.10
91.41
83.00
90.58
41.20
CLiD
96.52
99.17
90.14
86,50
90.27
68.80
91.10
95.13
77.20
Table 7: Performance comparison of membership inference methods when the ground-truth captions are accessible.
MS-COCO
Flickr
(a)
&
Denaty
0.3
0.4
0.6
0.2
0.3
34
0.5
0.3
0.4
0.5
0.6
0.2
0.3
0.4
0.5
0.6
Cuncand
Coond
Luncond
Member (x,c)
Hold oud-outtk (x.
200
1500
(b)
Density
13 3
Danaity
500
750
Densay
150
100
Density
1000
0.090
2o-03
1e:03
3e-03
Cuncond
Ccond
Cuncond
Cord
D Member (x'.0")
Hold-out (r" (r",a"")
Figure 11:
Luncons (dotted lines) and Lood for member and hold-out samples from (a) MS-COCO and Flickr datasets, and (b) model-fitter pairs used in MOFIT.
results in a clear degradation in performance, suggesting that the clean image alone does not yield sufficiently discriminative embeddings.
In contrast, MOFIT cireumvents this limitation by consistently overfitting both member and hold- out query images to the model. Fig. (fbb demonstrates that both conn and Luncond distributions of model-fitted pairs (x*, x*) exhibit substantial overlap between member and hold-out samples. This observation implies that MOFIT effectively constructs pairs aligned with the model's learned representation, irrespective the sample's true membership status.
Building upon this analysis, Fig. 10 presents the Luncond and Lcond distributions for two additional datasets MS-COCO and Flickr alongside the corresponding distributions for the model-fitee fitted pairs in MOFIT. For both datasets, the distributions of (x* o²) exhibit lower magnitude and higher density compared to those of the original pairs (x, c). These results indicate that MoFIT effectively maps and intensively overfits both the surrogate and their coupled embeddings onto the model's generative manifold, regardless of the dataset.
A.6 RESULTS OF BASELINES WITH GROUND-TRUN CAPTIONS
In Tab. 7, we present the performance of baseline methods evaluated with access to ground-truth captions. CLiDD included in the main paper (Sec. (.2) as it demonstrates the best performance in this setting. Notably, MOFIT performs competitively with CLiD under ground-truth conditions the MS-COCO dataset.
A.7 MEMBERSHIP INFERENCE AGAINST ADAPTED LDMS
In Tab. c)c,, we report the degradation membership inference methods against LoRA-adapted Stable Diffusion v1.4 fine-tuned the Pokemon dataset. While Luo et al. (2025) report vulner- abilities in LoRA-adapted LDMs, their analysis focuses on early MIA methods (c.g., the "Loss" baseline) and does not recent approaches.
19

---

Published as a conference paper at ICLR 2026
(a) Surrogate
VRAM (MB)
Runtime (sec)
(b) Einbedding
VRAM (MB)
Runtime (scc)
Adam
18053.79
333.24
Adam (Ours)
14799.11
38.28
SGD
18052.38
333.19
SGD
14797.22
38.21
RMSProp
18050.83
333.66
RMSProp
14803,43
38.15
LION
18049.72
333.57
LION
14803.64
38,15
Ours
20667.50
358.96
Table 8: Comparison of GPU VRAM usage and runtime overhead across optimizers during (a)
surrogate optimization and (b) embedding extraction.
(a) Surrogate Optimziation
(b) Embedding Extraction
Threshold
Threshold
ASR
AUC
TPR@ FPF
Runtime
ASR
AUC
TPR 1%FP
Runtime
0.10
81.50
88.53
32.00
10.71
0.009
81.00
88.17
35.00
29.11
0.09
82.00
89.67
39.00
16.33
0.008
82.00
89.62
33.00
34.37
0.08
84.00
90.15
42.00
22.32
0.007
84.00
90.50
31.00
40.45
0.07
85.00
90.68
49.00
33.62
0.006
85.00
91.57
37.00
48.07
0.06
86.50
91.00
39.00
55.41
0.005
86.00
92.97
43.00
58.19
Total
358.96
Total
116.38
Table Performance of MOFIT under an early stopping regime applied to both surrogate optimiza- tion and embedding extraction.
PFAMI robustness. Unlike other baselines, PFAMI remains relatively robust under LoRA. We attribute this to its distinct metric formulation.
Baselines other than PFAMI rely on cross-query comparisons - typically based on the relative rank- ing of diffusion losses across different queries. The poor performance of these baselines under LoRA, as shown in Tab. (5cc), suggests that the loss values of member and non-member queries be- come increasingly similar. This makes it difficult for cross-quent quss-query methods to distinguish between them based on relative loss ordering.
However, PFAMI uses within-query scores by applying multiple augmentations (i.e., crop) to a single query image and computing the loss for each. The membership score is based on the relative variation within these losses. Since it does not rely on comparisons across different queries, PFAMI remains robust even when LoRA causes the overall loss values of member and non-member samples to become similar. The relative differences within single query are preserved, making it less sensitive to such global shifts.
A.8 RUNTIME OF MOFIT AND EARLY STOPPING STRATEGY
We report VRAM usage and runtime of MOFIT in Appendix I.8.I While MOFIT incurs high time cost, our focus is on achieving strong membership inference performance without relying on VLM- generated captions- demonstrating the first such result in the coption-free setting. Given this goal, we prioritized accuracy over efficiency. Nevertheless, recognizing the importance of cost-efficiency, we further explore the trade-off via early stopping in Appendix A.8.2
A.8.1 SHIMIZER-SPECIFIEE VRAM USAGE AND RUNTIME
We perform ablation studies comparing lightweight optimizers SGD, RMSProp, and LION CCen
et al. 2023).
(i Model-fitted surrogate optimization. MOFIT updates the perturbation using single-step, first-order update that directly applies the sign of the loss gradient sign(VL,))n - to the input image. As evaluated Tab. a[)) for 1,000 steps (default setting), this design results higher GPU con- sumption (up 2,618 1188M alternatives. However, this single-step method avoids the need for multiple optimization trials of hy- perparameter tuning (e.g., learning rate or momentum), which be costly and detaset-dependent. We view this trade-off as a design choice left to the adversary, depending on their resource con- straints and deployment scenario.
MB VRAM) and slightly longer runtime (up to 25.77s) than optimizer-base
(ii) Surrogate-driven embedding extraction. For the embedding extraction step from the model- fitted surrogate, employ the Adam optimizer by default. In Tab. (b) while the optimizers
20

---

Published as a conference paper at ICLR 2026
evaluated using 200 optimization steps (default setting for SD v1.4 fine-tuned with Pokemon dataset), we find that all optimizers perform consistently, with only a subtle difference. Given its stability and wide applicability across diverse tasks, we find Adam to be practical default.
A.8.2 EARLY STOPPING STRATEGY
To reduce the runtime of MOFIT, we perform additional experiments using an early stopping strat- egy. Specifically, we terminate the optimization once a predefined loss threshold is reached, applying this strategy to both (i) surrogate optimization and (ii) embedding extraction. All experiments are conducted using the SD v1.4 model fine-tuned on the MS-COCO dataset. This setup allows us to assess whether competitive performance be achieved with reduced computational cost.
(i) Model-fitted surrogate optimization. We first average the final loss (Eq, 5) from full opti- mization runs, which yield 0.02446 for members and 0.02259 for hold-outs. We then re-run the optimization and terminate early when the loss reaches a higher predefined threshold, choosing from 0.1, 0.09, 0.08, 0.07, 0.06. When a threshold is met, we the corresponding surrogate and proceed to extract the embedding using the same setup as in the main experiments (300 iterations for MS-COCO).
In Tab. 9l), we report membership inference results and average GPU runtime when applying early stopping the surrogate optimization stage of MOFIT. We evaluate on 100 member and 100 hold- out images from the MS-COCO dataset, using an NVIDIA RTX 4090. Notably, when compared to our baseline CLiD (83. 83.50% ASR and 87.37 AUC the same setting), MoFIT reaches comparable or superior performance even when stopped early- for instance, at a loss threshold of 0.08, it achieves competitive results CLiD while only taking 22.32 seconds per image (originally takes 358.9 sec). This suggests that adversary can strategically balance efficiency and effectiveness by choosing an appropriate early stopping criterion.
(ii) Surrogate-driven embedding extraction. We follow the same procedure and compute the average final loss of Eq. 6 after embedding extraction, obtaining values of 0.00232 for members and 0.00229 for hold-outs. We then re-run full surrogate optimization (1,000 steps default) for all samples, extract embeddings, and apply early stopping when the loss in Eq. 6 reaches a preset threshold: 0.009, 0.008, 0.007, 0.006, 0.005.
As shown in Tab. ()), we again observe a trade-off between runtime and attack performance, mir- roring the pattern in (i). Notably, MOFIT reaches comparable performance CL.iD at threshold of 0.007, while reducing runtime to 75.93 seconds per image. This demonstrates that early-stopping can be effectively applied at both optimization and embedding stages to flexibly adjust the cost- performance balance of MOFIT.
A.9 EVALUATION AGAINST STABLE DIFFUSION v1.5, v2.1, AND v3
We implement additional experiments on three publicly available pre-trained diffusion models with increasing scale and diversity: (1) Stable Diffusion v1.5 (SD v1.5), (2) Stable Diffusion v2.1 (SD v2.1), which differs the text encoder with SD vI (switch from CLIP to OpenCLIP (Cherti al. )023)) and (3) Stable Diffusion v3 (SD v3), which adopts different architecture based on Transformers instead of U-Net.
A.9.1 STABLE DIFFUSION v1.5
We evaluate on SD 1.5 using the LAION-mi split (Dubiñski al. 2024). a standard member/hold- out partition. We also increase the optimization timestep from f 140 to = 350, which improves MoFrT's performance on large-scale models. As shown in Tab. T0]a), although the task remains challenging, MOFIT consistently outperforms VLM-captioned baselines and even remains com- petitive with CLiD under GT captions. These results confirm MOFIT strong transferability to large-scale public models in realistic, caption-free settings.
ftttas://tthuu.oco//ttbbllll/// fdusiiioon
3- un-di ffusers
21

---

Published as a conference paper at ICLR 2026
(a) SD v1.3
(b) SD v2.1
(c) SD v3
Methods
Condition
ASR
AUC
TPR@ ISEPR
ASR
AUC
TPRØ 1%FPR
ASR
AUC
TPR ISFPR
CLii
GT
60.00
1.00
55.82
67.50
71.64
Loss
52.00
50.48
0.60
55.50
52.47
1.00
53.00
46.27
0.00
SecMI
52,50
50.28
2,00
57.50
54.37
0.00
62.50
65.06
4,00
PIA
VLM
51.50
49.75
1.60
59.00
57.53
55.00
50.84
0.00
PFAMI
54.00
52.58
0.75
51.50
39.63
1.00
68.50
69.81
3.00
CLiD
56.50
52.78
1.00
53.50
51.90
1.00
67.50
71.59
4.00
MONIT
6"
60.00
58.23
2.00
61.34
58.99
0.00
70.00
73.42
2,00
Table 10: Evaluation against three large-scale pre-trained models: SD v1.5 (LAION-m) train vs. test), SD v2.1 (LAION-mi train vs. CC3M), and SD v3 (COCO vs. CC3M).
A.9.2 STABLE DIFFUSION v2.1
Defining Member/HHoldouu Splits for SD v2.1. Unlike SD v1.5, which allows controlled evalu- ation via the curated LATON-mi split (Dubiñski et al. 2024). evaluating membership inference on SD v2.1 presents unique challenge. SD v2.1 is trained on LAION-5B, superset that contains both the member and hold-out images of LAION-mi. Thus, LAION-mi cannot be reused directly for testing generalization performance on SD v2.1.
To address this challenge, we construct dis- tributionally aligned evaluation split, follow- ing best practices in LAION-mi that recom- mend closely matched real-world distributions for member and hold-out samples. We com-
LAION-mi
LAION-mi hold-out
COCO
Flickr
CC3M
member vs.
53.6041
61.8270
16.0582
8,8673
pute FID scores between the LAION-m mem- ber set and candidate datasets (MS- (MS-COCO (Lin et al. 2014), Flickr (Young et al. 2014), CC3M (Sharma et al. 2018)) in Tab. 11 and find that CC3M yields the lowest FIID (16.06), closely matching the intra-L. Pitra-LAAION.. mi FID (8.87). Accordingly, we use LAION-mi members as the member set and CC3M as the hold-out for SDv 2.1 evaluation.
Table 11: FID scores between LAION-mi mem- ber set and several datasets.
Evaluation on SDv2.1 using real-world split. We perform membership inference attacks using 100 images from each split and evaluate MOFIT alongside all baselines. In Tab. [0]b). under VLM-generated captions- a realistic setting where the adversary lacks GT annotations all base- lines exhibit notable drops ASR and AUC. contrast, MOFIr consistently outperforms them, achieving +2.34% ASR gain over the second- best method under these restricted conditions.
A.9.3 STABLE DIFFUSION v3
Defining Member/Hold-out Splits for SD v3. licly available details about its training dataset aside from the number of train- ing images, making Empirically, in Fig. 12 observed that COCO samples yield significantly lower conditional and unconditional loss values (Eq. [ and Eq. 2 in the main paper) compared to CC3M. Based on this, we selected COCO as the member set and CC3M as the hold-out set for evaluating MOFIT on SDv3.
SD v3 provides no pub-
challenging to
construct reliable member/hold-out splits.
1.75
2.00
COCO
2.00
1:75
0000
1.50
CC:M
1.50
0C39
2 21.33
31
20.75
20.75
0.50
0.50
0.25
0.25
0.00
0.00
Evaluation on Pre-trained SDv3. We conducted membership inference using 100 images from each split, running all methods timestep (t 328. 1250, i.e., step 140). As shown Tab,
Goad
Canont
fixed diffusion
Figure 12: Lcond and Luncond of SD v3 for COCO and CC3M.
T c), MOFIT outperforms all baselines, in- cluding CLiD under ground-truth conditions. These results further confirm that MOFIT can match or even exceed methods that rely on full image-caption supervision, consistent with our findings on SD vl.4 (See Tab.
A. EVALUATION AGAINST STABLE DIFFUSION FINE-TUNED ON MEDICAL DATASET
22

---

Published as a conference paper at ICLR 2026
Methods
Condition
(a)
v1.5
(b) SD v2.1
(c) SD
ASR
AUC
TPR@1 IEEPR
ASR
TPR@1TFFR
ASR
AUC
TPRØ IEEPR
CLiD
MoonDream2
74.91
80,20
13.22
77.50
83,41
31.00
75.20
81.60
33.40
CLiD
Molmo
78.00
81.40
8.00
72.00
71.65
17.00
68.50
69.23
21.00
Table 13: Evaluation of two recent VLMs: MoonDream2 and Molmo Flux captioner.
We additionally evaluated MOFIT on a domain-specific Latent Diffusion Model (L.D)) trained on medical data. Specifically, we used Prompt2MedImaqe" [T] Stable Diffusion v1.4 model fine-tuned on the ROCO dataset (Pelka et al., 2018), which contains radiology image-caption pairs. The training and validation splits were used member and hold-out sets, respectively. Additionally, because general-purpose VLMs cannot re- liably caption domain-specific medical images, em- ploy a BLIP model fine-tuned on the ROCO datase to generate the required captions in the caption- -free set- ting.
Promp2Meellmage
Methods
Condition
ASR
AUC
TPR615FPR
CLiD
GT
60.50
60.30
0.00
Loss
$4,00
47.27
0.00
SeeMI
53.00
0.00
PIA
VLM
51.50
44.25
0.00
PFAMI
53,00
50.22
1,00
CLiD
55.00
49.78
1,00
MOFIT
57,00
54.44
2,00
Table 12:
Evaluation
against
Prompt2MedImage, tuned on ROCO dataset.
SD v1.4 fine-
As shown in Tab. MOFIT consistently outperforms all VLM-based baselines across all metrics, successfully distinguishing member samples from hold-outs even in this specialized medical do- main. These results confirm that MOFIT generalizes across domains, highlighting its effectiveness on hi-the-wild LDMs beyond general-purposs text-to-image settings.
A.11 RECENT VISION-LANGUAGO MODELS (VLMS)
In Tab. 13 we conduct experiments for MoonDream2 13 and Molmo Flux cap- tioner (Deitke et al.] 2025 to explore caption generation using recent vision- language models (VLMs). and evaluate on three fine-tuned SD v1.4 models (Pokemon, MS-COCO, and Flickr). Compared to Tab. 2 we still observe notable perfor- mance degradation when using VLM-generated cap- tions. These results highlight that even highly descrip- tive captions from powerful VLMs fail to recover the original image-caption supervision signal, underscoring the need for research in caption-free settings as our ap- proach.
We condition CLiD with the VLM-generated captions
ASR
AUC
TPR@ I%FP
Random I
94.50
96.96
18.00
Random 2
94.00
96.23
25.00
Random 3
94.00
95,85
18.00
Random 4
95.50
96.72
41.00
Performance of MoFIT under Table 14:
A.12 STABILITY OF MOFIT AGAINST RANDOM TARGET ÑOISE
four random target noise settings.
To evaluate MOFIT's sensitivity byper-parametesd the target noise is randomized. Specifically, we sample four random noise vectors from a stan- dard normal distribution and set each the target noise. This evaluation helps confirm that MoFrr remains robust to the choice of target noise, suggesting that any noise drawn from normal dis- tribution sufficient for our method. The experiment was conducted on SD v1.4 fine-tuned with the Pokemon dataset, using 100 member and 100 hold-out samples. As shown in Tab. 14 MOFIT performs consistently across all metrics when implemented with random noise.
conducted additional experiment where
A.13 FAILUREE CASES OF MOFIT
In this section, analyze extreme member and hold-out cases observed under MOFIT, highlighting their distinctive characteristics. Specifically, using the two fine-tume sune-tuned SD v1.4 models (MS-COCO and Flickr), select the top-20 samples from each of the following groups:
httpp: httpa:///uuggggsse....////tttcea
https::/huuginggaac.coo http://aaa....ccoaaraaaaaa//aaap medi cal capt on naninn-ro -a- roco
http::/meondreem..a/
23

---

Published as a conference paper at ICLR 2026
GT-captioned CLiD
FN
OR
Figure 13: Visualization extreme samples from (a) MOFIT and (b) GT-captioned CLiD settings.
MS-COCO
Flickr
Methods
TP
FN
FP
TN
TP
FN
FP
TN
Colorfulness
135.27
122.65
147.64
132.54
146.22
138.56
147.51
147,95
Entropy
15.22
13.70
15.11
14.27
15.55
15.05
14.92
15.33
Keypoint
500.00
471.95
499.25
492.20
500.00
494.55
499.65
497.75
Table 15: TP, FN, FP, and TN statistics using color diversity and image complexity metrics for two fine-tuned SD 1.4 models.
TP (True Positives): members predicted as members
FN (False Negatives): members predicted as hold-outs
FP (False Positives): hold-outs predicted members
TN (True Negatives): hold-outs predicted hold-outs
We then examine the color diversity and image complexity across these extreme samples. To quan- tify color diversity, used the colorfulness score (Hasler & Suesstrunk 2003), and to assess image complexity, we employed Shannon entropy (Shannon) 1948) and ORB keypoint count (Rublee et al.) 2011) Higher colorfulness indicates richer chromatic variation, while higher entropy and keypoint counts reflect greater structural complexity.
Interestingly, as shown in Tab. we observed that samples predicted members (TP and FP) consistently exhibit higher values across all metrics compared to their hold-out counterparts (FN and TN). This suggests that the model is more likely to associate colorful and structurally rich images with training membership. We believe this finding may offer an interesting direction for future work on understanding what types images diffusion models are more likely to retain.
Visualization of extreme samples. We first visualize representative extreme cases (TP, FN, FP, TN) of MS-COCO from our MoFIT setup in Fig. 13 a). Among member samples, FN images tend to exhibit low visual complexity and monotonous color schemes (e.g., plain black-and-whit scenes or sky/ocean backgrounds), whereas TP samples show vibrant colors (e.g., green hues) and diverse semantic components (e.g., kitchen items, road scenes). A similar pattern is observed in hold-out samples: FP images often include vivid colors (e.g., green, red) and rich details (e.g., reflection of a tower, food the table), while TNs are visually simpler (e.g., snowy landscapes, minimal objects).
We additionally include Fig. [3(b) to visualize representative extreme cases under the presence of GT captions, using samples extracted from the GT-captioned CLiD) setup. While the extreme
24

---

Published as a conference paper at ICLR 2026
(a) 500 images
(b) 1,000 images
Methods
Condition
ASR
AUC
TPR
ASR
AUC
IPR@1SFPR
CLiD
80.50
83.35
55.00
83.00
86.76
Loss
64.50
68.05
5.00
64.50
67.72
4.00
SecMI
62.00
62.31
4,00
58.00
59.49
4.00
PIA
VLM
66.50
70.71
7.00
69.50
71.83
9.00
PFAMI
83,00
87.26
23.00
83.50
90.45
31,00
CLID
76.00
80.32
23.00
78.50
82.55
29.00
MOFIT
86.00
91.89
27.00
88.00
90.35
30.00
Table 16: Performance against LDMs fine-tuned with only few training samples: (a) 500 or (b)
1,000 images.
Pokemon
MS-COCO
Flickr
Methods
Condition
ASR
AUC
IPR@1SFPR
ASR
AUC
TPR8 1%FPR
ASR
AUC
TPR@ I%FPR
ReDiffuse
GT
52.40
49.53
0.72
54,20
52.41
1.20
60.00
48.23
0.20
ReDiffuse
VLM
52.16
49.75
0.72
55.30
53.37
1.60
51.30
48.73
0.20
Table 17: Evaluation of ReDiffuse under GT and VL.M-generaedd captions.
samples are not fully identical, the same trend holds: FNs are visually plain, whereas FPs exhibit greater colorfulness and complexity. This consistency suggests that the observed behaviors of FPs and FNs may reflect general property of latent diffusion models, offering promising direction for future work on understanding which types of images are more likely to be retained less trained in model.
A.14 LIMITED NUMBER OF TRAINING SAMPLES
We assume case where a Stable Diffusion model is fine-tuned using limited number of training images. To simulate this low-resource scenario, we fine-tune SD v1.4 using only 500 or 1,000 images from the MS-COCO dataset over 30,000 60,000 steps, respectively maintaining the same steps-to-ing tees-to- ratio used in our main experiments (2,500 images for 150,000 steps). and following the setting introduced in CLiD.
Tab. reports the membership inference results for all baseline methods and MOFIT, evaluated on 100 member and 100 hold-out images. expected, prior methods show noticeable performance drop when moving from ground-truth captions to VLM-generated captions, highlighting the diffi- culty of caption-free MIA this sparse data regime. In contrast, MOFIT continues to outperform baselines even under the VLM-generated caption setting, demonstrating strong performance de- spite the reduced training data. These results indicate that MOFIT generalizes well low-resource LDMs, suggesting broader applicability and robustness in practical scenarios where only limited number private images are available for fine-tuning.
A.15 COMPARISON WITH A BLACK-BOX METHOD: REDIFFUSE
We additionally evaluate a recent black-box membership inference method, ReDiffuse (Li et al., 2024). ReDiffuse proposes black-box attack that infers membership based the reconstruction enror between a query image and multiple generated samples. While ReDiffuse black-box with respect to model architecture and parameters, it assumes access to ground-truth (GT) captions during inference. Thus, in the caption-free setting, it still relies on VLM-generated captions, similar to prior baselines.
While ReDiffuse was originally evaluated under an easier protocol where member and hold-out dis- tributions differ, re-assess under
realistic, distributionally aligned setting. Comparing Tab, [7 with Tab. its performance drops significantly under both GT and VLM -LM-ggnerated captions. This suggests that while ReDiffuse is effective when member and hold-out distributions are disjoint, its ability to infer membership deteriorates in more realistic scenarios where the distributions are matched.
25


Reviews:
[{"review_id": "ghavaKfZtD", "reviewer": "Reviewer_exJi", "summary": "The paper presents new methods for membership inference attacks on diffusion models, focuses on generative/representation learning, and addresses privacy.", "questions": "1. How does MoFit’s performance transfer to other classes of generative models or different LDM architectures? Could the authors provide results (or analysis) for other types of datasets (e.g., medical or highly out-of-distribution images) not covered here?\n\n2. What, in the authors' view, would be the most promising defense for LDMs against MoFit, aside from differential privacy? Are there results for even naïve defense schemes\n\n3. Can the authors provide distributions or examples of hard false positives/negatives at very low FPRs? Are there query images or image properties where MoFit’s advantage is diminished, and if so, what characteristics drive this behavior?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "By tackling the MIA problem in the absence of ground-truth captions, the work fills a clear gap in current literature, reflecting realistic adversarial constraints faced in practice.", "weaknesses": "1. Limited diversity in experiments with large models: While SD v1.5 is included, the exploration of truly large-scale public models (e.g., with more challenging real-world splits or harder negative scenarios) is limited. The LAION-mi split required special curation and so does not test MoFit “in the wild” under extreme generalization.\n\n\n2. The paper focuses heavily on the positive results of MoFit, but does not sufficiently examine its limitations, potential false-positive drivers (e.g., for out-of-distribution or near-duplicate samples), or how defender-side strategies (e.g., differential privacy or input randomization) could mitigate its efficacy.\n\n3. No significant discussion of defense strategies: The paper states findings are for diagnostic/privacy audit purposes, but does not offer suggestions or preliminary results for possible countermeasures or design recommendations for more robust LDMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QJyIRgVArm", "reviewer": "Reviewer_awTs", "summary": "Latent diffusion models (LDMs) have high text-to-image generation fidelity but risk training data memorization, audited by membership inference attacks (MIAs). Existing MIAs need ground-truth captions, ineffective with only images. This work proposes MOFIT, a caption-free MIA framework with two stages. Experiments show it outperforms VLM-conditioned baselines, even matching/surpassing caption-dependent methods.", "questions": "1. Optimize efficiency with adaptive stopping criteria and lightweight optimizers, and quantify efficiency-performance trade-offs for large-scale applicability.\n2. Test on non-ideal images/small-sample LDMs, add more VLMs/recent studies as baselines to boost result robustness.\n3. Derive member samples’ sensitivity from LDM mechanisms, analyze risks, and propose defense strategies and usage rules.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Fills the \"caption-free\" gap, fits real scenarios with undisclosed training annotations, and has practical value.\n2. Based on member samples’ higher sensitivity to mismatched conditions, its two-stage optimization is logically consistent.\n3. Outperforms VLM-based baselines across datasets/models, and surpasses caption-dependent methods in some cases.", "weaknesses": "1. Its two-stage optimization takes 7-9 minutes per image on RTX 4090, far slower than VLM-based baselines, unable to handle large-scale data.\n2. No tests on non-ideal images or small-sample LDMs; key parameters are only validated on Pokemon, lacking cross-scenario adaptability.\n3. It excludes recent caption-free MIA studies, uses only 2 VLMs as baselines, and lacks theoretical explanation for member samples’ sensitivity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UQyeQZS9Fa", "reviewer": "Reviewer_8oRv", "summary": "This paper proposes a new framework, called MOFIT, for performing membership inference attacks on latent diffusion models without ground-truth captions. The paper first studies the differences in how member and non-member samples respond to mismatched conditioning during the denoising process. Based on this observation, the authors propose an efficient method to improve MIA in the caption-free setting.", "questions": "Please see the weaknesses section", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The problem being studied is practical. Privacy risks in generative diffusion models are an important topic. The work addresses a realistic limitation in existing MIAs by removing the dependency on ground-truth captions, making it applicable to real-world scenarios.\n\n- Strong empirical performance: The results show consistent improvements over VLM-substituted baselines across multiple datasets and models.\n\n- Comprehensive analysis: The paper provides detailed analysis and discussion of different components.", "weaknesses": "- Despite arguing that the proposed method has a more practical setting, it still relies on internal model signals (e.g., likelihoods or denoising trajectories required by CLiD) that may not translate to black-box APIs. While common in MIA research, this reduces practicality in real-world applications.\n\n\n- The experiments focus on fine-tuned models on relatively small datasets (e.g., Pokémon with ~800 images), so generalization to large-scale, pre-trained models is unclear. It is important to conduct experiments on open-source pre-trained models with publicly available training sets. I am curious whether the proposed method still works on these models.\n\n\n- It is also important to provide experiment on computational overhead. Surrogate optimization in high-dimensional conditioning spaces can be expensive and resource-intensive, especially for large models. Reporting runtime and scaling behavior would help assess practicality.\n\n\n- Stability. The method relies on fixed noise bounds and timesteps for optimization, which might be sensitive to hyperparameters. The paper mentions using VLM initialization for stability but lacks ablation studies on failure cases or robustness to noise variations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iie0OKEthE", "reviewer": "Reviewer_edeD", "summary": "The authors are tackling the task of membership inference attack without access to ground truth captions. The idea is to find a way to distinguish between samples involved in the training of the generative diffusion model and those who don't. The problem become more feasible when not relying on ground truth captions since it is more applicable in real-life. The idea is to find cases of privacy violation of copyrights. The main conceptual idea of the method is that the difference between the conditional loss (conditioned on the caption, commonly referred as guidance-free generation), and the uncoditional one is at its highest on hold-out samples.", "questions": "* for $\\lambda$ optimization, in Eq. 5. The min is over all $\\lambda$s, when there is no $\\lambda$ in the formula itself. I assume that $z_t$ can be written as a function of $x_0$, it is somehow should be stated clearly, since minimizing on a variable which is not argument in the formula is confusing.\n* it is not stated explicitly based on which LDM table 2 was produced?\n\n\nI think that despite that caption-free methods were probably published earlier than this work, the paper is well written, and it seems novel to me. The paper flows nicely, there is a lot of elaboration on the intuition behind empirical decisions, which make it much more intuitive for the reader. The results of the paper are impressive, and I think that the authors cover relevant and timely topic with nice academic progress.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The problem is stated explicitly and in a very understandable and accessible manner.\n* The motivation of the method, and the intuition behind the structural choices is very clear and detailed, make it very accessible even for people which are less expert in this field.\n* The experimental part is very impressive, the results of MoFit are better with a large margin then the competitors.", "weaknesses": "* The authors claim to be the first to tackle the caption-free MIA technique on generative diffusion models whereas [1], which not cited, explicitly claim to be the first (and already published). The authors should mention it and differentiate themselves from it, and to tone down their primacy in the field if it was already done previously.\n* The evaluation time is super high - so the method is very slow and perhaps not feasible for set of images which is more than a few (7-9 minutes for a single image is reported in the Supp).\n* I didnt see (correct me if Im wrong) any evaluation on a wide range of diffusion processes. Im not talking about the same architecture trained on several different datasets. I think that if you show that your approach is architecture-agnostic, namely when a novel diffusion mechanism will arrive your approach will highly likely work on it as well.\n* I think it would be nice to analyze several fail cases, to better understand on which cases the method fails, even a short discussion on it (help follow-up works to understand the limitation of the method).\n\n\nrefs:\n\n[1] Towards Black-Box Membership Inference Attack for Diffusion Models. Li et al. ICML 25.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)