Conditional Overfitting Phenomenon
The rationale behind previous studies primarily hinges on the overfitting of diffusion models to training data (usually image data \mathbf{x} ) [ 32 , 11 , 23 ] . This overfitting tends to result in lower estimation errors for images in the member set (training data) compared to those in the hold-out set during the diffusion process. Various indicators [ 32 , 11 , 23 ] are designed based on this to expose membership information.
Specifically, let q_{\text{mem}} and q_{\text{out}} represent the image distributions of the member set and the hold-out set, respectively. p represents the diffusion’s estimated distribution, and D denotes a distance metric (which will be specified later). This rationale can be formulated as:

D(q_{mem}(\mathbf{x}),p(\mathbf{x}))\leq D(q_{out}(\mathbf{x}),p(\mathbf{x}))

However, if considering the membership inference on text-to-image diffusion models with image-text data (\mathbf{x},\mathbf{c}) , we emphasize the following assumption:

The overfitting of text-to-image diffusion models to the conditional distribution of (\mathbf{x},\mathbf{c}) is more salient than to the marginal distribution of \mathbf{x} :

\underbrace{\mathbb{E}_{\mathbf{c}}[D(q_{out}(\mathbf{x}|\mathbf{c}),p(\mathbf{x}|\mathbf{c}))-D(q_{mem}(\mathbf{x}|\mathbf{c}),p(\mathbf{x}|\mathbf{c}))]}_{overfitting to conditional distribution}\geq\underbrace{D(q_{out}(\mathbf{x}),p(\mathbf{x}))-D(q_{mem}(\mathbf{x}),p(\mathbf{x}))}_{overfitting to marginal distribution}

Empirically, we validate this assumption by using Fréchet Inception Distance (FID) [ 15 ] as the metric D , i.e., D_{FID} .
We calculate D_{FID}(q(\mathbf{x}|\mathbf{c}),p(\mathbf{x}|\mathbf{c})) using the MS-COCO [ 29 ] dataset on a fine-tuned Stable Diffusion [ 40 ] model. Then by gradually truncating the original condition text to \{2\big{/}3,1\big{/}3,Null\} to obtain \mathbf{c}^{*} , we calculate D_{FID}(q(\mathbf{x}|\mathbf{c}^{*}),p(\mathbf{x}|\mathbf{c}^{*})) as a stepwise approximation of D_{FID}(q(\mathbf{x}),p(\mathbf{x})) .
In Fig. 1 , we report the FID scores of synthetic images under different conditions of member set and hold-out set. A smaller FID value indicates a closer match between model distributions and dataset distributions.
From Fig. 1 (a), it can be observed that for the full condition, the FID difference between the member set and the hold-out set is consistently higher than that for the truncated conditions, which validates our assumptions.
We also demonstrate the validation utilizing other metrics in Appendix A .

We further compute the change in FID after truncating the condition and observe that the change in FID of the member set is consistently greater than that of the hold-out set (Fig. 1 (b)), which inspires revealing membership
by this overfitting discrepancy.
Recalling the aim of text-to-image diffusion model is to fit a latent space mapping from text to image,
image data augmentation is commonly used to enhance the model generalization.
For instance, the official fine-tuning script of Hugging-Face [ 19 ] employs Random-Crop and Random-Flip as the default augmentation [ 20 ] .
However, few trainers disturb the text condition as it is discrete and such disturbance would result in a decline of model utility ( Sec. 4.5 ).
Therefore, we believe that leveraging this phenomenon contributes to addressing the challenges of the strong generalization of text-to-image diffusion models with the resistance to data augmentation.