SAT-LDM: Provably Generalizable Image Watermarking for Latent Diffusion Models with Self-Augmented Training

Authors: Lu Zhang, Liang Zeng
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: image generation, watermarking, latent diffusion model

Abstract:
The proliferation of AI-generated images necessitates effective watermarking to protect intellectual property and identify fake content. While existing training-based watermarking methods show promise, they often struggle with generalization across diverse image styles and tend to produce noticeable artifacts. To this end, we introduce a provably generalizable image watermarking method for Latent Diffusion Models with Self-Augmented Training (SAT-LDM), which aligns the training and testing phases by a free generation distribution to bolster the watermarking module’s generalization capabilities. We theoretically consolidate our method by proving that the free generation distribution contributes to its tight generalization bound without the need to collect new data. Extensive experimental results demonstrate that SAT-LDM achieves robust watermarking while significantly improving the quality of watermarked images across diverse styles. Furthermore, we conduct experimental analyses to demonstrate the strong generalization abilities of SAT-LDM. We hope our method offers a practical and convenient solution for securing high-fidelity AI-generated content.

Decision: reject
Reviewer scores: 6, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1729640658861

thread_type:
official_review

<thread_object>
creator: reviewer_glwe

<content>
summary:
The paper observed a mismatch in the target image distribution to be watermarked in diffusion models with previous methods, i.e., external dataset (training phase) vs. generated dataset (testing phase). To address this, the authors propose improving the process by training the watermark using data generated by the diffusion model itself. This enhancement leads to improved performance in the authors' experimental settings.

strengths:
1. The observation that there is a mismatch between the training and testing phases is crucial and can greatly inform future research.
2. The theoretical section is informative.

weaknesses:
Below are the main concerns:

1. The biggest concern stems from the experimental setting. The authors claim that under 1,000 AI-generated prompts, the proposed method surpasses the baseline. However, this number of prompts is far too small for general case measurements, especially considering potential bias in the language models used. Additionally, this comparison seems unfair. The authors should provide a comparison of results using selected prompts from LAION-400M (but not used during training). The reported advantage may solely come from the prompt distribution shift between GPT-generated prompts and those from LAION-400M. Since the proposed method forgoes all prompt information during training (“using empty prompt”), it avoids this shift and may appear better (due to bias from GPT-generated prompts).
2. Another crucial drawback lies in the explanation of why the method works. The authors use generated data with an empty prompt and claim that this somehow represents the conditional distribution of diffusion models, showcasing a demo experiment in Sec. 5.3 Training Distributions. However, this demo experiment needs further verification. Fig. 3 can be interpreted as $d(z_{\text{laion-prompt}}, z_{\text{gpt-prompt}}) > d(z_{\text{no-prompt}}, z_{\text{gpt-prompt}})$, which does not provide any evidence that $d(z_{\text{real-prompt}}, z_{\text{no-prompt}})$ is small in most cases. In fact, it is widely believed that there is a large distance between the conditional and unconditional distributions in DMs, which is why we use (either classifier or classifier-free) guidance.
3. Another critical point is the "guidance" aspect. The authors need to show experimental results with guidance scale = 1, i.e., direct conditional generation (without incorporating an unconditional DM). I am concerned that the real mechanism behind the method's success is that the watermarking modules adapt to the negative output of the unconditional generation. Since in all cases with guidance scale > 1, the final output probability is inversely proportional to the unconditional distribution, this might be the actual reason for the results, rather than the claim that the method accurately reflects the generated distribution of diffusion models.
4. The paper claims two improvements: training data is shifted from a dataset to generated data, and the model structure is updated in Sec. 4.2. However, it is unclear which of these contributes to the observed improvements without an ablation study. There is a risk that the gains come from the structural update rather than the proposed training data change.

Here are some additional minor drawbacks, though they do not significantly impact my overall assessment of the paper:

1. While the intuition—using generated distribution instead of training data—is clear, the writing makes this idea more complicated than necessary. While formal theory is important, the explanation could be simplified to better emphasize the intuition, with theory introduced later.
2. In LaTeX, it would be better to use ``" for double quotes instead of "".

In summary, the method might work (under the specific experiment setting) only due to the fact that the prompt distance between the zero-prompt and GPT-generated ones is closer than the distance between LAION-400M prompts and GPT-generated ones. Given that the results are measured under GPT-generated prompts, it remains unclear how promising the method truly is. Additionally, the explanation for why the proposed method works based on the DM mechanism is insufficient.

Admittedly, the idea of using the generated distribution of DMs for watermarking is potentially promising. However, using the unconditional generated distribution seems less meaningful. It might be better to use conditional generated distributions with diverse prompts, such as those from LAION-5B.

questions:
The questions are connected to the points above, respectively:

1. Can the authors provide results using LAION-400M prompts during testing to ensure the performance improvement is more general?
2. Can the authors offer further insights regarding Fig. 3, such as whether there exists a good subset of LAION-400M prompts that generate images similar to those from GPT-generated prompts? From Fig. 3, there appears to be significant overlap between the External and Test regions. If so, this suggests the proposed testing scenarios may be biased, given that the external training set prompts are more diverse.
3. Could the authors provide additional results and discussion on guidance?
4. Could there be any ablation study, as previously mentioned, to differentiate between the contributions of the structural and training data improvements?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729685406482

thread_type:
official_review

<thread_object>
creator: reviewer_u5eu

<content>
summary:
The paper proposes an image watermarking scheme that generalizes across different image styles. The authors prove the generalization bound for the watermarked image generator and the message extractor and compare their method against several state-of-the-art approaches in terms of image quality and robustness to removal attacks. 

Please use this review sparingly.

strengths:
Authors provide a theoretical guarantee on the generalization error of their approach that, to my knowledge, has not been done in the previous works. Experimental evaluation demonstrates that the proposed method yields watermarked images of better quality than of the competitor's works. The robustness to removal attacks is on the level of state-of-the-art methods.

weaknesses:
The assumption on the Lipschitz continuity of the loss function is somewhat strong: 1) how can one check if it is true and 2) estimate the Lipschitz constant K (what can be quite large leading to unsatisfactory large upper bound)

questions:
I am willing to increase my score if authors provide a detailed explanation on the assumption of Lipschitz continuity of the loss function. How to verify an assumption? Does it hold in practice?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730233298941

thread_type:
official_review

<thread_object>
creator: reviewer_itqj

<content>
summary:
This paper introduces SAT-LDM, a watermarking method integrated within latent diffusion models (LDMs) to generate watermarked images. The authors argue that SAT-LDM improves generalization across image styles without compromising quality, unlike existing methods which reportedly degrade image content.

strengths:
The paper focuses on a significant topic—embedding watermarks in images generated by diffusion models like Stable Diffusion—to address the growing need for copyright protection of AI-generated content.

weaknesses:
- The paper's **motivation lacks clarity and appears misguided.**    
The authors claim that current diffusion-native watermarking methods degrade image quality by comparing watermarked images to unwatermarked counterparts generated from the same prompt. This comparison is irrelevant because users of diffusion-native watermarking methods would not see unwatermarked images beforehand. Such comparisons would only apply to post-watermarking methods, where preserving the original image content is necessary.    
Additionally, the claim that watermarking performance is compromised across image styles is questionable, as style diversity is largely dictated by the diffusion model's training data rather than the watermarking process itself.

- The paper’s Figure 2 depicts a VAE decoder as frozen while taking message embedding as input. If the authors follow the FSW structure, as stated in Section 4.2, **the figure is incorrect because FSW fuses message embeddings into the UNet-decoder, not the VAE decoder**. If the figure is correct, freezing the entire diffusion model (including the VAE) while training only the message processor and extractor introduces the risk of extracting false-positive messages from non-watermarked images (i.e., images generated without embedding). Thus, **the paper should include a false positive rate (FPR) analysis to evaluate this aspect**.

- The authors claim “no external data” usage in Figure 1, but they must have used training data for the two message components, just like the existing methods.

- The approach lacks innovation, as it mainly replicates FSW’s structure and uses the spatial transformer network from StegaStamp to improve robustness.

-  The experimental evaluation lacks critical ablations and comparisons. The paper should investigate the impact of using different pre-trained diffusion models and evaluate the method under strong attacks, such as diffusion-based attacks gauge robustness under adversarial conditions. WAVES [1] could be a reference in this case.

[1] An, Bang, et al. "WAVES: Benchmarking the Robustness of Image Watermarks." Forty-first International Conference on Machine Learning.

questions:
- What dataset did the authors use to train the message processor and extractor? How does this align with the claim of “no external data”?
- Given the potential for the message extractor to output messages from non-watermarked images, why is there no report on FPR?
- Does Figure 2 accurately represent the proposed structure? If so, how does a frozen VAE decoder process message embeddings, and what are the implications of only training the message processor and extractor?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707283842

thread_type:
official_review

<thread_object>
creator: reviewer_ufts

<content>
summary:
This paper studies how to watermark diffusion models across diverse image styles. The authors propose a training-based watermark method SAT-LDM. In particular, the authors plug a message processor into the VAE decoder to obtain watermarked images from latents. During the training, SAT-LDM jointly trains the message processor and message extractor. The diffusion model is fixed during the training. No external data is required for the training. Theoretical analysis is provided to demonstrate the generalization ability of the proposed method. Experiments show that SAT-LDM can generalize across different image styles.

strengths:
- The writing is clear and easy to follow.
- The method does not require additional training data.
- A theoretical guarantee is provided for the proposed method
- Experiments show that the method produces effective watermarks while maintaining high image fidelity.

weaknesses:
- More visualization results, especially of different image styles, can help demonstrate the generalization of the proposed SAT-LDM.

The authors addressed my concerns.
I agree with reviewer iTqJ that the novelty of the proposed method might be relatively limited.
Therefore I will maintain my score.

questions:
Please refer to the weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1732599665064

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We want to express our gratitude for the valuable comments and constructive feedbacks of all reviewers. We are encouraged that the advantages of our paper were generally appreciated by the reviewers. The reviewers thought *“The writing is clear and easy to follow. A theoretical guarantee is provided for the proposed method. Experiments show that the method produces effective watermarks while maintaining high image fidelity.”* (Reviewer UfTs), *“The paper focuses on a significant topic to address the growing need for copyright protection of AI-generated content.”* (Reviewer iTqJ), *“Authors provide a theoretical guarantee on the generalization error of their approach that, to my knowledge, has not been done in the previous works. The robustness to removal attacks is on the level of state-of-the-art methods.”* (Reviewer U5Eu), *“The observation that there is a mismatch between the training and testing phases is crucial and can greatly inform future research. The theoretical section is informative.”* (Reviewer gLwe).

We have carefully gone through each review and summarize several important questions commonly raised and address them in this thread. We have also uploaded a rebuttal version of the paper with the following revisions (all new texts and contents are in blue, note that this is not the final version):

1. **Additional Visual Comparisons**: We have included diverse visualization examples across different image styles in Appendix Section F.  
2. **New Experimental Analyses**:    
   - Extended experiments on watermark detection and user identification with larger datasets (e.g., 5,000 images).  
   - Results on different pretrained models.
   - Experimental analysis using LAION-400M as the test distribution.
   - Additional experiments conducted with a guidance scale of 1.
   - Evaluation of training performance when using LAION-400M captions as prompts.
3. **Discussion on Lipschitz Continuity of Loss Function**: We have included discussion on lipschitz continuity assumption of the loss function in Appendix Section G, to discuss its rationality and verifiability in practice.
</content>
</thread_object>

</thread>


Metareview:
metareview:
2x borderline accept, 2x borderline reject. This paper studies how to watermark diffusion models across diverse image styles by introducing a training-based watermark method that avoids external datasets and provides theoretical guarantees on generalization. The reviewers agree on the (1) clear writing and solid theoretical framing, (2) effectiveness of watermark embedding without reliance on external data, (3) promising performance on image fidelity and robustness, and (4) value of the self-augmented training approach. However, they note (1) limited novelty relative to existing diffusion-native watermarking methods, (2) incomplete analysis of false positives and distribution assumptions, (3) potential mismatch between unconditional and conditional generation in real use cases, and (4) insufficient ablations distinguishing the benefits of new training data from structural changes. The authors have followed up with additional experiments, clarifications on distribution gaps, and new results on FPR and guidance scales, yet several reviewers remain partially unconvinced, so the AC leans to not accept this submission.