Shallow Diffuse: Robust and Invisible Watermarking through Low-Dimensional Subspaces in Diffusion Models

Authors: Wenda Li, Huijie Zhang, Qing Qu
Primary area: generative models
Keywords: diffusion Model, watermark, low-dimensional subspace, consistency, robustness

Abstract:
The widespread use of AI-generated content from diffusion models has raised significant concerns regarding misinformation and copyright infringement. Watermarking is a crucial technique for identifying these AI-generated images and preventing their misuse. In this paper, we introduce *Shallow Diffuse*, a new watermarking technique that embeds robust and invisible watermarks into diffusion model outputs. Unlike existing approaches that integrate watermarking throughout the entire diffusion sampling process, *Shallow Diffuse* decouples these steps by leveraging the presence of a low-dimensional subspace in the image generation process. This method ensures that a substantial portion of the watermark lies in the null space of this subspace, effectively separating it from the image generation process. Our theoretical and empirical analyses show that this decoupling strategy greatly enhances the consistency of data generation and the detectability of the watermark. Extensive experiments further validate that our *Shallow Diffuse* outperforms existing watermarking methods in terms of robustness and consistency.

Decision: reject
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730275824275

thread_type:
official_review

<thread_object>
creator: reviewer_vsmj

<content>
summary:
Current watermarking techniques based on diffusion models often embed watermarks directly into the initial noise，which can alter the data distribution. This paper proposes "Shallow Diffuse," a method that disentangles the watermark embedding from the generation process by leveraging low-dimensional subspaces. This approach supports watermark embedding for both server-side and user-side applications while maintaining high robustness and consistency. Additionally, experiments were designed to validate robustness and conduct ablation studies across multiple datasets.

strengths:
1. The method utilizes the local linearity of low-dimensional subspaces. As a watermarking method based on diffusion models, it maintains the consistency of generated images.

2. This paper provides rigorous theoretical proof and presents a substantial number of computational formulas.

weaknesses:
1. The attack experiments are limited, consisting of only four fixed-parameter attacks, which do not demonstrate the method's robustness. For instance, the method can be viewed as a variant of treering, could experiments with additional attacks, such as rotation、regeneration be included?

2. The theoretical assumptions of the method are built upon [1], but the experimental results yield a different range of t values compared to the theoretical analysis in [1]. Although this can be explained by the errors introduced by DDIM-Inv, it remains perplexing.

3. The method relies on the properties of DDIM and DDIM-inverse, which may lack certain generalizability. It might not perform well for attacks executed in the latent space.

questions:
1. Shouldn't the formula $\hat{\boldsymbol{x}}_{0, t}:=\boldsymbol{f}_{\boldsymbol{\theta}, t}\left(\boldsymbol{x}_{t}+\lambda \Delta \boldsymbol{x}\right)$ on line 360 be  $\hat{\boldsymbol{x}}_{0, t}:=\boldsymbol{f}_{\boldsymbol{\theta}, t}\left(\boldsymbol{x}_{t}\right)$

2. Could you specify the parameters used in the Shallow Diffuse method in Section 5, such as the embedding channels and watermark radius?

3. The experiments in Appendix C only provide results, could you include some analysis?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730319947541

thread_type:
official_review

<thread_object>
creator: reviewer_a7ds

<content>
summary:
The paper proposed a new image watermarking method that embeds a watermark into the null space of a specific step during image denoising in diffusion model. It shows that the proposed watermarking method have smaller impact on the generated images compared to the existing methods like Stable Signature and Tree-Ring. Additionally, the proposed method shows good robustness to image processing methods like JPEG and Gaussian blur.

strengths:
1. The proposed method has smaller impact on the generated images compared to the existing watermarking methods designed for diffusion models.
2. Experiments are carried on several image-prompt datasets to show the effectiveness of the proposed methods.
3. The robustness of the propsoed method is evaluated.

weaknesses:
1. Compared to Tree-Ring, the technical contribution of the proposed method is limited.
2. In the experimental part, the authors mainly compare their method with the watermarking methods that embed watermark into the semantic space like Tree-Ring which changes the image a lot. More other watermarking methods should be evaluated.
3. In the robustness part, the authors only evaluate the robustness of the proposed method on some common perturbation.

questions:
1. Besides Tree-Ring and Stable Signature, I think there are more existing watermarking methods that introduce small perturbation to the image to embed watermark, like StegaStamp. To show the proposed method actually preserves image quality, I think the authors should compare the method with some watermarking methods that watermark the image after image generation with a small perturbation.
2. In the robustness part, I think the regeneration attacks and some adversarial perturbation should be evaluated on the proposed method to see whether the proposed method is actually robust under various attacks.
3. Since the authors mention the user scenario, if multiple users rewatermark the same image with the proposed method, can the watermark embeded by the specific user be detected in this circumstance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730613074324

thread_type:
official_review

<thread_object>
creator: reviewer_kjyg

<content>
summary:
This paper proposes a watermarking technique Shallow Diffuse. Unlike existing approaches that integrate watermarking throughout the entire diffusion sampling process,  Shallow Diffuse decouples these steps by leveraging the presence of a low-dimensional subspace in the image generation  process. This method ensures that a substantial portion of the watermark lies in the null space of this subspace,  effectively separating it from the image generation process.

strengths:
The originality of this paper is not great, but its quality, clarity and significance are good. It has the support of rich theoretical basis and has advantages in theoretical proof.

weaknesses:
The table in the paper is not very well drawn, it is very difficult to read, especially the header. At the same time, the experimental part is not detailed enough. For example, should the comparison method reproduce the results or use the pre-training model?

questions:
1. The table in the paper is very difficult to read clearly.

2. In Table 1, for the CLIP-Score index, yours is 0.3285, which seems to be the worst. Please explain further.

3. Please explain why the filter size in the Gaussian blurring is 8 × 8 and how the standard deviation is selected.

4. As can be seen from Table 2, the PSNR and SSIM of most methods are very low, so it is easy for human eyes to find modification traces, which easily leads to the risk of watermarked images being maliciously broken. Please further explain the visual quality of the generated watermarked image.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730626445280

thread_type:
official_review

<thread_object>
creator: reviewer_wrzj

<content>
summary:
This paper proposed Shallow Diffuse, a watermarking technique for diffusion models. The method is well-motivated and with proper theoretical justification. The proposed Shallow Diffuse has several key advantages compared to existing diffusion watermarks, 1) It is a training-free watermark but simultaneously maintains the consistency between watermarked and original images. 2) It is more robust than existing baselines, achieving nearly no performance drop under different robustness tests. Shallow Diffuse also considers two scenarios including the server (protect generated image) and user (protect existing image) scenarios for injecting the watermark.

strengths:
1. Shallow Diffuse's primary strength lies in its utilization of the low-rank property of the PMP's Jacobian matrix to minimize the visual impact of watermarks, thereby attaining visual consistency within a training-free watermark framework.
2. The injected watermark is more robust against several image distortions than existing baselines.

weaknesses:
1. The presentation of this paper is poor, for instance, the ablation studies (Appendix C) and index of experimental results (Table 4) are incomplete. Therefore, this leads to a shortage of critical ablation studies.
2. What is the performance of multi-key identification, specifically, is it possible for the Shallow Diffuse to inject multiple watermarks and distinguish between them?
3. The image distortions are less than that in previous studies, such as Tree-Ring, where they apply 6 distortions.
4. Can DiffPure purify the watermarked patterns?
5. The findings in Table 4 are confusing. It appears that employing channel averaging enhances robustness against image distortions. However, channel averaging involves averaging clean and watermarked images across specific channels. As per my understanding, this process might reduce watermark robustness. Can you explain this observation?

questions:
Please see the weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732282372677

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
To all reviewers and ACs

comment:
We sincerely thank all reviewers for their thorough evaluation of our work and for providing valuable and constructive feedback. We are encouraged by the positive remarks, including that our method is “well-motivated,” “significant,” and “effective” (WRZj, kjYg, a7Ds), as well as the recognition of our theoretical analysis as “proper,” “rich,” and “rigorous” (WRZj, kjYg, VSmJ).

**Summary of Our Contributions**:

In this work, we introduce Shallow Diffuse, a simple yet effective watermarking method that leverages the low-dimensional space inherent in the diffusion model generation process. By decoupling the sampling and watermarking steps, our approach achieves several notable advantages:

1.  **Flexibility**: To the best of our knowledge, Shallow Diffuse is the first training-free, diffusion-based watermarking method that can be efficiently applied in both user-side and server-side scenarios.
    
2.  **Consistency and Robustness**: Extensive experiments demonstrate that Shallow Diffuse consistently outperforms other diffusion-based methods in terms of both robustness and reproducibility.
    
3.  **Theoretical Foundations**: Unlike prior methods, our work provides theoretical bounds for both consistency and detectability, offering a solid foundation for the effectiveness of our approach.
    

 **Addressing reviewers’ major concerns**. We thank the reviewers for their feedback on our presentation and suggested experiments. In response, we have addressed key concerns, including adversarial attack evaluations, multi-key watermarking experiments, and presentation improvements, as detailed below. Reviewer-specific questions have also been addressed individually, with all changes highlighted in red in the revised paper.
___
> **Q1**: Additional adversarial attack evaluations

**A1**:
We have incorporated 7 more attack methods, including resize and restore, random drop, medium blurring, diffusion purification [1], VAE-based image compression models [2, 3], and stable diffusion-based image regeneration [4]. Detailed settings for these attacks are provided in Appendix C.1, while the experimental results are summarized in Tables 1–4. After taking these attacks into account, Shallow Diffuse is still one of the most robust methods in both the user and the server scenario.

___
> **Q2**: Experiments on multi-key watermarking.

**A2**:

We have designed two tasks for evaluating multi-key watermarking: multi-key identification and multi-key re-watermarking. 
-   **Multi-key identification:** This classification task tests the ability to identify individual watermarks among $N=2048$ keys, each with a distinct ring-shaped key $W_i$ for $i = 1, ..., N$. A random key is embedded into images, and after attacks, the task is to detect if the correct key is identified. The success rate serves as the evaluation metric. Results in Table 5 show that Shallow Diffuse outperforms Tree-Ring despite lacking a multi-key-specific design, while RingID achieves the highest success rate because it is specifically designed for multi-key identification. Future exploration of multi-key identification strategies is promising.
    
-   **Multi-key re-watermarking:** This task evaluates embedding and detecting multiple watermarks (tested with two) in the same image. Metrics include the average AUC and TPR@1%FPR over all watermarks. Results in Table 6 demonstrate Shallow Diffuse’s ability to handle multi-key re-watermarking problems, achieving 1.00 for most of the metrics.

Details of the experiment can be found in section C.2.

___
> **Q3**: Improved presentation in tables and appendix.

**A3**:

We have re-designed Tables 1, and 2 (layouts and captions) and split the detailed adversarial attack experiments in Tables 3, and 4. We have also added discussions about each ablation study in Appendix C. 

[1] Nie, Weili, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. "Diffusion models for adversarial purification." International Conference on Machine Learning (ICML 2022).

[2] Ballé, Johannes, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. "Variational image compression with a scale hyperprior." International Conference on Learning Representations  (ICLR 2018).

[3] Cheng, Zhengxue, Heming Sun, Masaru Takeuchi, and Jiro Katto. "Learned image compression with discretized gaussian mixture likelihoods and attention modules." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7939-7948. 2020.

[4] Zhao, Xuandong, Kexun Zhang, Yu-Xiang Wang, and Lei Li. "Generative autoencoders as watermark attackers: Analyses of vulnerabilities and threats." (2023).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732398085811

thread_type:
official_comment

<thread_object>
creator: area_chair_nenw

<content>
comment:
Dear Reviewers,
The authors have responded to your valuable comments.
Please take a look at them!

Best,
AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies diffusion-based digital watermarking to deal with the AI-generated content tracing problem. The proposed approach, ``Shallow Diffuse,'' tries to decouple both the watermarking and diffusion processes by leveraging the presence of a low-dimensional subspace in the image generation process. Both theoretical and empirical analyses were presented.

The authors claimed ``Consistency and Robustness: Extensive experiments demonstrate that Shallow Diffuse consistently outperforms other diffusion-based methods in terms of both robustness and reproducibility.'' in their response. Most reviewers are satisfied by the responses.
However, by checking Table 4, Shallow Diffuse presents robustness inferior to RingID under a vert limited set of selected attacks.
For multiple key identification presented in Table 5, ShallowDiffuse performs inferior to RingID.
In term of these watermarking requirements, it is hard to say that ShallowDiffuse advances the development of diffusion-based watermarking.
To surpass and to identify sufficient differences from the prior works, it is encouraged to conduct robustness evaluations under a broad range of attacks/distortions.