Shallow Diffuse: Robust and Invisible Watermarking through Low-Dimensional Subspaces in Diffusion Models

## Abstract

Abstract The widespread use of AI-generated content from diffusion models has raised significant concerns regarding misinformation and copyright infringement. Watermarking is a crucial technique for identifying these AI-generated images and preventing their misuse. In this paper, we introduce Shallow Diffuse , a new watermarking technique that embeds robust and invisible watermarks into diffusion model outputs. Unlike existing approaches that integrate watermarking throughout the entire diffusion sampling process, Shallow Diffuse decouples these steps by leveraging the presence of a low-dimensional subspace in the image generation process. This method ensures that a substantial portion of the watermark lies in the null space of this subspace, effectively separating it from the image generation process. Our theoretical and empirical analyses show that this decoupling strategy greatly enhances the consistency of data generation and the detectability of the watermark. Extensive experiments further validate that our Shallow Diffuse outperforms existing watermarking methods in terms of robustness and consistency. The codes will be released at https://github.com/liwd190019/Shallow-Diffuse .

## 1 Introduction

Diffusion models [, , ] have recently become a new dominant family of generative models, powering various commercial applications such as Stable Diffusion [, ], DALL-E [, ], Imagen [] Stable Audio [] and Sora []. These models have significantly advanced the capabilities of text-to-image, text-to-audio, text-to-video, and multi-modal generative tasks. However, the widespread usage of AI-generated content from commercial diffusion models on the Internet has raised several serious concerns: (a) AI-generated misinformation presents serious risks to societal stability by spreading unauthorized or harmful narratives on a large scale [, , ]; (b) the memorization of training data by those models [, , , , ] challenges the originality of the generated content and raises potential copyright infringement issues; (c) Iterative training on AI-generated content, known as model collapse [, , , , ] can degrade the quality and diversity of outputs over time, resulting in repetitive, biased, or low-quality generations that may reinforce misinformation and distortions in the wild Internet.

To deal with these challenges, watermarking is a crucial technique for identifying AI-generated content and mitigating its misuse. Typically, it can be applied in two main scenarios: (a) the server scenario: where given an initial random seed, the watermark is embedded to the image during the generation process; and (b) the user scenario: where given a generated image, the watermark is injected in a post-process manner; (as shown in the left two blocks in Figure 3). Traditional watermarking methods [, , , ] are mainly designed for the user scenario, embedding detectable watermarks directly into images with minimal modification. However, these methods are vulnerable to attacks. For example, watermarks can become undetectable with simple corruptions such as blurring on watermarked images. Learning-based methods [, , , ] enhance robustness in the user scenario but are inefficient at large scales. More recent methods considered the server scenario [, , , , ], where they improve robustness by integrating watermarking into the sampling process of diffusion models. For example, [, ] embed watermarks into initial random seeds in the Fourier domain and then sample images from the watermarked seeds. As illustrated in Figure 1, these approaches often lead to inconsistent watermarked images because they significantly alter the noise distribution away from Gaussian. Moreover, they require access to the initial random seed, limiting their use in the user scenario. To the best of our knowledge, there is currently no robust, consistent, and efficient watermarking method suitable for both the server and user scenarios (more detailed discussion about related works could be found in Section 6).

To address these limitations, we proposed Shallow Diffuse (^2^22The term Shallow has two meanings: (1) the watermark is embedded at an earlier, ’shallow’ timestep (i.e., $t=0.3T$) rather than the final timestep ($t=T$); (2) the watermark appears invisibly or ’shallowly’ within the watermarked image.), a robust, consistent, and efficient watermarking approach that can be employed for both the server and user scenarios. Unlike prior works [, ] that embed watermarks into the initial random seed and entangle the watermarking process with sampling, Shallow Diffuse decouples these two steps by leveraging the low-dimensional subspace in the generation process of diffusion models [, ]. The key insight is that, due to the low dimensionality of the subspace, a significant portion of the watermark will lie in the null space of this subspace, effectively separating the watermarking from the sampling process (see Figure 3 for an illustration). Our theoretical and empirical analyses demonstrate that this decoupling strategy significantly improves the consistency of the watermark. With better consistency as well as independence from the initial random seed, Shallow Diffuse is flexible for both server and user scenarios.

Our contributions. The proposed Shallow Diffuse offers several key advantages over existing watermarking techniques [, , , , , , , , ] that we highlight below:

- •
Flexibility. Watermarking via Shallow Diffuse works seamlessly under both server-side and user-side scenarios. In contrast, most of the previous methods only focus on one scenario without a straightforward extension to the other; see Table 1 and Table 2 for demonstrations.
- •
Efficiency. As a learning-free approach, Shallow Diffuse avoids additional optimization steps. Embedding watermarks at an earlier timestep reduces the time for both watermark injection and detection.
- •
Consistency and Robustness. By decoupling the watermarking from the sampling process, Shallow Diffuse achieves higher robustness and better consistency. Extensive experiments (Table 1 and Table 2 ) support our claims, with extra ablation studies in Figure 5(a) and Figure 5(b).
- •
Provable Guarantees. Unlike previous methods, the consistency and detectability of our approach are theoretically justified. Assuming a proper low-dimensional image data distribution (see 1), we rigorously establish bounds for consistency (1) and detectability (2).

Figure: Figure 1: Sampling variance of Tree-Ring Watermarks, RingID and Shallow Diffuse. On the left are the original images, and on the right are the corresponding watermarked images generated using three different techniques: Tree-Ring [], RingID [], and Shallow Diffuse. For each technique, we generate watermarks using two distinct random seeds, resulting in the respective watermarked images.
Refer to caption: extracted/5959320/figure/comparison.jpg

## 2 Preliminaries

We start by reviewing the basics of diffusion models [, , ], followed by several key empirical properties that will be used in our approach: the low-rankness and local linearity of the diffusion model [, ].

### 2.1 Preliminaries on diffusion models

#### Basics of diffusion models.

In general, diffusion models consist of two processes:

- •
*The forward diffusion process.* The forward process progressively perturbs the original data $\bm{x}_{0}$ to a noisy sample $\bm{x}_{t}$ for some integer $t\in[0,T]$ with $T\in\mathbb{Z}$. As in [], this can be characterized by a conditional Gaussian distribution $p_{t}(\bm{x}_{t}|\bm{x}_{0})=\mathcal{N}(\bm{x}_{t};\sqrt{\alpha_{t}}\bm{x}_{0
},(1-\alpha_{t})\textbf{I}_{d})$. Particularly, parameters $\{\alpha_{t}\}_{t=0}^{T}$ sastify: (*i*) $\alpha_{0}=1$, and thus $p_{0}=p_{\rm data}$, and (*ii*) $\alpha_{T}=0$, and thus $p_{T}=\mathcal{N}(\bm{0},\textbf{I}_{d})$.
- •
*The reverse sampling process.* To generate a new sample, previous works [, , , ] have proposed various methods to approximate the reverse process of diffusion models. Typically, these methods involve estimating the noise $\bm{\epsilon}_{t}$ and removing the estimated noise from $\bm{x}_{t}$ recursively to obtain an estimate of $\bm{x}_{0}$. Specifically, One sampling step of Denoising Diffusion Implicit Models (DDIM) [] from $\bm{x}_{t}$ to $\bm{x}_{t-1}$ can be described as:
$\bm{x}_{t-1}=\sqrt{\alpha_{t-1}}\underbrace{\left(\frac{\bm{x}_{t}-\sqrt{1-
\alpha_{t}}\bm{\epsilon}_{\bm{\theta}}(\bm{x}_{t},t)}{\sqrt{\alpha_{t}}}\right
)}_{\coloneqq\bm{f}_{\bm{\theta},t}(\bm{x}_{t})}+\sqrt{1-\alpha_{t-1}}\bm{
\epsilon}_{\bm{\theta}}(\bm{x}_{t},t),$
(1)
where $\bm{\epsilon}_{\bm{\theta}}(\bm{x}_{t},t)$ is parameterized by a neural network and trained to predict the noise $\bm{\epsilon}_{t}$ at time $t$. From previous works [, ], the first term in Equation 1, defined as $\bm{f}_{\bm{\theta},t}(\bm{x}_{t})$, is the *posterior mean predictor* (PMP) that predict the posterior mean $\mathbb{E}[\bm{x}_{0}|\bm{x}_{t}]$. DDIM could also be applied to a clean sample $\bm{x}_{0}$ and generate the corresponding noisy $\bm{x}_{t}$ at time $t$, named DDIM Inversion. One sampling step of DDIM inversion is similar to Equation 1, by mapping from $\bm{x}_{t-1}$ to $\bm{x}_{t}$. For any $t_{1}$ and $t_{2}$ with $t_{2}>t_{1}$, we denote multi-time steps DDIM operator and its inversion as $\bm{x}_{t_{1}}=\texttt{DDIM}(\bm{x}_{t_{2}},t_{1})\text{ and }\bm{x}_{t_{2}}=
\texttt{DDIM-Inv}(\bm{x}_{t_{1}},t_{2}).$

#### Text-to-image (T2I) diffusion models & classifier-free guidance (CFG).

The diffusion model can be generalized from unconditional to T2I [, ], where the latter enables controllable image generation $\bm{x}_{0}$ guided by a text prompt $\bm{c}$. In more detail, when training T2I diffusion models, we optimize a conditional denoising function $\bm{\epsilon}_{\bm{\theta}}(\bm{x}_{t},t,\bm{c})$. For sampling, we employ a technique called classifier-free guidance (CFG) [], which substitutes the unconditional denoiser $\bm{\epsilon}_{\bm{\theta}}(\bm{x}_{t},t)$ in Equation 1 with its conditional counterpart $\bm{\tilde{\epsilon}}_{\bm{\theta}}(\bm{x}_{t},t,\bm{c})$ that can be described as $\bm{\tilde{\epsilon}}_{\bm{\theta}}(\bm{x}_{t},t,\bm{c})=(1-\eta)\bm{\epsilon}
_{\bm{\theta}}(\bm{x}_{t},t,\bm{\varnothing})+\eta\bm{\epsilon}_{\bm{\theta}}(
\bm{x}_{t},t,\bm{c}).$ Here, $\bm{\varnothing}$ denotes the empty prompt and $\eta>0$ denotes the strength for the classifier-free guidance. For simplification, for any $t_{1}$ and $t_{2}$ with $t_{2}>t_{1}$, we denote multi-time steps CFG operator as $\bm{x}_{t_{1}}=\texttt{CFG}(\bm{x}_{t_{2}},t_{1},\bm{c}).$ DDIM and DDIM inversion could also be generalized to T2I version, denotes as $\bm{x}_{t_{1}}=\texttt{DDIM}(\bm{x}_{t_{2}},t_{1},\bm{c})\text{ and }\bm{x}_{t
_{2}}=\texttt{DDIM-Inv}(\bm{x}_{t_{1}},t_{2},\bm{c}).$

### 2.2 Local Linearity and Intrinsic Low-Dimensionality in PMP

In this work, we will leverage two key properties of the PMP $f_{\bm{\theta},t}(\bm{x}_{t})$ introduced in Equation 1 for watermarking diffusion models. Parts of these properties have been previously identified in recent papers [, , ], and they have been extensively studied across various datasets and diffusion model frameworks in [].
At one given timestep $t\in[0,T]$, let us consider the first-order Taylor expansion of the PMP $\bm{f}_{\bm{\theta},t}(\bm{x}_{t}+\lambda\Delta\bm{x})$ at the point $\bm{x}_{t}$:

$$ $\displaystyle\boxed{\quad\bm{l}_{\bm{\theta}}(\bm{x}_{t};\lambda\Delta\bm{x}) \;:=\;\bm{f}_{\bm{\theta},t}(\bm{x}_{t})+\lambda\bm{J}_{\bm{\theta},t}(\bm{x}_ {t})\cdot\Delta\bm{x},\quad}$ (2) $$

where $\Delta\bm{x}\in\mathbb{S}^{d-1}$ is a perturbation direction with unit length, $\lambda\in\mathbb{R}$ is the perturbation strength, and $\bm{J}_{\bm{\theta},t}(\bm{x}_{t})=\nabla_{\bm{x}_{t}}\bm{f}_{\bm{\theta},t}(
\bm{x}_{t})$ is the Jacobian of $\bm{f}_{\bm{\theta},t}(\bm{x}_{t})$. As shown in [], it has been found that within a certain range of noise levels, the learned PMP $\bm{f}_{\bm{\theta},t}$ exhibits local linearity, and its Jacobian $\bm{J}_{\bm{\theta},t}\in\mathbb{R}^{d\times d}$ is low rank:

- •
Low-rankness of the Jacobian $\bm{J}_{\bm{\theta},t}(\bm{x}_{t})$.
As shown in Figure 2(a) of [], the *rank ratio* for $t\in[0,T]$ *consistently* displays a U-shaped pattern across various network architectures and datasets: (*i*) it is close to $1$ near either the pure noise $t=T$ or the clean image $t=0$, (*ii*) $\bm{J}_{\bm{\theta},t}(\bm{x}_{t})$ is low-rank (i.e., the numerical rank ratio less than $10^{-1}$) for all diffusion models within the range $t\in[0.2T,0.7T]$, (*iii*) it achieves the lowest value around mid-to-late timestep, slightly differs on different architectures and datasets.
- •
Local linearity of the PMP $\bm{f}_{\bm{\theta},t}(\bm{x}_{t})$. As shown in Figure 2(b) of [], the mapping $\bm{f}_{\bm{\theta},t}(\bm{x}_{t})$ exhibits strong linearity across a large portion of the timesteps, which is consistently true among different architectures trained on different datasets. In particular, the work [] evaluated the linearity of $\bm{f}_{\bm{\theta},t}(\bm{x}_{t})$ at $t=0.7T$ where the rank ratio is close to the lowest value, showing that $\bm{f}_{\bm{\theta},t}(\bm{x}_{t}+\lambda\Delta\bm{x})\approx\bm{l}_{\bm{
\theta}}(\bm{x}_{t};\lambda\Delta\bm{x})$ even when $\lambda=40$,

## 3 Watermarking by Shallow Diffuse

In this section, we introduce Shallow Diffuse for watermarking diffusion models. Building on the benign properties of PMP discussed in Section 2.2, we explain how to inject and detect invisible watermarks in *unconditional* diffusion models in Section 3.1 and Section 3.2, respectively. Algorithm 1 outlines the overall watermarking method for unconditional diffusion models. In Section 3.3, we extend this approach to *text-to-image* diffusion models, illustrated in Figure 3.

Figure: Algorithm 1 Unconditional Shallow Diffuse

### 3.1 Injecting invisible watermarks

Consider an unconditional diffusion model $\bm{\epsilon}_{\bm{\theta}}(\bm{x}_{t},t)$ as we introduced in Section 2.1.
Instead of injecting the watermark $\Delta\bm{x}$ in the initial noise, we inject it in a particular timestep $t\in[0,T]$ with

$$ $\displaystyle\bm{x}^{\mathcal{W}}_{t}=\bm{x}_{t}+\lambda\Delta\bm{x},$ (3) $$

where $\lambda\in\mathbb{R}$ is the watermarking strength, $\bm{x}_{t}=\texttt{DDIM-Inv}\left(\bm{x}_{0},t\right)$ under the user scenario and $\bm{x}_{t}=\texttt{DDIM}\left(\bm{x}_{T},t\right)$ under the server scenario. Based upon Section 2.2, we choose the timestep $t$ so that the Jacobian of the PMP $\bm{J}_{\bm{\theta},t}(\bm{x}_{t})=\nabla_{\bm{x}_{t}}\bm{f}_{\bm{\theta},t}(
\bm{x}_{t})$ is *low-rank*. Moreover, based upon the linearity of PMP discussed in Section 2.2, we approximately have

$$ $\displaystyle\bm{f}_{\bm{\theta},t}(\bm{x}^{\mathcal{W}}_{t})\;=\;\bm{f}_{\bm{ \theta},t}(\bm{x}_{t})+\lambda\underset{\approx\bm{0}}{\bm{J}_{\bm{\theta},t}( \bm{x}_{t})\cdot\Delta\bm{x}}\;\approx\;\bm{f}_{\bm{\theta},t}(\bm{x}_{t})\;= \;\hat{\bm{x}}_{0,t},$ (4) $$

where we select the watermark $\Delta\bm{x}$ to span the entire space $\mathbb{R}^{d}$ *uniformly*; a more detailed discussion on the pattern design of $\Delta\bm{x}$ is provided in Section 3.2. The key intuition for Equation 4 to hold is that, when $r_{t}=\mathrm{rank}(\bm{J}_{\bm{\theta},t}(\bm{x}_{t}))\ll d$ is low, a significant proportion of $\lambda\Delta\bm{x}$ lies in the *null space* of $\bm{J}_{\bm{\theta},t}(\bm{x}_{t})$ so that $\bm{J}_{\bm{\theta},t}(\bm{x}_{t})\Delta\bm{x}\approx\bm{0}$.

Therefore, the selection of $t$ is based on ensuring that $\bm{f}_{\bm{\theta},t}(\bm{x}_{t})$ is locally linear and that the dimensionality of its Jacobian $r_{t}\ll d$. In practice, we choose $t=0.3T$ based on results from the ablation study in Section 5.4. As a results, the injection in Equation 4 maintains better consistency without changing the predicted $\bm{x}_{0}$. In the meanwhile, it is very robust because any attack on $\bm{x}_{0}$ would remain disentangled from the watermark, so that $\lambda\Delta\bm{x}$ remains detectable.

Although in practice we employ the DDIM method instead of PMP for sampling high-quality images, the above intuition still carries over to DDIM. From Equation 1, one step sampling of DDIM in terms of $\bm{f}_{\bm{\theta},t}(\bm{x}_{t})$ becomes:

$$ $\displaystyle\bm{x}_{t-1}$ $\displaystyle=\sqrt{\alpha_{t-1}}\underbrace{\bm{f}_{\bm{\theta},t}(\bm{x}_{t} )}_{\text{"predicted }\bm{x}_{0}"}+\frac{\sqrt{1-\alpha_{t-1}}}{\sqrt{1-\alpha _{t}}}\underbrace{\left(\bm{x}_{t}-\sqrt{\alpha_{t}}\bm{f}_{\bm{\theta},t}(\bm {x}_{t})\right)}_{"\text{the direction pointing to }\bm{x}_{t}"}.$ (5) $$

As explained in [], the first term predicts $\bm{x}_{0}$ while the second term points towards $\bm{x}_{t}$. When we inject the watermark $\Delta\bm{x}$ into $\bm{x}_{t}$ as given in Equation 3, we know that

$$ $\displaystyle\bm{x}^{\mathcal{W}}_{t-1}$ $\displaystyle=\sqrt{\alpha_{t-1}}\bm{f}_{\bm{\theta},t}(\bm{x}_{t}^{\mathcal{W }})+\frac{\sqrt{1-\alpha_{t-1}}}{\sqrt{1-\alpha_{t}}}\left(\bm{x}_{t}^{ \mathcal{W}}-\sqrt{\alpha_{t}}\bm{f}_{\bm{\theta},t}(\bm{x}_{t}^{\mathcal{W}})\right)$ $\displaystyle\approx\sqrt{\alpha_{t-1}}\bm{f}_{\bm{\theta},t}(\bm{x}_{t})+ \frac{\sqrt{1-\alpha_{t-1}}}{\sqrt{1-\alpha_{t}}}\left(\bm{x}_{t}+{\color[rgb] {0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\lambda\Delta\bm{x}}- \sqrt{\alpha_{t}}\bm{f}_{\bm{\theta},t}(\bm{x}_{t})\right),$ (6) $$

where the second approximation follows from Equation 4. This implies that the watermark $\lambda\Delta\bm{x}$ is embedded into the DDIM sampling process entirely through the second term of Equation 6 and it decouples from the first which predicts $\bm{x}_{0}$.
Therefore, similar to our analysis for PMP, the first term in (6) maintains the consistency of data generation, while the difference in second term highlighted by blue would be useful for detecting the watermark which we will discuss next. In Section 4, we provide more rigorous proofs validating the consistency and detectability of our approach.

### 3.2 Watermark Design and Detection

Second, building on the watermark injection method described in Section 3.1, we discuss the design of the watermark pattern and the techniques for effective detection.

#### Watermark pattern design.

Building on the method proposed by [], we inject the watermark in the frequency domain to enhance robustness against adversarial attacks. Specifically, we adapt this approach by defining a watermark $\lambda\Delta\bm{x}$ for the input $\bm{x}_{t}$ at timestep $t$ as follows:

$$ $\lambda\Delta\bm{x}\;:=\;\texttt{DFT-Inv}\left(\texttt{DFT}\left(\bm{x}_{t} \right)\odot\left(1-\bm{M}\right)+\bm{W}\odot\bm{M}\right)-\bm{x}_{t},$ (7) $$

where the Hadamard product $\odot$ denotes the element-wise multiplication. Additionally, we have the following for Equation 7:

- •
Transformation into the frequency domain. Let $\texttt{DFT}(\cdot)$ and $\texttt{DFT-Inv}(\cdot)$ represent the forward and inverse Discrete Fourier Transform (DFT) operators, respectively. As shown in Equation 7, we first apply $\texttt{DFT}(\cdot)$ to transform $\bm{x}_{t}$ into the frequency domain, where we then introduce the watermark via a mask. Finally, the modified input is transformed back into the pixel domain using $\texttt{DFT-Inv}(\cdot)$.
- •
The mask and key of watermarks. $\bm{M}$ is the mask used to apply the watermark in the frequency domain as shown in the top-left of Figure 2, and $\bm{W}$ denotes the key of the watermark. Typically, the mask $\bm{M}$ is circular, with the white area representing $1$ and the black area representing 0 0 in Figure 2, where we use it to modify specific frequency bands of the image. In the following, we discuss the design of $\bm{M}$ and $\bm{W}$ in detail.

Previous methods [, ] design the mask $\bm{M}$ to modify the low-frequency components of the initial noise input. While this approach works, as most of the energy in natural images is concentrated in the low-frequency range, it tends to distort the image when such watermarks are injected (see Figure 1 for an illustration). In contrast, as shown in Figure 2, we design the mask $\bm{M}$ to target the high-frequency components of the image. Since high-frequency components capture fine details where the energy is less concentrated on these bands, modifying them results in less distortion of the original image. This is especially true in our case because we are modifying $\bm{x}_{t}$, which is closer to $\bm{x}_{0}$, compared to the initial noise used in [, ].To modify the high-frequency components, we apply the DFT without shifting and centering the zero frequency, as illustrated in the bottom-left of Figure 2.

Figure: Figure 2: Visualization of Watermark Patterns. The top left shows the circular mask, $\bm{M}$, while the top right displays the key within the mask, $\bm{M}\odot\bm{W}$, where the key $\bm{W}$ consists of multiple rings, each with values sampled from a Gaussian distribution. The bottom images illustrate the low- and high-frequency regions applying DFT, both before and after centering the zero frequency.
Refer to caption: extracted/5959320/figure/watermark_pattern.jpg

In terms of designing the key $\bm{W}$, we follow []. The key $\bm{W}$ is composed of multi-rings and each ring has the same value that is drawn from Gaussian distribution; see the top-right of Figure 2 for an illustration. Further ablation studies on the choice of $\bm{M}$, $\bm{W}$, and the effects of selecting low-frequency or high-frequency regions for watermarking can be found in Table 3.

#### Watermark detection.

During watermark detection, suppose we are given a watermarked image $\bar{\bm{x}}^{\mathcal{W}}_{0}$ with certain corruptions, we apply the DDIM Inversion to recover the watermarked image at timestep $t$, denoted as $\bar{\bm{x}}^{\mathcal{W}}_{t}=\texttt{DDIM-Inv}\left(\bar{\bm{x}}^{\mathcal{W
}}_{0},t\right)$. To detect the watermark, following [, ], the $\texttt{Detector}(\cdot)$ in Algorithm 1 calculates the following p-value:

$$ $\eta=\frac{\texttt{sum}(\bm{M})\cdot||\bm{M}\odot\bm{W}-\bm{M}\odot\texttt{DFT }\left(\bar{\bm{x}}^{\mathcal{W}}_{t}\right)||_{F}^{2}}{||\bm{M}\odot\texttt{ DFT}\left(\bar{\bm{x}}^{\mathcal{W}}_{t}\right)||_{F}^{2}},$ (8) $$

where $\texttt{sum}(\cdot)$ is the summation of all elements of the matrix. Ideally, if $\bar{\bm{x}}^{\mathcal{W}}_{t}$ is a watermarked image, $\bm{M}\odot\bm{W}=\bm{M}\odot\texttt{DFT}\left(\bar{\bm{x}}^{\mathcal{W}}_{t}\right)$ and $\eta=0$. When $\bar{\bm{x}}^{\mathcal{W}}_{t}$ is a non-watermarked image, $\bm{M}\odot\bm{W}\neq\bm{M}\odot\texttt{DFT}\left(\bar{\bm{x}}^{\mathcal{W}}_{
t}\right)$ and $\eta>0$. By choosing a threshold $\eta_{0}$, non-watermarked images will have $\eta>\eta_{0}$ and watermarked images will have $\eta<\eta_{0}$. Theoretically, the derivation of the p-value $\eta$ could be found in [].

Figure: Figure 3: Overview of Shallow Diffuse for T2I Diffusion Models. The server scenario (top left) shows watermark embedding during generation using CFG, while the user scenario (bottom left) demonstrates post-generation watermark embedding with DDIM inversion. In both scenarios, the watermark is applied within a low-dimensional subspace (top right), where most of the watermark resides in the null space of $\bm{J}_{\bm{\theta},t}$ due to its low dimensionality. The channel average technique is then used to produce the final watermarked image. The adversarial detection (bottom right) highlights the watermark’s robustness, enabling the detector to retrieve the watermark even under adversarial attacks.
Refer to caption: extracted/5959320/figure/method.jpg

### 3.3 Extension to Text-to-Image (T2I) Diffusion Models

Up to this point, our discussion has focused exclusively on unconditional diffusion models. Next, we demonstrate how our approach can be readily extended to text-to-image (T2I) diffusion models, which are predominantly used in practice.

Figure 3 provides an overview of our method for T2I diffusion models, which can be flexibly applied to both server and user scenarios. Specifically,

- •
Watermark injection. Shallow Diffuse embeds watermarks into the noise corrupted image $\bm{x}_{t}$ at a specific timestep $t=0.3T$. In the server scenario, given $\bm{x}_{T}\sim\mathcal{N}(\bm{0},\bm{I}_{d})$ and prompt $\bm{c}$, we calculate $\bm{x}_{t}=\texttt{CFG}\left(\bm{x}_{T},t,\bm{c}\right)$. In the user scenario, given the generated image $\bm{x}_{0}$, we compute $\bm{x}_{t}=\texttt{DDIM-Inv}\left(\bm{x}_{0},t,\bm{\varnothing}\right)$, using an empty prompt $\bm{\varnothing}$. Next, similar to Section 3.1, we apply DDIM to obtain the watermarked image $\bm{x}^{\mathcal{W}}_{0}=\texttt{DDIM}\left(\bm{x}^{\mathcal{W}}_{t},0,\bm{
\varnothing}\right)$ and channel averaging $\bm{x}^{*\mathcal{W}}_{0}\leftarrow\texttt{ChannelAverage}\left(\bm{x}^{
\mathcal{W}}_{0},\texttt{DDIM}\left(\bm{x}_{t},0\right)\right)$. The detailed discussion about channel averaging is in Appendix A.
- •
Watermark detection. During watermark detection, suppose we are given a watermarked image $\bar{\bm{x}}^{\mathcal{W}}_{0}$ with certain corruptions, we apply the DDIM Inversion to recover the watermarked image at timestep $t$, denoted as $\bar{\bm{x}}^{\mathcal{W}}_{t}=\texttt{DDIM-Inv}\left(\bar{\bm{x}}^{\mathcal{W
}}_{0},t,\bm{\varnothing}\right)$. We detect the watermark $\Delta\bm{x}$ in $\bar{\bm{x}}^{\mathcal{W}}_{t}$ by calculating $\eta$ in Equation 8, with detail explained in Section 3.2.

## 4 Theoretical Justification

In this section, we provide theoretical justifications for the consistency and the detectability of Shallow Diffuse introduced in Section 3 for unconditional diffusion models. Proof could be found in Appendix C First, we make the following assumptions on the watermark and the diffusion model process.

###### Assumption 1 .

Suppose the following hold for the PMP $\bm{f}_{\bm{\theta},t}(\bm{x}_{t})$:

- •
Linearity: For any small $t$ and $\Delta\bm{x}\in\mathbb{S}^{d-1}$, we always have
$\bm{f}_{\bm{\theta},t}(\bm{x}_{t}+\lambda\Delta\bm{x})=\bm{f}_{\bm{\theta},t}(
\bm{x}_{t})+\lambda\bm{J}_{\bm{\theta},t}(\bm{x}_{t})\Delta\bm{x}.$
- •
$L$-Lipschitz continuous: we assume that $\bm{f}_{\bm{\theta},t}(\bm{x})$ is a $L$-Lipschiz continuous at every $t$:
$||\bm{J}_{\bm{\theta},t}(\bm{x})||_{2}\leq L,\quad\forall\bm{x}\in\mathbb{R}^{
d}.$

It should be noted that our assumptions are mild. The $L$-Lipschitz continuity is a common assumption for analysis. The approximated linearity have been shown in [] with the assumption of data distribution to be a mixture of low-rank Gaussians. Here, we assume the linearity to be exact for the ease of analysis, and it can be generalized to approximate linear case.

Now consider injecting a watermark $\lambda\Delta\bm{x}$ in Equation 3, where $\lambda>0$ is a scaling factor and $\Delta\bm{x}$ is a *random* vector uniformly distributed on the unit hypersphere $\mathbb{S}^{d-1}$, i.e., $\Delta\bm{x}\sim\mathrm{U}(\mathbb{S}^{d-1})$. Then the following hold for the PMP $\bm{f}_{\bm{\theta},t}(\bm{x}_{t})$.

###### Theorem 1 (Consistency of the watermarks) .

Suppose 1 holds and $\Delta\bm{x}\sim\mathrm{U}(\mathbb{S}^{d-1})$.
Let us define $\hat{\bm{x}}_{0,t}^{\mathcal{W}}\coloneqq\bm{f}_{\bm{\theta},t}(\bm{x}_{t}+
\lambda\Delta\bm{x})$, $\hat{\bm{x}}_{0,t}\coloneqq\bm{f}_{\bm{\theta},t}(\bm{x}_{t})$. The $\ell_{2}$-norm distance between $\hat{\bm{x}}_{0,t}^{\mathcal{W}}$ and $\hat{\bm{x}}_{0,t}$ can be bounded by:

$$ $||\hat{\bm{x}}_{0,t}^{\mathcal{W}}-\hat{\bm{x}}_{0,t}||_{2}\leq\lambda Lh(r_{t }),$ (9) $$

with probability at least $1-r_{t}^{-1}$. Here, $h(r_{t})=\sqrt{\dfrac{r_{t}}{d}+\sqrt{\dfrac{18\pi^{3}}{d-2}\textup{log}\left(
2r_{t}\right)}}$.

Our 1 guarantees that adding the watermark $\lambda\Delta\bm{x}$ would only change the estimation by an amount of $\lambda Lh(r_{t})$ with a constant probability. In particular, when $r_{t}$ is small, it implies that the change in the prediction would be small. Given the relationship between PMP and DDIM in (1), the consistency also applies to the practical use. On the other hand, in the following we show that the injected watermark can be detected based upon the second term in Equation 6.

###### Theorem 2 (Detectability of the watermarks) .

Suppose 1 holds and $\Delta\bm{x}\sim\mathrm{U}(\mathbb{S}^{d-1})$. With $\bm{x}^{\mathcal{W}}_{t}$ given in Equation 3, define $\bm{x}^{\mathcal{W}}_{t-1}\coloneqq\texttt{DDIM}\left(\bm{x}^{\mathcal{W}}_{t}
,t-1\right)$ and $\tilde{\bm{x}}^{\mathcal{W}}_{t}\coloneqq\texttt{DDIM-Inv}\left(\bm{x}^{
\mathcal{W}}_{t-1},t\right)$. The $\ell_{2}$-norm distance between $\tilde{\bm{x}}^{\mathcal{W}}_{t}$ and $\bm{x}^{\mathcal{W}}_{t}$ can be bounded by:

$$ $||\tilde{\bm{x}}^{\mathcal{W}}_{t}-\bm{x}^{\mathcal{W}}_{t}||_{2}\leq\lambda L \left(-g\left({\alpha_{t},\alpha_{t-1}}\right)+g\left({\alpha_{t-1},\alpha_{t} }\right)\left(1-Lg\left({\alpha_{t},\alpha_{t-1}}\right)\right)\right)h(\max\{ r_{t-1},r_{t}\})$ (10) $$

with probability at least $1-r^{-1}_{t}-r^{-1}_{t-1}$.
Here, $g(x,y)\coloneqq\dfrac{\sqrt{1-y}\sqrt{x}-\sqrt{1-x}\sqrt{y}}{\sqrt{1-x}}$, $\forall x,y\in(0,1)$.

Here $-g\left({\alpha_{t},\alpha_{t-1}}\right)+g\left({\alpha_{t-1},\alpha_{t}}
\right)\left(1-Lg\left({\alpha_{t},\alpha_{t-1}}\right)\right)$ is a small number under the $\alpha_{t}$ designed for variance preserving (VP) noise scheduler [] and $h(\max\{r_{t-1},r_{t}\})$ is small when $r_{t}$ is small. This indicates that the difference between $\tilde{\bm{x}}^{\mathcal{W}}_{t}$ and $\bm{x}^{\mathcal{W}}_{t}$ is small when $r_{t}$ is small and $\bm{x}^{\mathcal{W}}_{t}$ could be recovered by $\tilde{\bm{x}}^{\mathcal{W}}_{t}$ from one-step DDIM. Therefore, 2 implies that the injected watermark can be detected with constant probability.

###### Assumption 1 .

###### Theorem 1 (Consistency of the watermarks) .

###### Theorem 2 (Detectability of the watermarks) .

## 5 Experiments

In this section, we present comprehensive experiments to demonstrate the robustness and consistency of Shallow Diffuse across various datasets. We begin by highlighting its performance in terms of robustness and consistency in both the server scenario (Section 5.1) and the user scenario (Section 5.2). Additionally, we compare Shallow Diffuse with other related works in the trade-off between robustness and consistency, as detailed in Section 5.3. Moreover, we investigate the effect of timestep $t$ on both robustness and consistency, with results presented in Section 5.4. Lastly, we provide an ablation study on watermark pattern design, and channel averaging in Appendix B.

#### Baseline

For the server scenario, we select the following methods as baselines: DWtDct [], DwtDctSvd [], RivaGAN [], Stable Signature [], Tree-Ring Watermarks [], RingId [], and Gaussian Shading []. In the user scenario, we adopt the same baseline methods, except for Stable Signature and Gaussian Shading, as these methods are not suitable for this setting.

#### Datasets

We use Stable Diffusion 2.1 [] as the underlying model for our experiments, applying Shallow diffusion within its latent space. For the server scenario (Section 5.1), all diffusion-based methods are based on the same Stable Diffusion, with the original images $\bm{x}_{0}$ generated from identical initial seeds $\bm{x}_{T}$. Non-diffusion methods are applied to these same original images $\bm{x}_{0}$ in a post-watermarking process. A total of 5000 original images are generated for evaluation in this scenario. For the user scenario (Section 5.2), we utilize the MS-COCO [], WikiArt [], and DiffusionDB datasets []. The first two are real-world datasets, while DiffusionDB is a collection of diffusion model-generated images. From each dataset, we select 500 images for evaluation. For the remaining experiments in Section 5.3, Section 5.4, Appendix B, we use the server scenario and sample 100 images for evaluation.

#### Metric

To evaluate image consistency under the user scenario, we use peak signal-to-noise ratio (PSNR) [], structural similarity index measure (SSIM) [], and Learned Perceptual Image Patch Similarity (LPIPS) [], comparing watermarked images to their original counterparts. In the server scenario, we assess the generation quality of the watermarked images using Contrastive Language-Image Pretraining Score (CLIP-Score) [] and Fréchet Inception Distance (FID) []. To evaluate robustness, we vary the threshold $\eta_{0}$ and plot the true positive rate (TPR) against the false positive rate (FPR) for the receiver operating characteristic (ROC) curve. We use the area under the curve (AUC) and TPR when FPR = 0.01 (TPR @1% FPR) as robustness metrics. Robustness is evaluated both under clean conditions (no attacks) and with various attacks, including JPEG compression, Gaussian blurring (G.Blur), Gaussian noise (G.Noise), and color jitter. Details of these attacks are provided in Section B.1.

### 5.1 Consistency and robustness under the server scenario

**Table 1: Comparison of invisibility and robustness under the server scenario.**
| Method | CLIP-Score $\uparrow$ | FID $\downarrow$ | Watermarking Robustness (AUC $\uparrow$/TPR@1%FPR$\uparrow$) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Clean | JPEG | G.Blur | G.Noise | Color Jitter | Average |  |  |  |
| Non-diffusion Method |  |  |  |  |  |  |  |  |
| DwtDct | 0.3298 | 25.73 | 0.97/0.85 | 0.64/0.00 | 0.78/0.00 | 0.44/0.02 | 0.53/0.09 | 0.60/0.03 |
| DwtDctSvd | 0.3291 | 26.00 | 1.00/1.00 | 0.80/0.08 | 0.99/0.80 | 0.97/0.84 | 0.50/0.09 | 0.82/0.45 |
| RivaGAN | 0.3252 | 24.60 | 1.00/0.99 | 0.98/0.76 | 0.97/0.72 | 1.00/0.99 | 0.96/0.77 | 0.98/0.81 |
| Diffusion Method |  |  |  |  |  |  |  |  |
| Stable Diffusion w/o WM | 0.3286 | 25.56 | - | - | - | - | - | - |
| Stable Signature | 0.3622 | 30.86 | 1.00/1.00 | 0.99/0.76 | 0.57/0.00 | 0.71/0.14 | 0.96/0.87 | 0.81/0.46 |
| Tree-Ring Watermarks | 0.3310 | 25.82 | 1.00/1.00 | 0.99/0.97 | 0.98/0.98 | 0.94/0.50 | 0.96/0.67 | 0.97/0.80 |
| RingID | 0.3285 | 27.13 | 1.00/1.00 | 1.00/1.00 | 1.00/1.00 | 1.00/0.99 | 0.99/0.98 | 1.00/0.99 |
| Gaussian Shading | 0.3631 | 26.17 | 1.00/1.00 | 1.00/1.00 | 1.00/1.00 | 1.00/1.00 | 1.00/1.00 | 1.00/1.00 |
| Shallow Diffuse (ours) | 0.3285 | 25.58 | 1.00/1.00 | 1.00/1.00 | 1.00/1.00 | 1.00/1.00 | 1.00/1.00 | 1.00/1.00 |
|  |  |  |  |  |  |  |  |  |

Table 1 compares the performance of Shallow Diffuse with other methods in the user scenario. For reference, we also apply stable diffusion to generate images from the same random seeds, without adding watermarks (referred to as "Stable Diffusion w/o WM" in Table 1). In terms of generation quality, Shallow Diffuse achieves the best FID score among the diffusion-based methods. Additionally, the FID and CLIP scores of Shallow Diffuse are very close to those of Stable Diffusion w/o WM. This similarity arises because the watermarked distribution produced by Shallow Diffuse remains highly consistent with the original generation distribution. Regarding robustness, Shallow Diffuse outperforms all other methods. Although both Gaussian Shading and RingID exhibit comparable generation quality and robustness in the server scenario, they are less suitable for the user scenario. Specifically, Gaussian Shading embeds the watermark into $\bm{x}_{T}$, which is not accessible to the user, while RingID suffers from poor consistency, as demonstrated in Figure 1 and Table 2.

### 5.2 Consistency and robustness under the user scenario

**Table 2: Comparison of consistency and robustness under the user scenario.**
| Method | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | Watermarking Robustness (AUC $\uparrow$/TPR@1%FPR$\uparrow$) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Clean | JPEG | G.Blur | G.Noise | Color Jitter | Average |  |  |  |  |
| COCO |  |  |  |  |  |  |  |  |  |
| DwtDct | 37.88 | 0.97 | 0.02 | 0.98/0.83 | 0.48/0.02 | 0.50/0.00 | 0.30/0.06 | 0.57/0.16 | 0.46/0.06 |
| DwtDctSvd | 38.06 | 0.98 | 0.02 | 1.00/1.00 | 0.70/0.26 | 0.98/0.83 | 0.93/0.55 | 0.54/0.14 | 0.79/0.45 |
| RivaGAN | 40.57 | 0.98 | 0.04 | 1.00/1.00 | 1.00/1.00 | 0.99/0.86 | 1.00/0.99 | 0.97/0.83 | 0.99/0.92 |
| Stable Diffusion w/o WM | 32.28 | 0.78 | 0.06 | - | - | - | - | - | - |
| Tree-Ring Watermarks | 28.22 | 0.51 | 0.41 | 1.00/1.00 | 0.99/0.87 | 0.99/0.86 | 1.00/1.00 | 0.88/0.49 | 0.97/0.81 |
| RingID | 28.22 | 0.38 | 0.61 | 1.00/1.00 | 1.00/1.00 | 1.00/1.00 | 0.98/0.86 | 1.00/0.99 | 0.99/0.96 |
| Shallow Diffuse (ours) | 32.11 | 0.77 | 0.06 | 1.00/1.00 | 1.00/1.00 | 1.00/1.00 | 1.00/1.00 | 1.00/0.99 | 1.00/1.00 |
| DiffusionDB |  |  |  |  |  |  |  |  |  |
| DwtDct | 37.77 | 0.96 | 0.02 | 0.96/0.76 | 0.71/0.23 | 0.96/0.70 | 0.35/0.01 | 0.52/0.12 | 0.64/0.27 |
| DwtDctSvd | 37.84 | 0.97 | 0.02 | 1.00/1.00 | 0.71/0.23 | 0.53/0.00 | 0.93/0.59 | 0.50/0.09 | 0.72/0.23 |
| RivaGAN | 40.6 | 0.98 | 0.04 | 1.00/0.98 | 1.00/0.72 | 0.96/0.76 | 0.99/0.94 | 0.96/0.76 | 0.98/0.80 |
| Stable Diffusion w/o WM | 33.42 | 0.85 | 0.03 | - | - | - | - | - | - |
| Tree-Ring Watermarks | 28.3 | 0.62 | 0.29 | 1.00/1.00 | 0.99/0.68 | 0.94/0.62 | 1.00/1.00 | 0.84/0.15 | 0.94/0.61 |
| RingID | 27.9 | 0.21 | 0.77 | 1.00/1.00 | 1.00/1.00 | 1.00/1.00 | 0.98/0.86 | 1.00/0.98 | 0.99/0.96 |
| Shallow Diffuse (ours) | 33.07 | 0.84 | 0.04 | 1.00/1.00 | 1.00/0.99 | 1.00/0.99 | 1.00/1.00 | 1.00/1.00 | 1.00/0.99 |
| WikiArt |  |  |  |  |  |  |  |  |  |
| DwtDct | 38.84 | 0.97 | 0.02 | 0.96/0.75 | 0.44/0.00 | 0.51/0.01 | 0.26/0.00 | 0.49/0.12 | 0.43/0.03 |
| DwtDctSvd | 39.14 | 0.98 | 0.02 | 1.00/1.00 | 0.69/0.13 | 0.97/0.76 | 0.97/0.72 | 0.50/0.15 | 0.78/0.44 |
| RivaGAN | 40.44 | 0.98 | 0.05 | 1.00/1.00 | 0.97/0.81 | 1.00/0.95 | 1.00/1.00 | 0.90/0.65 | 0.97/0.85 |
| Stable Diffusion w/o WM | 31.6 | 0.7 | 0.09 | - | - | - | - | - | - |
| Tree-Ring Watermarks | 28.24 | 0.53 | 0.34 | 1.00/1.00 | 1.00/0.97 | 1.00/0.88 | 1.00/1.00 | 0.71/0.26 | 0.92/0.78 |
| RingID | 27.90 | 0.19 | 0.78 | 1.00/1.00 | 1.00/1.00 | 1.00/1.00 | 0.95/0.82 | 0.99/0.98 | 0.99/0.95 |
| Shallow Diffuse (ours) | 31.4 | 0.68 | 0.10 | 1.00/1.00 | 1.00/0.99 | 1.00/0.99 | 1.00/1.00 | 1.00/0.99 | 1.00/0.99 |
|  |  |  |  |  |  |  |  |  |  |

Table 2 presents a comparison of Shallow Diffuse’s performance against other methods in the user scenario. In terms of consistency, Shallow Diffuse outperforms all other diffusion-based approaches. To measure the upper bound of diffusion-based methods, we apply stable diffusion with $\bm{\hat{}}{x}_{0}=\texttt{DDIM}(\texttt{DDIM-Inv}(\bm{x}_{0},t,\varnothing),0
,\varnothing)$, and measure the data consistency between $\bm{\hat{}}{x}_{0}$ and $\bm{x}_{0}$ (denotes in Stable Diffusion w/o WM in Table 2). The upper bound is constrained by errors introduced through DDIM inversion, and Shallow Diffuse comes the closest to reaching this limit. For non-diffusion-based methods, which are not affected by DDIM inversion errors, better image consistency is achievable. As for the robustness, Shallow Diffuse outperforms all other methods in all three datasets. While RivaGAN achieves the best image consistency and comparable watermarking robustness to Shallow Diffuse in the user scenario, Shallow Diffuse is much more efficient. Unlike RivaGAN, which requires training for each individual image, Shallow Diffuse only involves the computational overhead of DDIM and DDIM inversion.

### 5.3 Trade-off between consistency and robustness

Figure: Figure 4: Trade-off between consistency and robustness for Tree-Ring Watermarks, RingID, and Shallow Diffuse.
Refer to caption: extracted/5959320/figure/robustness_consistency_tradeoff.jpg

Figure 4 illustrates the trade-off between consistency and robustness for Shallow Diffuse and other baselines. As the radius of $\bm{M}$ increases, the watermark intensity $\lambda$ also increases, reducing image consistency but improving robustness. By adjusting the radius of $\bm{M}$, we plot the trade-off using PSNR, SSIM, and LPIPS against TPR@1%FPR. From Figure 4, curve of Shallow Diffuse is consistently above the curve of Tree-Ring Watermarks and RingID, demonstrating Shallow Diffuse’s better consistency at the same level of robustness.

### 5.4 Relation between injecting timestep, consistency and robustness

Figure: (a) Consistency
Refer to caption: extracted/5959320/figure/low-rank-robust.jpg

Figure 5 shows the relationship between the watermark injection timestep $t$ and both consistency and robustness (^3^33In this experiment, we do not incorporate additional techniques like channel averaging or enhanced watermark patterns. Therefore, when $t=1.0T$, the method is equivalent to Tree-Ring Watermarks.). Shallow Diffuse achieves optimal consistency at $t=0.2T$ and optimal robustness at $t=0.3T$. In practice, we select $t=0.3T$. This result aligns with the intuitive idea proposed in Section 3.1 and the theoretical analysis in Section 4: low-dimensionality enhances both data generation consistency and watermark detection robustness. However, according to [], the optimal timestep $t$ for minimizing $r_{t}$ satisfies $t^{*}\in[0.5T,0.7T]$. We believe the best consistency and robustness are not achieved at $t^{*}$ due to the error introduced by DDIM-Inv. As $t$ increases, this error grows, leading to a decline in both consistency and robustness. Therefore, the best tradeoff is reached at $t\in[0.2T,0.3T]$, where $\bm{J}_{\bm{\theta},t}(\bm{x}_{t})$ remains low-rank but $t$ is still below $t^{*}$.

## 6 Discussion on Related Works

### 6.1 Image watermarking

Image watermarking has long been a crucial method for protecting intellectual property in computer vision [, , , ]. Traditional techniques primarily focus on user-side watermarking, where watermarks are embedded into images post-generation. These methods [, ] typically operate in the frequency domain to ensure the watermarks are imperceptible. However, such watermarks remain vulnerable to adversarial attacks and can become undetectable after applying simple image manipulations like blurring.

Some learning-based approaches for watermarking [, , , , , ] leveraged neural networks to embed watermarks. While these methods improved robustness and imperceptibility, they often suffer from high computational costs during fine-tuning and lack flexibility. Each new watermark requires additional fine-tuning or retraining, limiting their practicality.

More recently, diffusion model-based watermarking techniques have gained attraction due to their ability to seamlessly integrate watermarks during the generative process without incurring extra computational costs. Techniques such as [, , ] embed watermarks directly into the initial noise and retrieve the watermark by reversing the diffusion process. These methods enhance robustness and invisibility but are typically restricted to server-side watermarking, requiring access to the initial random seed. Moreover, the watermarks introduced by [, ] significantly alter the data distribution, leading to variance towards watermarks in generated outputs (as shown in Figure 1).

In contrast to [, ], our proposed shallow diffuse disentangles the watermark embedding from the generation process by leveraging the high-dimensional null space. This approach, both empirically and theoretically validated, significantly improves watermark consistency and robustness. To the best of our knowledge, this is the first method that supports watermark embedding for both server-side and user-side applications while maintaining high robustness and consistency.

### 6.2 Low-dimensional subspace in diffusion model

In recent years, there has been growing interest in understanding deep generative models through the lens of the manifold hypothesis [, , ]. This hypothesis suggests that high-dimensional real-world data actually lies in latent manifolds with a low intrinsic dimension. Focusing on diffusion models, [] empirically and theoretically shows that the approximated score function (the gradient of the log density of a noise-corrupted data distribution) in diffusion models is orthogonal to a low-dimensional subspace. Building on this, [, ] find that the estimated posterior mean from diffusion models lies within this low-dimensional space. Additionally, [] discovers strong local linearity within the space, suggesting that it can be locally approximated by a linear subspace. This observation motivates our 1, where we assume the estimated posterior mean lies in a low-dimensional subspace.

Building upon these findings, [, ] introduce a local intrinsic dimension estimator, while [] proposes a method for detecting out-of-domain data. [] offers theoretical insights into how diffusion model training transitions from memorization to generalization, and [, ] explores the semantic basis of the subspace to achieve disentangled image editing. Unlike these previous works, our approach leverages the low-dimensional subspace for watermarking, where both empirical and theoretical evidence demonstrates that this subspace enhances robustness and consistency.

## 7 Conclusion

We proposed Shallow Diffuse, a novel and flexible watermarking technique that operates seamlessly in both server-side and user-side scenarios. By decoupling the watermark from the sampling process, Shallow Diffuse achieves enhanced robustness and greater consistency. Our theoretical analysis demonstrates both the consistency and detectability of the watermarks. Extensive experiments further validate the superiority of Shallow Diffuse over existing approaches. For future directions, Shallow Diffuse could be extended to more modalities, e.g. video [, ] and more applications like inverse problem watermarking [, ].

## Acknowledgement

We acknowledge funding support from NSF CAREER CCF-2143904, NSF CCF-2212066, NSF CCF-
2212326, NSF IIS 2312842, NSF IIS 2402950, ONR N00014-22-1-2529, a gift grant from KLA, an
Amazon AWS AI Award, and MICDE Catalyst Grant.

## Appendix A Channel averaging

### A.1 Technique details

Figure: Figure 6: Illustration of channel average
Refer to caption: extracted/5959320/figure/channel_average.jpg

Natural images have multiple channels denoted by $C$. Instead of applying watermark $\lambda\Delta$ to all channels of $\bm{x}_{t}$, we can apply the watermark to a specific channel $c$ to make it even more invisible and robust. For this consideration, let us reshape the image $\bm{x}_{t}$ and the watermark $\Delta\bm{x}$ into the form $\bm{x}_{t}\in\mathbb{R}^{H\times W\times C},\lambda\Delta\bm{x}\in\mathbb{R}^{
H\times W\times C}$, where $H$, $W$, and $C$ represent the height, width, and channel dimensions for the image, respectively. These dimensions satisfy $HWC=d$.

Denote $[\bm{x}_{t}]_{i}\in\mathbb{R}^{H\times W}$ as the $i$th channel of $\bm{x}_{t}$, with $i\in[C]$. Thus $[\bm{x}^{\mathcal{W}}_{t}]_{c}=[\bm{x}_{t}]_{c}+[\lambda\Delta\bm{x}]_{c}$ and $[\bm{x}^{\mathcal{W}}_{t}]_{i}=[\bm{x}_{t}]_{i}$ for $i\neq c$. For the watermark in Equation 3, the channel averaging is defined as:

$$ $\displaystyle[\bm{x}^{*\mathcal{W}}_{0}]_{i}$ $\displaystyle=\texttt{ChannelAverage}\left(\bm{x}^{\mathcal{W}}_{0},\bm{x}^{*} _{0}\right),$ (11) $\displaystyle=\begin{cases}&[\bm{x}^{\mathcal{W}}_{0}]_{i},i=c\\ &(1-\gamma)[\bm{x}^{\mathcal{W}}_{0}]_{i}+\gamma[\bm{x}^{*}_{0}]_{i},i\neq c\\ \end{cases},$ (12) $$

where we applied $\gamma=1$. In our experiments, we found that we can increase both imperceptibility and robustness by further employing this simple approach. See our ablation study in Section B.3 for a more detailed analysis.

## Appendix B Additional Experiments

### B.1 Details about attacks

In this work, we intensively tested our method on four different watermarking attacks, both in the server scenario and in the user scenario. These watermarking attacks represent the most common image distortion methods in real life, including

- •
JPEG compression with a compression rate of 25%.
- •
Gaussian blurring (G.Blur) with an $8\times 8$ filter size.
- •
Gaussian noise (G.Noise) with $\sigma=0.1$.
- •
Color jitter with brightness factor uniformly ranges between 0 and 6.

### B.2 ablation study on different watermark patterns

In Table 3, we examine various combinations of watermark patterns $\bm{M}\odot\bm{W}$. For the shape of the mask $\bm{M}$, "Circle" refers to a circular mask $\bm{M}$ (see Figure 2 top left), while "Ring" represents a ring-shaped $\bm{M}$. Since the mask is centered in the middle of the figure, "Low" and "High" denote frequency regions: "Low" represents a DFT with zero-frequency centering, whereas "High" indicates a DFT without zero-frequency centering, as illustrated in Figure 2 bottom. For the distribution of $\bm{W}$, "Zero" implies all values are zero, "Rand" denotes values sampled from $\mathcal{N}(\bm{0},\bm{1})$, and "Rotational Rand" represents multiple concentric rings in $\bm{W}$, with each ring’s values sampled from $\mathcal{N}(\bm{0},\bm{1})$.

As shown in Table 3, watermarking in high-frequency regions (Rows 7-9) yields improved image consistency compared to low-frequency regions (Rows 1-6). Additionally, the "Circle" $\bm{M}$ combined with "Rotational Rand" $\bm{W}$ (Rows 3 and 9) demonstrates greater robustness than other watermark patterns. Consequently, Shallow Diffuse employs the "Circle" $\bm{M}$ with "Rotational Rand" $\bm{W}$ in the high-frequency region.

**Table 3: ablation study on different watermark patterns.**
| Method & Dataset | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | Average Watermarking Robustness (AUC $\uparrow$/TPR@1%FPR$\uparrow$) |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Frequency Region | Shape | Distribution |  |  |  |  |
| Low | Circle | Zero | 29.10 | 0.90 | 0.06 | 0.93/0.65 |
| Low | Circle | Rand | 29.37 | 0.92 | 0.05 | 0.92/0.25 |
| Low | Circle | Rotational Rand | 29.13 | 0.90 | 0.06 | 1.00/1.00 |
| Low | Ring | Zero | 36.20 | 0.95 | 0.02 | 0.78/0.35 |
| Low | Ring | Rand | 38.23 | 0.97 | 0.01 | 0.87/0.49 |
| Low | Ring | Rotational Rand | 35.23 | 0.93 | 0.02 | 0.99/0.98 |
| High | Circle | Zero | 38.3 | 0.96 | 0.01 | 0.80/0.34 |
| High | Circle | Rand | 42.3 | 0.98 | 0.004 | 0.86/0.35 |
| High | Circle | Rotational Rand | 38.0 | 0.94 | 0.01 | 1.00/1.00 |
|  |  |  |  |  |  |  |

### B.3 ablation study on channel average

We evaluate Shallow Diffuse with channel averaging enabled ($\gamma=1.0$) and disabled ($\gamma=0.0$), as shown in Table 4. Applying channel averaging slightly enhances robustness against certain attacks while maintaining comparable consistency. Therefore, we set $\gamma=1.0$ for Shallow Diffuse.

**Table 4: ablation study on channel average.**
| Channel average intensity $\gamma$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | Watermarking Robustness (TPR@1%FPR$\uparrow$) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Clean | JPEG | G.Blur | G.Noise | Color Jitter |  |  |  |  |
| 0 | 37.1103 | 0.941 | 0.0154 | 1.0000 | 1.0000 | 0.9971 | 1.0000 | 0.9584 |
| 1.0 | 36.6352 | 0.931 | 0.0151 | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 1.0000 |
|  |  |  |  |  |  |  |  |  |

### B.4 ablation study on watermarking embedded channel.

As shown in Table 5, we evaluate specific embedding channels $c$ for Shallow Diffuse, where "0," "1," "2," and "3" denote $c=0,1,2,3$, respectively, and "0 + 1 + 2 + 3" indicates watermarking applied across all channels (^4^44Here we apply Shallow Diffuse on the latent space of Stable Diffusion, the channel dimension is 4.). Since applying watermarking to any single channel yields similar results (Row 1-4), but applying it to all channels (Row 5) negatively impacts image consistency and robustness, we set $c=3$ for Shallow Diffuse.

**Table 5: ablation study on watermarking embedded channel.**
| Watermark embedding channel | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | Watermarking Robustness (TPR@1%FPR$\uparrow$) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Clean | JPEG | G.Blur | G.Noise | Color Jitter |  |  |  |  |
| 0 | 36.46 | 0.93 | 0.02 | 1.00 | 1.00 | 1.00 | 1.00 | 0.99 |
| 1 | 36.57 | 0.93 | 0.02 | 1.00 | 1.00 | 1.00 | 1.00 | 0.99 |
| 2 | 36.13 | 0.92 | 0.02 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 3 | 36.64 | 0.93 | 0.02 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 0 + 1 + 2 + 3 | 33.19 | 0.83 | 0.05 | 1.00 | 1.00 | 1.00 | 1.00 | 0.95 |
|  |  |  |  |  |  |  |  |  |

## Appendix C Proofs in Section 4

### C.1 Proofs of Theorem 1

###### Proof of 1 .

According to 1, we have $||\hat{\bm{x}}_{0,t}^{\mathcal{W}}-\hat{\bm{x}}_{0,t}||^{2}_{2}=\lambda||\bm{J
}_{\bm{\theta},t}(\bm{x}_{t})\cdot\Delta\bm{x}||_{2}^{2}$. From Levy’s Lemma proposed in [], given function $||\bm{J}_{\bm{\theta},t}(\bm{x}_{t})\cdot\Delta\bm{x}||_{2}^{2}:\mathbb{S}^{d-
1}\rightarrow\mathbb{R}$ we have:

$$ $\mathbb{P}\left(\left|||\bm{J}_{\bm{\theta},t}(\bm{x}_{t})\cdot\Delta\bm{x}||_ {2}^{2}-\mathbb{E}\left[||\bm{J}_{\bm{\theta},t}(\bm{x}_{t})\cdot\Delta\bm{x}| |_{2}^{2}\right]\right|\geq\epsilon\right)\leq 2\exp\left(\frac{-C(d-2) \epsilon^{2}}{L^{2}}\right),$ $$

given $L$ to be the Lipschitz constant of $||\bm{J}_{\bm{\theta},t}(\bm{x}_{t})||_{2}^{2}$ and $C$ is a positive constant (which can be taken to be $C=(18\pi^{3})^{-1}$). From Lemma 2 and Lemma 3, we have:

$$ $\mathbb{P}\left(\left|||\bm{J}_{\bm{\theta},t}(\bm{x}_{t})\cdot\Delta\bm{x}||_ {2}^{2}-\frac{||\bm{J}_{\bm{\theta},t}(\bm{x}_{t})||^{2}_{F}}{d}\right|\geq \epsilon\right)\leq 2\exp\left(\frac{-(18\pi^{3})^{-1}(d-2)\epsilon^{2}}{||\bm {J}_{\bm{\theta},t}(\bm{x}_{t})||^{4}_{2}}\right).$ $$

Define $\dfrac{1}{r_{t}}$ as the desired probability level, set

$$ $\frac{1}{r_{t}}=2\exp\left(\dfrac{-(18\pi^{3})^{-1}(d-2)\epsilon^{2}}{||\bm{J} _{\bm{\theta},t}(\bm{x}_{t})||^{4}_{2}}\right),$ $$

Solving for $\epsilon$:

$$ $\epsilon=||\bm{J}_{\bm{\theta},t}(\bm{x}_{t})||^{2}_{2}\sqrt{\frac{18\pi^{3}}{ d-2}\log\left(2r_{t}\right)}.$ $$

Therefore, with probability $1-\dfrac{1}{r_{t}}$, we have:

$$ $\displaystyle||\hat{\bm{x}}_{0,t}^{\mathcal{W}}-\hat{\bm{x}}_{0,t}||^{2}_{2}$ $\displaystyle=\lambda^{2}||\bm{J}_{\bm{\theta},t}(\bm{x}_{t})\cdot\Delta\bm{x} ||_{2}^{2},$ $\displaystyle\leq\frac{\lambda^{2}||\bm{J}_{\bm{\theta},t}(\bm{x}_{t})||^{2}_{ F}}{d}+\lambda^{2}||\bm{J}_{\bm{\theta},t}(\bm{x}_{t})||^{2}_{2}\sqrt{\frac{18 \pi^{3}}{d-2}\log\left(2r_{t}\right)},$ $\displaystyle\leq\lambda^{2}||\bm{J}_{\bm{\theta},t}(\bm{x}_{t})||^{2}_{2} \left(\frac{r_{t}}{d}+\sqrt{\frac{18\pi^{3}}{d-2}\log\left(2r_{t}\right)} \right),$ $\displaystyle=\lambda^{2}L^{2}\left(\frac{r_{t}}{d}+\sqrt{\frac{18\pi^{3}}{d-2 }\log\left(2r_{t}\right)}\right),$ $$

where the last inequality is obtained from $||\bm{J}_{\bm{\theta},t}(\bm{x}_{t})||^{2}_{F}\leq r_{t}||\bm{J}_{\bm{\theta},
t}(\bm{x}_{t})||^{2}_{2}$. Therefore, with probability $1-\dfrac{1}{r_{t}}$,

$$ $||\hat{\bm{x}}_{0,t}^{\mathcal{W}}-\hat{\bm{x}}_{0,t}||_{2}\leq\lambda L\sqrt{ \frac{r_{t}}{d}+\sqrt{\frac{18\pi^{3}}{d-2}\log\left(2r_{t}\right)}}=\lambda Lh (r_{t}).$ $$

∎

###### Proof of 2 .

According to Equation 1, one step of DDIM sampling at timestep $t$ could be represented by PMP $\bm{f}_{\bm{\theta},t}(\bm{x}_{t})$ as:

$$ $\displaystyle\bm{x}_{t-1}$ $\displaystyle=\sqrt{\alpha_{t-1}}\bm{f}_{\bm{\theta},t}(\bm{x}_{t})+\sqrt{1- \alpha_{t-1}}\left(\frac{\bm{x}_{t}-\sqrt{\alpha_{t}}\bm{f}_{\bm{\theta},t}( \bm{x}_{t})}{\sqrt{1-\alpha_{t}}}\right),$ (13) $\displaystyle=\sqrt{\frac{1-\alpha_{t-1}}{1-\alpha_{t}}}\bm{x}_{t}+\frac{\sqrt {1-\alpha_{t}}\sqrt{\alpha_{t-1}}-\sqrt{1-\alpha_{t-1}}\sqrt{\alpha_{t}}}{ \sqrt{1-\alpha_{t}}}\bm{f}_{\bm{\theta},t}(\bm{x}_{t}),$ (14) $$

If we inject a watermark $\lambda\Delta\bm{x}$ to $\bm{x}_{t}$, so $x^{\mathcal{W}}_{t}=\bm{x}_{t}+\lambda\Delta\bm{x}$. To solve $x^{\mathcal{W}}_{t-1}$, we could plugging Equation 2 to Equation 14, we could obtain:

$$ $\displaystyle\bm{x}^{\mathcal{W}}_{t-1}$ $\displaystyle=\sqrt{\frac{1-\alpha_{t-1}}{1-\alpha_{t}}}\bm{x}^{\mathcal{W}}_{ t}+\frac{\sqrt{1-\alpha_{t}}\sqrt{\alpha_{t-1}}-\sqrt{1-\alpha_{t-1}}\sqrt{ \alpha_{t}}}{\sqrt{1-\alpha_{t}}}\bm{f}_{\bm{\theta},t}(\bm{x}^{\mathcal{W}}_{ t}),$ (15) $\displaystyle=\bm{x}_{t-1}+\sqrt{\frac{1-\alpha_{t-1}}{1-\alpha_{t}}}\lambda \Delta\bm{x}+\frac{\sqrt{1-\alpha_{t}}\sqrt{\alpha_{t-1}}-\sqrt{1-\alpha_{t-1} }\sqrt{\alpha_{t}}}{\sqrt{1-\alpha_{t}}}\bm{J}_{\bm{\theta},t}(\bm{x}_{t}) \Delta\bm{x}$ (16) $\displaystyle=\bm{x}_{t-1}+\lambda\underbrace{\left(\sqrt{\frac{1-\alpha_{t-1} }{1-\alpha_{t}}}\bm{I}+\frac{\sqrt{1-\alpha_{t}}\sqrt{\alpha_{t-1}}-\sqrt{1- \alpha_{t-1}}\sqrt{\alpha_{t}}}{\sqrt{1-\alpha_{t}}}\bm{J}_{\bm{\theta},t}(\bm {x}_{t})\right)}_{\coloneqq\bm{W}_{t}}\Delta\bm{x},$ (17) $$

One step DDIM Inverse sampling at timestep $t-1$ could be represented by PMP $\bm{f}_{\bm{\theta},t}(\bm{x}_{t})$ as:

$$ $\displaystyle\bm{x}_{t}$ $\displaystyle=\sqrt{\frac{1-\alpha_{t}}{1-\alpha_{t-1}}}\bm{x}_{t-1}+\frac{ \sqrt{1-\alpha_{t-1}}\sqrt{\alpha_{t}}-\sqrt{1-\alpha_{t}}\sqrt{\alpha_{t-1}}} {\sqrt{1-\alpha_{t-1}}}\bm{f}_{\bm{\theta},t-1}(\bm{x}_{t-1}),$ (18) $$

To detect the watermark, we apply one step DDIM Inverse on $\bm{x}^{\mathcal{W}}_{t-1}$ at timestep $t-1$ to obtain $\bm{\tilde{}}{x}^{\mathcal{W}}_{t}$:

$$ $\displaystyle\bm{\tilde{}}{x}^{\mathcal{W}}_{t}$ $\displaystyle=\sqrt{\frac{1-\alpha_{t}}{1-\alpha_{t-1}}}\bm{x}^{\mathcal{W}}_{ t-1}+\frac{\sqrt{1-\alpha_{t-1}}\sqrt{\alpha_{t}}-\sqrt{1-\alpha_{t}}\sqrt{ \alpha_{t-1}}}{\sqrt{1-\alpha_{t-1}}}\bm{f}_{\bm{\theta},t-1}(\bm{x}^{\mathcal {W}}_{t-1}),$ $\displaystyle=\bm{x}_{t}+\lambda\underbrace{\left(\sqrt{\frac{1-\alpha_{t}}{1- \alpha_{t-1}}}\bm{I}+\frac{\sqrt{1-\alpha_{t-1}}\sqrt{\alpha_{t}}-\sqrt{1- \alpha_{t}}\sqrt{\alpha_{t-1}}}{\sqrt{1-\alpha_{t-1}}}\bm{J}_{\bm{\theta},t-1} (\bm{x}_{t-1})\right)}_{\coloneqq\bm{W}_{t-1}}\bm{W}_{t}\Delta\bm{x},$ $\displaystyle=\bm{x}_{t}+\lambda\bm{W}_{t-1}\bm{W}_{t}\Delta\bm{x}=\bm{x}^{ \mathcal{W}}_{t}+\lambda\left(\bm{W}_{t-1}\bm{W}_{t}-\bm{I}\right)\Delta\bm{x}.$ $$

Therefore:

$$ $\displaystyle||\bm{\tilde{}}{x}^{\mathcal{W}}_{t}-\bm{x}^{\mathcal{W}}_{t}||_{2}$ $\displaystyle=\lambda||\left(\bm{W}_{t-1}\bm{W}_{t}-\bm{I}\right)\Delta\bm{x}| |_{2},$ $\displaystyle=\lambda||\frac{\sqrt{1-\alpha_{t-1}}\sqrt{\alpha_{t}}-\sqrt{1- \alpha_{t}}\sqrt{\alpha_{t-1}}}{\sqrt{1-\alpha_{t}}}\bm{J}_{\bm{\theta},t-1}( \bm{x}_{t-1})\Delta\bm{x},$ $\displaystyle\ \ \ \ \ \ \ +\frac{\sqrt{1-\alpha_{t}}\sqrt{\alpha_{t-1}}-\sqrt {1-\alpha_{t-1}}\sqrt{\alpha_{t}}}{\sqrt{1-\alpha_{t-1}}}\bm{J}_{\bm{\theta},t }(\bm{x}_{t})\Delta\bm{x},$ $\displaystyle\ \ \ \ \ \ \ -\frac{\left(\sqrt{1-\alpha_{t}}\sqrt{\alpha_{t-1}} -\sqrt{1-\alpha_{t-1}}\sqrt{\alpha_{t}}\right)^{2}}{\sqrt{1-\alpha_{t-1}}\sqrt {1-\alpha_{t}}}\bm{J}_{\bm{\theta},t-1}(\bm{x}_{t-1})\bm{J}_{\bm{\theta},t}( \bm{x}_{t})\Delta\bm{x}||_{2},$ $\displaystyle\leq-\lambda g\left({\alpha_{t},\alpha_{t-1}}\right)||\bm{J}_{\bm {\theta},t-1}(\bm{x}_{t-1})\Delta\bm{x}||_{2}+\lambda g\left({\alpha_{t-1}, \alpha_{t}}\right)||\bm{J}_{\bm{\theta},t}(\bm{x}_{t})\Delta\bm{x}||_{2}$ $\displaystyle\ \ \ \ \ \ \ -\lambda g\left({\alpha_{t-1},\alpha_{t}}\right)g \left({\alpha_{t},\alpha_{t-1}}\right)||\bm{J}_{\bm{\theta},t-1}(\bm{x}_{t-1}) \bm{J}_{\bm{\theta},t}(\bm{x}_{t})\Delta\bm{x}||_{2},$ $\displaystyle\leq-\lambda g\left({\alpha_{t},\alpha_{t-1}}\right)||\bm{J}_{\bm {\theta},t-1}(\bm{x}_{t-1})\Delta\bm{x}||_{2}$ $\displaystyle\ \ \ \ \ \ \ +\lambda g\left({\alpha_{t-1},\alpha_{t}}\right) \left(1-g\left({\alpha_{t},\alpha_{t-1}}\right)L\right)||\bm{J}_{\bm{\theta},t }(\bm{x}_{t})\Delta\bm{x}||_{2},$ $\displaystyle=-g\left({\alpha_{t},\alpha_{t-1}}\right)||\hat{\bm{x}}_{0,t-1}^{ \mathcal{W}}-\hat{\bm{x}}_{0,t-1}||_{2}$ $\displaystyle\ \ \ \ \ \ \ +g\left({\alpha_{t-1},\alpha_{t}}\right)\left(1-g \left({\alpha_{t},\alpha_{t-1}}\right)L\right)||\hat{\bm{x}}_{0,t}^{\mathcal{W }}-\hat{\bm{x}}_{0,t}||_{2},$ $$

The first inequality holds because $g\left({\alpha_{t-1},\alpha_{t}}\right)<0$ and $g\left({\alpha_{t},\alpha_{t-1}}\right)>0$. The second inequality holds because $||\bm{J}_{\bm{\theta},t-1}(\bm{x}_{t-1})\bm{J}_{\bm{\theta},t}(\bm{x}_{t})
\Delta\bm{x}||_{2}\leq||\bm{J}_{\bm{\theta},t-1}(\bm{x}_{t-1})||_{2}||\bm{J}_{
\bm{\theta},t}(\bm{x}_{t})\Delta\bm{x}||_{2}\leq L||\bm{J}_{\bm{\theta},t}(\bm
{x}_{t})\Delta\bm{x}||_{2}$. From 1, with probability $1-\dfrac{1}{r_{t-1}}$,

$$ $||\hat{\bm{x}}_{0,t-1}^{\mathcal{W}}-\hat{\bm{x}}_{0,t-1}||_{2}\leq\lambda Lh( r_{t-1}),$ $$

with probability $1-\dfrac{1}{r_{t}}$,

$$ $||\hat{\bm{x}}_{0,t}^{\mathcal{W}}-\hat{\bm{x}}_{0,t}||_{2}\leq\lambda Lh(r_{t }),$ $$

Thus, from the union of bound, with a probability at least $1-\dfrac{1}{r_{t}}-\dfrac{1}{r_{t-1}}$,

$$ $\displaystyle||\bm{\tilde{}}{x}^{\mathcal{W}}_{t}-\bm{x}^{\mathcal{W}}_{t}||_{2}$ $\displaystyle\leq-\lambda Lg\left({\alpha_{t},\alpha_{t-1}}\right)h(r_{t-1})+ \lambda Lg\left({\alpha_{t-1},\alpha_{t}}\right)\left(1-g\left({\alpha_{t}, \alpha_{t-1}}\right)L\right)h(r_{t})$ $\displaystyle\leq\lambda L\left(-g\left({\alpha_{t},\alpha_{t-1}}\right)+g \left({\alpha_{t-1},\alpha_{t}}\right)\left(1-Lg\left({\alpha_{t},\alpha_{t-1} }\right)\right)\right)h(\max\{r_{t-1},r_{t}\})$ $$

∎

###### Proof of 1 .

###### Proof of 2 .

## Appendix D Auxiliary Results

###### Lemma 1 .

Given a unit vector $\bm{v}_{i}$ with and $\bm{\epsilon}\sim\mathcal{N}(\bm{0},\bm{I}_{d})$, we have

$$ $\mathbb{E}_{\bm{\epsilon}\sim\mathcal{N}(\bm{0},\bm{I}_{d})}[\left(\bm{v}_{i}^ {T}\bm{\epsilon}\right)^{2}/||\bm{\epsilon}||^{2}_{2}]=\dfrac{1}{d}.$ $$

###### Proof of Lemma 1 .

Because $\bm{\epsilon}\sim\mathcal{N}(\bm{0},\bm{I}_{d})$,

$$ $\bm{v}_{i}^{T}\bm{\epsilon}\sim\mathcal{N}(\bm{v}_{i}^{T}\bm{0},\bm{v}_{i}^{T} \bm{I}_{d}\bm{v}_{i})=\mathcal{N}(\bm{v}_{i}^{T}\bm{0},\bm{v}_{i}^{T}\bm{I}_{d }\bm{v}_{i})=\mathcal{N}(0,1),$ (19) $$

Assume a set of $d$ unit vecotrs $\{v_{1},v_{2},\ldots,\bm{v}_{i},\ldots,v_{d}\}$ are orthogonormal and are basis of $\mathbb{R}^{d}$, similarly, we could show that $\forall j\in[d],X_{j}\coloneqq v_{j}^{T}\bm{\epsilon}\sim\mathcal{N}(0,1).$ Therefore, we could rewrite $\left(\bm{v}_{i}^{T}\bm{\epsilon}\right)^{2}/||\bm{\epsilon}||^{2}_{2}$ as:

$$ $\displaystyle\left(\bm{v}_{i}^{T}\bm{\epsilon}\right)^{2}/||\bm{\epsilon}||^{2 }_{2}$ $\displaystyle=\frac{\left(\bm{v}_{i}^{T}\bm{\epsilon}\right)^{2}}{||\sum_{k=1} ^{d}v_{k}v_{k}^{T}\bm{\epsilon}||^{2}_{2}},$ (20) $\displaystyle=\frac{\left(\bm{v}_{i}^{T}\bm{\epsilon}\right)^{2}}{\sum_{k=1}^{ d}\left(v_{k}^{T}\bm{\epsilon}\right)^{2}},$ (21) $\displaystyle=\frac{X_{i}^{2}}{\sum_{k=1}^{d}X_{k}^{2}}.$ (22) $$

Let $Y_{i}\coloneqq\frac{X_{i}^{2}}{\sum_{j=1}^{d}X_{j}^{2}}$. Because $\forall j\in[d],X_{j}\coloneqq v_{j}^{T}\bm{\epsilon}\sim\mathcal{N}(0,1)$, $\forall j\in[d],Y_{j}$ has the same distribution. Additionally, $\sum_{j=1}^{d}Y_{j}=1$. So:

$$ $\displaystyle\mathbb{E}_{\bm{\epsilon}\sim\mathcal{N}(\bm{0},\bm{I}_{d})}[ \frac{\left(\bm{v}_{i}^{T}\bm{\epsilon}\right)^{2}}{||\bm{\epsilon}||^{2}_{2}} ]=\mathbb{E}[Y_{i}]=\frac{1}{d}\mathbb{E}[\sum_{j=1}^{d}Y_{j}]=\frac{1}{d}.$ $$

∎

###### Lemma 2 .

Given a matrix $\bm{J}\in\mathbb{R}^{d\times d}$ with $\operatorname{rank}\left(\bm{J}\right)=r$. Given $\bm{x}$ which is uniformly sampled on the unit hypersphere $\mathbb{S}^{d-1}$, we have:

$$ $\displaystyle\mathbb{E}_{\bm{x}}\left[||\bm{J}\bm{x}||_{2}^{2}\right]=\frac{|| \bm{J}||^{2}_{F}}{d}.$ $$

###### Proof of Lemma 2 .

Let’s define the singular value decomposition of $\bm{J}=\bm{U}\bm{\Sigma}\bm{V}^{T}$ with $\Sigma=\operatorname{diag}\left(\sigma_{1},\ldots,\sigma_{r},0\ldots,0\right)$. Therefore, $\mathbb{E}_{\bm{x}}\left[||\bm{J}\bm{x}||_{2}^{2}\right]=\mathbb{E}_{\bm{x}}
\left[||\bm{U}\bm{\Sigma}\bm{V}^{T}\bm{x}||_{2}^{2}\right]=\mathbb{E}_{\bm{z}}
\left[||\bm{\Sigma}\bm{z}||_{2}^{2}\right]$ where $\bm{z}\coloneqq\bm{V}^{T}\bm{x}$ is is uniformly sampled on the unit hypersphere $\mathbb{S}^{d-1}$. Thus, we have:

$$ $\displaystyle\mathbb{E}_{\bm{z}}\left[||\bm{\Sigma}\bm{z}||_{2}^{2}\right]$ $\displaystyle=\mathbb{E}_{\bm{z}}\left[||\sum_{i=1}^{r}\sigma_{i}\bm{e}_{i}^{T }\bm{z}||_{2}^{2}\right],$ $\displaystyle=\mathbb{E}_{\bm{z}}\left[\sum_{i=1}^{r}\sigma^{2}_{i}||\bm{e}_{i }^{T}\bm{z}||_{2}^{2}\right],$ $\displaystyle=\sum_{i=1}^{r}\sigma^{2}_{i}\mathbb{E}_{\bm{z}}\left[||\bm{e}_{i }^{T}\bm{z}||_{2}^{2}\right]=\frac{||\bm{J}||^{2}_{F}}{d},$ $$

where $\bm{e}_{i}$ is the standard basis with $i$-th element equals to 0. The second equality is because of independence between $\bm{e}_{i}^{T}\bm{z}$ and $\bm{e}_{j}^{T}\bm{z}$. The fourth equality is from Lemma 1.
∎

###### Lemma 3 .

Given function $f\left(\bm{x}\right)=||\bm{J}\bm{x}||^{2}_{2}$, the lipschitz constant $L_{f}$ of function $f\left(\bm{x}\right)$ is:

$$ $L_{f}=2||\bm{J}||^{2}_{2}.$ $$

###### Proof of Lemma 3 .

The jacobian of $f(\bm{x})$ is:

$$ $\nabla_{\bm{x}}f(\bm{x})=2\bm{J}^{T}\bm{J}\bm{x},$ $$

Therefore, the lipschitz constant $L$ follows:

$$ $L_{f}=\sup_{\bm{x}\in\mathbb{S}^{d-1}}||\nabla_{\bm{x}}f(\bm{x})||_{2}=2\sup_{ \bm{x}\in\mathbb{S}^{d-1}}||\bm{J}^{T}\bm{J}\bm{x}||_{2}=||\bm{J}^{T}\bm{J}||_ {2}=||\bm{J}||^{2}_{2}$ $$

∎

###### Lemma 1 .

###### Proof of Lemma 1 .

###### Lemma 2 .

###### Proof of Lemma 2 .

###### Lemma 3 .

###### Proof of Lemma 3 .

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: Please see the weaknesses part.
soundness: 3
strengths: 1. Shallow Diffuse's primary strength lies in its utilization of the low-rank property of the PMP's Jacobian matrix to minimize the visual impact of watermarks, thereby attaining visual consistency within a training-free watermark framework.
2. The injected watermark is more robust against several image distortions than existing baselines.
summary: This paper proposed Shallow Diffuse, a watermarking technique for diffusion models. The method is well-motivated and with proper theoretical justification. The proposed Shallow Diffuse has several key advantages compared to existing diffusion watermarks, 1) It is a training-free watermark but simultaneously maintains the consistency between watermarked and original images. 2) It is more robust than existing baselines, achieving nearly no performance drop under different robustness tests. Shallow Diffuse also considers two scenarios including the server (protect generated image) and user (protect existing image) scenarios for injecting the watermark.
weaknesses: 1. The presentation of this paper is poor, for instance, the ablation studies (Appendix C) and index of experimental results (Table 4) are incomplete. Therefore, this leads to a shortage of critical ablation studies.
2. What is the performance of multi-key identification, specifically, is it possible for the Shallow Diffuse to inject multiple watermarks and distinguish between them?
3. The image distortions are less than that in previous studies, such as Tree-Ring, where they apply 6 distortions.
4. Can DiffPure purify the watermarked patterns?
5. The findings in Table 4 are confusing. It appears that employing channel averaging enhances robustness against image distortions. However, channel averaging involves averaging clean and watermarked images across specific channels. As per my understanding, this process might reduce watermark robustness. Can you explain this observation?

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. The table in the paper is very difficult to read clearly.

2. In Table 1, for the CLIP-Score index, yours is 0.3285, which seems to be the worst. Please explain further.

3. Please explain why the filter size in the Gaussian blurring is 8 × 8 and how the standard deviation is selected.

4. As can be seen from Table 2, the PSNR and SSIM of most methods are very low, so it is easy for human eyes to find modification traces, which easily leads to the risk of watermarked images being maliciously broken. Please further explain the visual quality of the generated watermarked image.
soundness: 3
strengths: The originality of this paper is not great, but its quality, clarity and significance are good. It has the support of rich theoretical basis and has advantages in theoretical proof.
summary: This paper proposes a watermarking technique Shallow Diffuse. Unlike existing approaches that integrate watermarking throughout the entire diffusion sampling process,  Shallow Diffuse decouples these steps by leveraging the presence of a low-dimensional subspace in the image generation  process. This method ensures that a substantial portion of the watermark lies in the null space of this subspace,  effectively separating it from the image generation process.
weaknesses: The table in the paper is not very well drawn, it is very difficult to read, especially the header. At the same time, the experimental part is not detailed enough. For example, should the comparison method reproduce the results or use the pre-training model?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Besides Tree-Ring and Stable Signature, I think there are more existing watermarking methods that introduce small perturbation to the image to embed watermark, like StegaStamp. To show the proposed method actually preserves image quality, I think the authors should compare the method with some watermarking methods that watermark the image after image generation with a small perturbation.
2. In the robustness part, I think the regeneration attacks and some adversarial perturbation should be evaluated on the proposed method to see whether the proposed method is actually robust under various attacks.
3. Since the authors mention the user scenario, if multiple users rewatermark the same image with the proposed method, can the watermark embeded by the specific user be detected in this circumstance?
soundness: 2
strengths: 1. The proposed method has smaller impact on the generated images compared to the existing watermarking methods designed for diffusion models.
2. Experiments are carried on several image-prompt datasets to show the effectiveness of the proposed methods.
3. The robustness of the propsoed method is evaluated.
summary: The paper proposed a new image watermarking method that embeds a watermark into the null space of a specific step during image denoising in diffusion model. It shows that the proposed watermarking method have smaller impact on the generated images compared to the existing methods like Stable Signature and Tree-Ring. Additionally, the proposed method shows good robustness to image processing methods like JPEG and Gaussian blur.
weaknesses: 1. Compared to Tree-Ring, the technical contribution of the proposed method is limited.
2. In the experimental part, the authors mainly compare their method with the watermarking methods that embed watermark into the semantic space like Tree-Ring which changes the image a lot. More other watermarking methods should be evaluated.
3. In the robustness part, the authors only evaluate the robustness of the proposed method on some common perturbation.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Shouldn't the formula $\hat{\boldsymbol{x}}_{0, t}:=\boldsymbol{f}_{\boldsymbol{\theta}, t}\left(\boldsymbol{x}_{t}+\lambda \Delta \boldsymbol{x}\right)$ on line 360 be  $\hat{\boldsymbol{x}}_{0, t}:=\boldsymbol{f}_{\boldsymbol{\theta}, t}\left(\boldsymbol{x}_{t}\right)$

2. Could you specify the parameters used in the Shallow Diffuse method in Section 5, such as the embedding channels and watermark radius?

3. The experiments in Appendix C only provide results, could you include some analysis?
soundness: 2
strengths: 1. The method utilizes the local linearity of low-dimensional subspaces. As a watermarking method based on diffusion models, it maintains the consistency of generated images.

2. This paper provides rigorous theoretical proof and presents a substantial number of computational formulas.
summary: Current watermarking techniques based on diffusion models often embed watermarks directly into the initial noise，which can alter the data distribution. This paper proposes "Shallow Diffuse," a method that disentangles the watermark embedding from the generation process by leveraging low-dimensional subspaces. This approach supports watermark embedding for both server-side and user-side applications while maintaining high robustness and consistency. Additionally, experiments were designed to validate robustness and conduct ablation studies across multiple datasets.
weaknesses: 1. The attack experiments are limited, consisting of only four fixed-parameter attacks, which do not demonstrate the method's robustness. For instance, the method can be viewed as a variant of treering, could experiments with additional attacks, such as rotation、regeneration be included?

2. The theoretical assumptions of the method are built upon [1], but the experimental results yield a different range of t values compared to the theoretical analysis in [1]. Although this can be explained by the errors introduced by DDIM-Inv, it remains perplexing.

3. The method relies on the properties of DDIM and DDIM-inverse, which may lack certain generalizability. It might not perform well for attacks executed in the latent space.

Meta-review: This paper studies diffusion-based digital watermarking to deal with the AI-generated content tracing problem. The proposed approach, ``Shallow Diffuse,'' tries to decouple both the watermarking and diffusion processes by leveraging the presence of a low-dimensional subspace in the image generation process. Both theoretical and empirical analyses were presented.

The authors claimed ``Consistency and Robustness: Extensive experiments demonstrate that Shallow Diffuse consistently outperforms other diffusion-based methods in terms of both robustness and reproducibility.'' in their response. Most reviewers are satisfied by the responses.
However, by checking Table 4, Shallow Diffuse presents robustness inferior to RingID under a vert limited set of selected attacks.
For multiple key identification presented in Table 5, ShallowDiffuse performs inferior to RingID.
In term of these watermarking requirements, it is hard to say that ShallowDiffuse advances the development of diffusion-based watermarking.
To surpass and to identify sufficient differences from the prior works, it is encouraged to conduct robustness evaluations under a broad range of attacks/distortions.

Decision: Reject