WMAdapter: Adding WaterMark Control to Latent Diffusion Models

## Abstract

Abstract Watermarking is crucial for protecting the copyright of AI-generated images. We propose WMAdapter, a diffusion model watermark plugin that takes user-specified watermark information and allows for seamless watermark imprinting during the diffusion generation process. WMAdapter is efficient and robust, with a strong emphasis on high generation quality. To achieve this, we make two key designs: (1) We develop a contextual adapter structure that is lightweight and enables effective knowledge transfer from heavily pretrained post-hoc watermarking models. (2) We introduce an extra finetuning step and design a hybrid finetuning strategy to further improve image quality and eliminate tiny artifacts.
Empirical results demonstrate that WMAdapter offers strong flexibility, exceptional image generation quality and competitive watermark robustness.

## 1 Introduction

With the widespread adoption of diffusion models [, , , , , ], diffusion-generated images are proliferating across media and the internet. While these models meet the demand for high-quality creative content, their misuse raises significant concerns about copyright protection and the security of images against deepfakes []. Watermarking technology [] provides a tailored solution for resolving copyright disputes and identifying the sources of forgeries.

Previous watermarking methods added watermarks to images in a post-hoc way through frequency domain transformations [, , ] or by training watermark encoding networks [, , ]. However, in the context of watermarking diffusion images, post-hoc methods introduce additional workflows and separate from diffusion pipeline. Recently, more efforts [, , , , , , , ] have focused on leveraging the characteristics of the diffusion process to seamlessly integrate watermarking into the diffusion pipeline, known as diffusion-native watermarking.
Among these, Stable Signature [] proposed a method that fine-tunes the VAE decoder of a latent diffusion model [] using a pretrained watermark decoder [].
This approach has shown promising results. However, it requires fine-tuning a different VAE decoder for each unique watermark, making it difficult to scale to millions of keys necessary for real large-scale scenarios. Additionally, the tuning of VAE decoder on a small amount of data results in blurrier generated images and lens flare-like artifacts (see Fig. 6).

To address these issues, we propose WMAdapter, a diffusion watermark plugin (Fig. 1) that accepts arbitrary user input watermark bits and imprints the watermark on-the-fly without per-watermark finetuning.
Since WMAdapter serves as a plug-and-play module and keeps the original VAE decoder intact, it can also produce sharper images with minimal artifacts.

Many concurrent works [, , , , , ] also explores watermark plugins for diffusion models. However, their approaches typically involve lengthy training pipeline or heavy structures, such as finetuning a heavy UNet [] plugin at each denoising time step [], training the plugin and watermark decoder from scratch [, ], or using a heavy watermark decoder []. In contrast, our motivation is to create a solution that is easy to train and use for diffusion practices.
To achieve this, we have designed a lightweight structure, utilizing a pretrained watermark decoder and a minimal training pipeline. Tab. 1 compares several concurrent methods.
This lightweight design also poses significant challenges in achieving a robust and high-quality watermark adapter. To be both efficient and powerful, we propose a novel Contextual Adapter structure for efficient and effective knowledge transfer from pretrained decoders.
Unlike widely adopted non-contextual adapters [, , , ] that only condition on the watermark bits, our contextual structure also conditions on the cover image features from the VAE (hence "contextual").
We found this dual conditioning better adapts watermark to the generated images, being crucial for high-quality knowledge transfer and image generation.

**Table 1: Comparison between diffusion watermark plugins. "Orange" colors modified modules. “WM Dec” is short for “Watermark Decoder”. WADIFF [] and AquaLoRA [] alter the layout of generated images. FSW [] has to work with a finetuned VAE decoder.**
|  | Plug onto | Modules Involved in Training | Plug-and-play | Imperceptible |
| --- | --- | --- | --- | --- |
| WADIFF [] | UNet | Plugin+UNet+VAE+WM Dec | ✓ | ✗ |
| AquaLoRA [] | UNet | Plugin+UNet+VAE+WM Dec | ✓ | ✗ |
| FSW [] | VAE | Plugin+VAE+WM Dec | ✗ | ✓ |
| WOUAF [] | VAE | Plugin+VAE+WM Dec | ✓ | ✓ |
| Ours | VAE | Plugin+VAE+WM Dec | ✓ | ✓ |

To further improve the image generation quality, we introduce an extra finetuning stage and propose a novel Hybrid Finetuning strategy. This strategy involves jointly finetuning the Adapter and the VAE decoder, while using the original VAE decoder during inference. It effectively suppresses tiny artifacts and significantly enhances image sharpness.
We summarize our contributions as follows:

- 1.
We introduce WMAdapter, a novel framework that dynamically imprints different watermarks for diffusion models without requiring individual finetuning for each watermark.
- 2.
We propose a compact contextual adapter structure and a hybrid finetuning strategy, ensuring efficient learning and high quality watermarking.
- 3.
Experimental results show that WMAdapter not only maintains watermark robustness but also significantly enhances flexibility, scalability, and image generation quality,

## 2 Related work

### 2.1 Post-hoc watermarking

Post-hoc methods include traditional frequency domain transformation methods such as DwtDct [] and DwtDctSvd [], per-image optimization-based methods like SSL [] and FNNS [], and forward encoding/decoding methods such as HiDDeN [], Stegastamp [], MBRS []. Different methods have various design focuses. For instance, FNNS emphasizes hiding more bits, HiDDeN and MBRS prioritizes robustness against JPEG compression, and Stegastamp targets robustness against real-world re-photography.

### 2.2 Diffusion-native watermarking

According to the location of the watermark, we classify diffusion-native watermarking methods into two categories.
Adding to initial noise:
Tree-Ring [] proposed adding watermarks to the diffusion initial noise, achieving remarkable robustness. However, this approach lacks multi-key identification capabilities []. Subsequent methods improve by using enhanced imprinting techniques [, ], or employing encoder/decoder on initial noise []. However, these methods significantly alter the layout of the generated images, which is not desirable in some production scenarios.
Adding to latent space:
This category involves imprinting watermarks into the latent (or feature) space of the VAE or the diffusion UNet, with most methods not altering the generated image layout. Methods like RoSteALS [], LW [], and PAP [] imprint watermarks into the VAE’s latent space, requiring more effort to train both the watermark plugin and the watermark decoder from scratch. RoSteALS and LW also suffer from unstable training, necessitating multi-stage training or subtle hyperparameter tuning.
WADIFF [] and AquaLoRA [] imprints watermarks into the UNet backbone, resulting in longer training pipelines and changes to the generated image layout. FSW [], StableSignature [], and WOUAF [] are similar to our approach in that they inject watermarks into the VAE feature space, but they need to modify the VAE decoder, which tends to degrade the generation quality.
In contrast, WMAdapter keeps the entire diffusion pipeline intact and can be directly plugged onto the VAE decoder (Fig. 1), providing greater flexibility and higher generation quality.

## 3 Method

In this section, we will introduce the framework of WMAdapter, detail its contextual structure, and discuss the training and fine-tuning strategies.

### 3.1 Framework overview

Fig.1 illustrates the overall framework of WMAdapter. WMAdapter is a plug-and-play watermark module that can be directly attached to the VAE decoder of a latent diffusion model []. It imprints the watermark during image generation, seamlessly integrating into the diffusion generation workflow. WMAdapter employs a novel contextual adapter structure, which takes both watermark bits and image features from the VAE decoder as input and outputs feature residuals containing watermark information.
Watermarked images can be directly fed into a pretrained watermark decoder, such as HiDDeN [], to retrieve the watermark information.

The training of WMAdapter consists of two stages: large-scale training and fast finetuning. In the training stage, we freeze the VAE decoder and the watermark decoder and train only the Adapter on a large scale dataset.
We then finetune the Adapter and VAE decoder on a small amount of data.
Specifically, we present a novel hybrid finetuning strategy that is able to suppress tiny artifacts and significantly enhance generation quality. We also discuss several different strategies concerning different tradeoffs between robustness and quality.

Figure: Figure 1: Framework overview. WMAdapter is plugged onto the VAE decoder. It takes user input watermark bits and image features from the VAE decoder, imprinting the watermark on-the-fly during VAE decoding. In contrast, traditional non-contextual adapters take only watermark conditions as input. WMAdapter can be trained with a post-hoc watermark decoder for efficient knowledge transfer. The image and icons credit to [, ].
Refer to caption: x1.png

### 3.2 Contextual adapters

In this section, we provide a detailed overview of the contextual structure of WMAdapter. Fig. 2 (Left) illustrates the internal structure of WMAdapter, which comprises a series of independent Fuser modules. Each Fuser $\phi_{i}(\cdot)$ is attached before a corresponding VAE decoder block $i$. It receives both VAE feature $f_{i}$ and watermark bits $w$ as inputs, and outputs a feature residual $y_{i}$ to update $f_{i}$. Formally,

$$ $\displaystyle y_{i}$ $\displaystyle=\phi_{i}\left(f_{i},w\right),$ (1) $\displaystyle f_{i}^{\prime}$ $\displaystyle=f_{i}+y_{i}.$ $$

We put a total of 6 Fusers before the Conv Block, Middle Block and four Up Blocks in the kl-f8 VAE decoder used by Stable Diffusion [].

Fig. 2 (Right) illustrates the internal structure of an Fuser. An Fuser consists of two main components: the Embedding module and the Fusing module. The Embedding module maps the 01 bit sequence into a 48-dimensional watermark feature vector. This feature vector is then expanded along the width and height dimensions to produce a watermark feature map with the same dimensions as the image feature. The image feature and watermark feature are concatenated along the channel dimension and fed into the Fusing module, which outputs the image feature residuals.
Keeping lightweight in mind, we use two MLPs with 256 intermediate feature channels for the Embedding module, and two 1x1 convolutions with half the image feature channels $\frac{c}{2}$ as intermediates for the Fusing module. We employ LeakyReLU as the non-linearity. The total parameters of WMAdapter are only 1.3M, making it a small and efficient plugin.

Figure: Figure 2: The architecture of WMAdapter. Left: The structure of WMAdapter. It comprises several independent Fusers with identical structures. Right: The structure of Fuser. It consists of a watermark Embedding module and a Fusing module.
Refer to caption: x2.png

### 3.3 Training

In the training stage, we use a pretrained watermark decoder to decode watermark bits from the watermarked images. We freeze the watermark decoder and the VAE decoder, and only train the Adapter. Why do we use a pretrained decoder instead of training a watermark decoder from scratch along with the Adapter? We observe that training an encoder/decoder pair from scratch, as post-hoc methods do, typically requires significant training effort. For example, HiDDeN takes 300 epochs to converge on the COCO dataset. Using a pretrained post-hoc decoder facilitates efficient knowledge transfer, allowing WMAdapter to converge in just 1-2 epochs.
We use two types of losses as our objective: the consistency loss between the watermarked image $x_{w}$ and the unwatermarked image $x$, and the accuracy of decoded bits. The total loss function is defined as:

$$ $\mathcal{L}=\lambda_{1}\mathcal{L}_{mae}\left(x,x_{w}\right)+\lambda_{2} \mathcal{L}_{lpips}\left(x,x_{w}\right)+\lambda_{3}\mathcal{L}_{vgg}\left(x,x_ {w}\right)+\lambda_{4}\mathcal{L}_{bce}\left(w,w^{\prime}\right)$ (2) $$

where the first three terms represent image consistency losses. We use MAE and LPIPS loss [] to maintain consistency with VAE pretraining []. Additionally, we include a Watson-VGG loss [] similar to Stable Signature [] to enhance human visual preference. For watermark decoding accuracy, we use binary cross-entropy loss bewteen decoded bits $w^{\prime}$ and input bits $w$. We empirically set $\lambda_{1},\lambda_{2},\lambda_{3},\lambda_{4}$ to 0.2, 0.2, 0.08, 1.0, respectively.

### 3.4 Hybrid finetuning

After the training stage, we obtain a watermark adapter that performs well in both accuracy and image quality (Sec. 4.4.2). However, when we zoom in on the generated images by 20 times, grid-like artifacts can often be observed (Fig. 5). To further improve image quality and eliminate these tiny artifacts, we introduce a fine-tuning stage with a small amount of data. In addition to the first stage training losses, we incorporate a total variation loss [] on the watermarked images to enhance smoothness, setting its weight to 0.02.

Additionally, we propose a novel Hybrid Finetuning strategy. We finetune both the Adapter and the VAE decoder, but use the fine-tuned Adapter and the original VAE decoder for inference. Fig. 3 distinguishes this method from two other classic finetuning strategies: Fixed and Joint Finetuning.
The Fixed Finetuning strategy uses the same training approach as in the first stage, fixing the VAE decoder and quickly finetuning the Adapter with a high learning rate. The Joint Finetuning strategy jointly finetunes the Adapter and the VAE decoder, using both finetuned copies for inference.

Empirical results in Sec. 4.4.2 demonstrate that Hybrid Finetuning effectively removes tiny artifacts and, by keeping the VAE intact, produces the sharpest and clearest images while maintaining the plug-and-play advantage, making it ideal for scenarios requiring high image quality. The Fixed Finetuning strategy improves image quality compared to the first stage, but it fails to effectively eliminate tiny artifacts. Joint Finetuning can slightly enhance robustness, but the modifications to the VAE decoder result in blurrier and smoother images, as well as lens flare-like artifacts.

Figure: Figure 3: Illustration of 3 different finetunig strategies. They differ in how to treat the VAE decoder.
Refer to caption: x3.png

### 3.5 Discussion

The design of WMAdapter emphasizes lightweight, efficient, and flexible operation. We prioritize watermark invisibility while maintaining watermark robustness. This contrasts with previous methods that focused more on robustness at the expense of image quality or simplicity.
We argue that high image quality, along with light weight, flexibility, and ease of use should be the top priorities when deploying watermarking technology in diffusion products.
One evidence may be that the Stable Diffusion series [, ] use traditional DwtDct [] as the default watermarking method. This focus is the rationale behind WMAdapter’s lightweight design and hybrid finetuning strategy.

## 4 Experiment

### 4.1 Experimental setup

##### Model and dataset

We experiment with a popular latent diffusion model Stable Diffusion 2.1 [] and its associated kl-f8 VAE. We adopt the pretrained watermark decoder from HiDDeN []. The checkpoint we use was pretrained by [], encoding 48-bits watermark information. This checkpoint is also used to finetune Stable Signature []. Thus, our adapter can be directly compared with []. ALL training and finetuning steps are performed on MS-COCO 2017 [] training set. Validation is performed on COCO 2017 validation set. We train and evaluate our adapters on images at resolution $512\times 512$. For images smaller than this size, we resize their shorter edge to 512, then center crop to get a $512\times 512$ image.

##### Training strategies

For the first stage training, we adopt 8 $\times$ NVIDIA A5000 GPUs of 24 GB memory, with per-GPU batchsize of 2, AdamW optimizer [], a learning rate of 5e-4. We train the model for 2 epochs, taking about 5 hours. In comparison, post-hoc methods like HeDDeN [] usually take hundreds of epochs to converge, showing the value of adapting from pretrained models.

For the second stage finetuning, we use a single A5000 GPU. We set the mini-batch to 2. We also use the AdamW optimizer and a start learning rate of 5e-4. However, we adopt a per-step cosine learning rate decay with 20 warm-up steps. Unless otherwise specified, the total
fine-tuning process defaults to 2,000 steps, lasting for about 50 minutes.
Different finetuning strategies result in several different adapter variants. We use Adapter-B, Adapter-F, Adapter-V, and Adapter-I to denote the adapters obtained by No Finetuning, Fixed Finetuning, Joint Finetuning and Hybrid Finetuning, respectively.

##### Evaluation metric

Following previous conventions [, , ], we use average bit accuracy to evaluate the watermarking performance of our adapter. Bit accuracy is defined by the ratio of correctly decoded bits in a 48-bit watermark sequence.
Apart from the bit accuracy, we also report the tracing accuracy among different numbers of users following concurrent works [, , ]. We adopt the evaluation protocol of [].
Concretely, we construct user pools of different sizes, ranging from $10^{4}$ to $10^{6}$, to evaluate the accuracy of user tracing at different scales. Each user is assigned a unique key.
For each user pool. we randomly select 1,000 users and watermark 5 images per user, resulting in 5,000 watermarked images. For each of the 5,000 images, we find the best match among the user pool and check if it’s a correct match. Tracing accuracy is then averaged over all 5,000 images.

In addition to accuracy measurements, we are also interested in the watermark’s invisibility and image generation quality. We report the Peak Signal-Noise-Ratio (PSNR), Structural Similarity Index Measure (SSIM) [], and Fréchet Inception Distance (FID) [] metrics between images before and after watermarking. Typically, higher PSNR and SSIM means more similar to the original image, with less noise and distortion, leading to sharper and clearer images. While lower FID means the watermarked images have higher fidelity and more closely resemble the real images in terms of appearance and variety.

**Table 2: Comparison with other watermarking methods on generation quality and robustness. All methods are evaluated on COCO 2017 val set [] with image size $512\times 512$. Since Stable Signature [] requires finetuning of separate VAE decoders to embed different keys, we report its average results on 10 randomly sampled keys. For robustness, we use Crop 0.3, JPEG 80, Brightness 1.5. Following [], we use (Crop 0.5, Jpeg 80, Brightness 1.5) for combined attacks.**
|  |  |  |  |  | Bit Accuracy $\uparrow$ |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Method | PSNR $\uparrow$ | SSIM $\uparrow$ | FID $\downarrow$ | None | JPEG | Crop | Bright | Comb |
| Post-hoc | DwtDct [] | 39.1 | 0.97 | 2.6 | 0.79 | 0.50 | 0.51 | 0.79 | 0.50 |
| DwtDctSvd [] | 39.1 | 0.98 | 3.2 | 0.72 | 0.71 | 0.50 | 0.49 | 0.50 |  |
| SSL [] | 33.0 | 0.89 | 14.8 | 1.00 | 0.99 | 0.97 | 0.98 | 0.92 |  |
| HiDDeN [] | 34.1 | 0.95 | 3.1 | 0.98 | 0.84 | 0.97 | 0.98 | 0.87 |  |
| Merged | Stable Signature [] | 29.7 | 0.87 | 3.2 | 0.99 | 0.93 | 0.99 | 0.99 | 0.94 |
| WMAdapter-F | 33.1 | 0.95 | 2.7 | 0.99 | 0.92 | 0.99 | 0.99 | 0.93 |  |
| WMAdapter-I | 34.8 | 0.96 | 2.5 | 0.98 | 0.90 | 0.97 | 0.97 | 0.91 |  |

### 4.2 Comparison with other methods

##### Bit accuracy and image quality

We compare our method with four post-hoc watermarking methods [, , ] and one merged diffusion watermarking method Stable Signature []. Among the post-doc methods, DwtDct [] and DwtDctSvd [] are traditional frequency-based watermarking methods. DwtDct is also the default watermarking method used by Stable Diffusion []. SSL [] and HiDDeN [] are neural network-based methods. SSL is an iterative optimization method, while HiDDeN is an encoder-decoder method. For HiDDeN, we use the model provided by [].

Tab. 2 shows the comparison.
Traditional methods DwtDct and DwtDctSvd achieve the highest PSNR and SSIM, at 39.1 dB and 0.98 respectively, indicating their ability to generate high-quality images with high watermark invisibility. However, these methods exhibit relatively weak robustness and cannot handle common image transformations such as cropping and JPEG compression. Even on clean images, they achieve only about 0.7-0.8 bit accuracy.

Neural network-based methods like SSL, HiDDeN, and Stable Signature are significantly more robust against various attacks due to training time augmentation. They achieve $0.9$ or higher bit accuracy under JPEG compression, cropping, brightness adjustment, and combined attacks. However, this robustness comes at the cost of image quality. SSL produces noticeable artifacts on generated images (Fig. 11) and results in a very high FID. Stable Signature, on the other hand, produces smoother but blurrier images due to the fine-tuning of the VAE decoder, achieving a PSNR of 29.7 dB, an SSIM of 0.87, and an FID of 3.2. We also observe that Stable Signature tends to produce lens flare-like artifacts. This is shown in Fig. 6 and will be discussed in Sec. 4.5.

Our adapters achieve the best image quality among neural network-based methods, excelling in PSNR, SSIM, and FID metrics. WMAdapter-F achieves a PSNR of 33.1 dB, an SSIM of 0.95, and an FID of 2.7, significantly outperforming Stable Signature while maintaining comparable robustness. Notably, Stable Signature requires fine-tuning the VAE decoder for each specific watermark key, making it impractical for scaling to millions of keys. In contrast, our adapter can embed up to $2^{48}$ different keys on the fly.
Employing a hybrid fine-tuning strategy, WMAdapter-I further improves PSNR, SSIM, and FID to 34.8 dB, 0.96, and 2.5, at a little cost of 0.01-0.02 bit accuracy. Its FID score even surpasses the traditional methods DwtDct and DwtDctSvd. Qualitative comparisons in Sec. 4.5 also show that WMAdapter-I produces sharper images with minimal artifacts. Overall, WMAdapter-I strikes good balance between robustness and generation quality, making it suitable for scenarios where high image quality is prioritized..

##### Tracing accuracy

We further compare the tracing accuracy with 3 diffusion-native watermarking methods [, , ] in Tab. 4. We can see that our adapters achieve nearly perfect tracing accuracy with different scales of users.
Tree-Ring [] achieves zero tracing accuracy due to its design flaws uncovered by []. WADIFF [] is a concurrent effort, which also employs HiDDeN decoder to finetune
a watermark plugin for diffusion models. Since it also evaluates on COCO dataset, we report its numbers as a reference.
We can see that its tracing accuracy gradually drops as the scale grows despite they employ a heavier adapter and longer training pipeline. Both ours and Stable Signature perform consistently at different user scales. Notably, Stable Signature has higher average bit accuracy but gets slightly worse tracing accuracy than our adapters. We attribute this to its larger performance variance among different keys.

Figure: Table 3: Accuracy of tracing different numbers of keys. All methods are evaluated on COCO dataset []. For WADIFF^∗ [], we use the number reported by its original paper.
Refer to caption: x4.png

### 4.3 Robustness to more attacks

##### Various attack intensities

Fig. 7 shows the robustness of our adapters against a range of attack intensities. Our adapters achieve comparable performance to Stable Signature under various levels of attacks, while offering flexibility, scalability and higher image quality.

##### Against auto-encoders

We evaluate our WMAdapter-Fagainst compression attacks by two neural auto-encoders [, ] at different compression rates. We use models provided by [].
For cheng’20 [] and bmshj’18 [], we evaluate at compression rate 1-6 and 1-8, respectively.
Fig. 4 shows the Accuracy-PSNR curve. To successfully reduce bit accuracy to around 0.5, a strong compression rate has to be used, resulting in approximately a 5 dB loss in PSNR, showing similar trend to [, ].

### 4.4 Ablation Study

#### 4.4.1 Why contextual structure?

We study the effects of different adapter structures. Tab. 4 shows the comparison between different adapter variants after the first stage training. If we use $3\times 3$ conv kernels instead of $1\times 1$ kernels to fuse image features and watermark bits, the adapter suffers from unstable training.
If the adapter is non-contextual, i.e., only takes user bit sequence as input and does not receive image features, the bit accuracy on clean images will decrease by 0.02, and the PSNR will significantly drop by 4.1 dB to 28.7 dB. This demonstrates the importance of image features for watermark accuracy and invisibility. This is a crucial observation. We have found that many concurrent watermarking efforts still use the non-contextual plugin architecture [, , , ]. Our observation provides a simple yet promising approach to further enhance these methods.

**Table 4: Ablation study on adapter structures.**
|  | Ours | Non-contextual | Conv $3\times 3$ |
| --- | --- | --- | --- |
| Bit Acc | 0.99 | 0.97 | 0.49 |
| PSNR | 32.8 | 28.7 | 12.0 |

#### 4.4.2 Role of finetuning

Tab. 5 and Fig. 5 compare different finetuning strategies quantitatively and qualitatively.
From Tab. 5, we can see that Adapter-B achieves good numerical results. However, when we zoom in on the generated images by 20x times, tiny grid-like artifacts become visible. If we freeze the VAE decoder and perform a quick fine-tuning for 2k steps using a large learning rate, resulting in Adapter-F-2k. We find that while the accuracy remains almost unchanged, the PSNR and SSIM metrics improve, but the artifacts are still there.

If we employ the Hybrid Finetuning (Adapter-I), we can find that the artifacts are effectively removed. Additionally, because the VAE remains intact, this results in the sharpest and most visually appealing images, with the PSNR significantly improving to 34.8 dB. However, this comes at a little cost of a 0.02 loss in bit accuracy under combined attacks.

If we adopt the Joint Finetuning (Adapter-V), we observe an additional 0.01 improvement in bit accuracy compared to the baseline. However, this approach results in a decline in PSNR, SSIM, and FID. Visual comparisons in Fig. 5 show that Adapter-V produces smoother and blurrier images, which accounts for the decrease in metrics and also the elimination of the tiny grid-like artifacts seen with Adapter-B. As the fine-tuning steps increase from 200 to 2k, the images become sharper, but the grid-like artifacts re-emerge.
Notably, using the fine-tuned VAE during inference introduces lens flare-like artifacts, as shown in Fig. 6 and discussed in Sec. 4.5.

In summary, finetuning can further enhance image quality or accuracy, but different finetuning strategies introduce subtle trade-offs between accuracy, image sharpness, and artifacts. Considering the plug-and-play nature of adapters, we adopt Adapter-F-2k and Adapter-I-2k as our default choices, as they do not require a fine-tuned VAE decoder for inference.

Figure: Figure 5: Qualitative comparison between different finetuning strategies on COCO [] images. Adapter-B and Adapter-F produces tiny grid-like artifacts. Finetuning with VAE (Adapter-I and -V) alleviates this issues. Using fintuned VAE at inference time (Adapter-V) leads to blurry images. Using original VAE (Adapter-I) achieves the most visually appealing results. Zoom in for best view.
Refer to caption: x5.png

**Table 5: Comparison between different finetuning strategies. "Adapter-B" means no extra finetuning. "-2k" and "-2h" means finetuning with extra 2000 and 200 steps, respectively. Bit Acc is evaluated under combined attacks.**
|  | Bit Acc | PSNR | SSIM | FID |
| --- | --- | --- | --- | --- |
| Adapter-B | 0.93 | 32.8 | 0.94 | 2.7 |
| Adapter-F-2k | 0.93 | 33.0 | 0.95 | 2.7 |
| Adapter-I-2k | 0.91 | 34.8 | 0.96 | 2.5 |
| Adapter-V-2h | 0.94 | 29.1 | 0.86 | 3.9 |
| Adapter-V-2k | 0.94 | 29.9 | 0.87 | 3.1 |

### 4.5 Qualitative Results

We qualitatively compare our WMAdapter with Stable Signature [] in Fig. 6. We can find that both Stable Signature and ours perform well on the image with rich textures (column a). However, we observe that Stable Signature tends to produce green/yellow spot-like artifacts (columns bcdeg) as pointed by the yellow arrows. This may be a side effect by finetuning the VAE decoder. As we did not observe such artifacts in Adapter-F, but found slight signs in Adapter-I, which also finetunes the VAE decoder. Since Adapter-I uses the original VAE decoder during inference, artifacts are greatly suppressed. This spot-like artifact can also be observed in other works [] that also finetune the VAE decoder. By comparing column (e), we can see that our adapters produce sharper images, which is also supported by the PSNR metric. By comparing columns (df), we can see that Stable Signature has a slight reddish tint. Our adapters keep the tint more accurately. Please also see the "Green Apple" example in Fig. 11. Column (g) shows the failure case of Adapter-F. It creates artifacts around the plane. This may happen when the background and foreground objects are of uniform color with little textures. In short, compared with StbaleSignature that finetunes VAE decoder to embed watermarks, WMAdapter can generate sharper images with higher watermark invisibility.

Figure: Figure 6: Comparison between WMAdapter and StableSignature []. Yellow arrows point to the generated artifacts. Best viewed in color and zoomed in.
Refer to caption: x6.png

## 5 Conclusion and limitation

In this paper, we present the WMAdapter, a plug-and-play watermark plugin that enables latent diffusion models to embed arbitrary bit information during image generation. Our adapter is lightweight, easy to train, and offers greater flexibility, comparable robustness, and higher image quality compared to previous diffusion watermarking counterparts due to its dedicated design. However, we find one variant Adapter-F tends to produce noticeable artifacts when watermarking images with homogenous background colors. In summary, WMAdapter provides a simple yet effective baseline for further exploration on diffusion watermarking.

## Appendix A Appendix

### A.1 Experiment statistical significance

For the first training stage, we ran 3 independent training and found the standard deviation of average validation bit accuracy across 3 runs to be $0.0006$, and the standard deviation of validation PSNR to be $0.03$ dB.

For the second finetuning stage, we also ran 3 independent trials. The standard deviation of average validation bit accuracy across 3 runs was also $0.0006$, and the std of validation PSNR was $0.04$ db. The small standard deviation at both stages demonstrates the stability of our method. Since the standard deviation is too small to be clearly viewed in Fig. 7, we report the numbers in text.

### A.2 Broader impacts

The proposed diffusion watermarking technique offers significant positive societal impacts, such as enhancing copyright protection for digital creators and helping to prevent the spread of fake news by enabling the authentication of images. However, it also poses potential negative impacts, including privacy concerns, the risk of misuse for malicious purposes, technical challenges that may disadvantage smaller creators, and possible degradation of image quality. Balancing these benefits and drawbacks is crucial to ensure the responsible and effective use of this technology.

In terms of applications, our proposed WMAdapter can also be directly applied to video generation models such as AnimateDiff [] and StableVideoDiffusion [], which share the same VAE architecture as image Diffusion models. We leave further exploration on video to the future work.

### A.3 Evaluation on various distortion intensities

Fig. 7 evaluates our method under larger ranges of distortion intensities and more attacks. We can see that our adapters remain comparable robustness to Stable Signature [] over range of attack intensities. Note that all three methods exhibit limited robustness to significant Gaussian noise. This limitation arises because the pretrained HiDDeN checkpoint [] was not specifically trained to handle noise attacks. To provide a comprehensive evaluation of the different methods, we still present their results under Gaussian noise.

Figure: Figure 7: Robustness comparison over various distortion intensities.
Refer to caption: x7.png

### A.4 Results on different VAEs

We train several watermark adapters for VAEs used by SD1.5&2.1 [], SDXL [] and DiT [] (kl-f8-mse) at resolution $512\times 512$. We compare the adapters before the finetuning stage. Tab. 6 shows the results. We observe that WMAdapter consistently performs well across various VAEs, making it applicable in a wide range of contexts. The PSNR of SDXL adapter is lower compared to SD2.1 and DiT VAE. This may be caused by the resolution mismatch.

**Table 6: Evaluation on VAEs from different models.**
|  | SD1.5 | SD2.1 | SDXL | DiT |
| --- | --- | --- | --- | --- |
| Bit Acc | 0.99 | 0.99 | 0.99 | 0.99 |
| PSNR | 32.1 | 32.8 | 31.2 | 32.4 |

We further evaluate the ability of WMAdapter to zero-shot transfer to different VAEs. Specifically, we apply the adapter trained on SD2.1 directly to SD1.5 VAE and find that it is able to handle SD1.5 image latents with little performance loss. This empirical result demonstrates the zero-shot transfer potential of WMAdapter to different customized SD VAEs.

### A.5 Visualization of attacks

Fig. 8 shows different attacks evaluated in the paper.

Figure: Figure 8: Visualization of different augmentations.
Refer to caption: x8.png

### A.6 More qualitative results on COCO dataset

Fig. 9 shows the watermarked images and their difference with the original images. We find that both WMAdapter-F and WMAdapter-I can adaptively embed watermark information into regions with significant color variations and richer textures in the images, significantly enhancing their invisibility.

### A.7 Generalization to Ideogram dataset

Fig. 10 shows our results on images generated by Ideogram []. These images exhibit completely different styles. However, our WMAdapter, trained on COCO, transfers seamlessly to them.

### A.8 Comparison with other methods

Fig. 11 compares different watermarking methods on COCO [] val images. We observe that the traditional method DwtDct [] introduces minimal modifications to the original image, which consequently makes it sensitive to various attacks. SSL [] produces visible artifacts that significantly degrade visual quality. Following [], we employ the post-hoc JND (Just Noticeable Difference) mask [] for HiDDeN [] to enhance its invisibility. Note that HiDDeN imprints watermark perturbation even to uniform black backgrounds in images ‘Apple’. In contrast, our approach automatically learns to apply stronger modifications only to edges or texture-rich regions, achieving results similar to the post-hoc mask. Importantly, neither of our adapters attempts significant alterations in the black background of the ‘Apple’ image.

Fig. 12 and Fig. 13 compares WMAdapter and two recent watermark plugin methods RoSteALS [] and AquaLoRA [] on SD generated images. We can see that RoSteALS tends to produce visible patch-like artifacts and AquaLoRA introduces visible purple biases. WMAdapter produces images that are most consistent with the original images and have the least artifacts.

Figure: Figure 9: Qualitative results on COCO dataset at resolution 512.
Refer to caption: x9.png

Figure: Figure 10: Qualitative results on Ideogram [] at resolution 512.
Refer to caption: x10.png

Figure: Figure 11: Qualitative comparison on MS-COCO with other invisible watermarking methods. For HiDDeN results [], we use the model trained by [] with inference time modulation by a perceptual just noticeable difference (JND) mask to enhance invisibility []. Zoom in for best view.
Refer to caption: x11.png

Figure: Figure 12: Qualitative comparison with other diffusion watermark plugins.
Refer to caption: x12.png

Figure: Figure 13: Qualitative comparison with other diffusion watermark plugins.
Refer to caption: x13.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: Please refer to the weakness.
soundness: 3
strengths: 1. WMAdapter is compatible with various diffusion models and watermarking decoders, enhancing its adaptability for real-world applications.
2. The hybrid fine-tuning scheme is innovative, effectively reducing artifacts.
summary: This paper presents WMAdapter, a plug-in solution for diffusion-based watermarking. The authors propose training a contextual adapter and introduce a hybrid fine-tuning scheme to mitigate grid artifacts. Experimental results highlight WMAdapter's advantages, particularly in terms of visual quality.
weaknesses: 1. The mechanism behind hybrid fine-tuning is not fully explained. After fine-tuning the adapter with the VAE decoder, it is unclear why it achieves better quality with the original decoder (Adaptor-I in Table 5), yet produces degraded reconstructions with the fine-tuned decoder (Adaptor-V in Table 5). Further clarification is needed.

2. Although WMAdapter demonstrates superior visual quality, its robustness is weaker than that of StableSignature (Figure 8) and SSL (Table 2). Given the trade-off between distortion and robustness, a more balanced comparison could involve evaluating all methods at comparable visual quality levels to assess robustness fairly.

3. The bit length used by methods like RoSteALS is not specified. Do these methods also embed a consistent 48 bits? If so, are they trained on this bit length for a fair comparison?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The watermarking pipeline shares similarities with Stable Signature, particularly in its foundational approach. However, while Stable Signature fine-tunes the Diffusion U-Net, this study emphasizes modifications to the VAE decoder. Additionally, the concept of integrating an external adaptor is not novel. It is recommended to explicitly highlight the distinctions between this work and prior methods, such as Stable Signature and ControlNet, to better establish its unique contributions.

2. For robustness part, most perturbations evaluated in experiments are common image post-processing methods, such as JPEG and Crop. Apart from surrogate detecor attack and regeneration attack, there are more adaptive attacks such as query-based black-box attack or white-box attack. It is important to consider a comprehensive setting.

3. In terms of robustness, it is observed that the bit accuracy drops to approximately 0.9 when JPEG compression at a quality level of 80 is applied. Is this reduction sufficient for practical robustness? Given that JPEG 80 is considered a moderate perturbation, it would be valuable to analyze the system's response to more severe JPEG compression levels to understand the potential vulnerabilities if an attacker deploys more aggressive degradation techniques.

4. Is there existing literature that defines PSNR values of 32 or 30 as indicative of poor visual quality? This question is relevant since some adaptive attacks result in significant decreases in bit accuracy even when the PSNR remains above 30.
soundness: 2
strengths: 1. The paper is generally well-written and easy to follow.

2. : The idea of Content-Aware Watermarking is knid of interesting and different from previous work: Contextual Adapter adapts watermark embedding based on the image's content, improving the concealment and robustness of watermarks while minimizing visual impact.

3. High Image Quality Preservation.
summary: The paper proposes WMAdapter, a watermarking solution for latent diffusion models that embeds user-specific watermarks without altering the underlying diffusion components, maintaining image quality. Key innovations include a Contextual Adapter that adapts watermark embeddings based on image content and a Hybrid Finetuning strategy to enhance visual quality while preserving watermark robustness.
weaknesses: 1. Limited novelty in watermarking pipeline.

2. Not comprehensive evaluation of robustness in adaptive settings.

3. Dependency on Pretrained Components.

---

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: Please refer to Weaknesses part.
soundness: 3
strengths: This method offers a dynamic watermarking mechanism, allowing watermarking of a given image with any arbitrary key (binary message) without needing retraining to adapt to a new key.
summary: The paper proposes a method for generating watermarked images without visible artifacts by introducing a module called the contextual adapter and a targeted fine-tuning strategy.
weaknesses: 1. **Lack of Baseline Comparisons**:
* Post-hoc watermarking: The authors have shown some comparisons with frameworks like StegaStamp [1] and RivaGAN [2]. However, a more comprehensive evaluation could have been conducted to establish stronger baseline comparisons.
* Native watermarking: Stable Messenger [3], a similar steganography framework to RoSteALS, demonstrates a better trade-off between accuracy and image quality. However, the authors did not provide any comparisons with this approach.

2. **Confusing Contribution**:
The authors claim hybrid fine-tuning as a key contribution of this paper. However, in Table 5, the results of no extra fine-tuning and hybrid fine-tuning primarily demonstrate a trade-off between accuracy and image quality, a common issue in this field. This leaves the exact contribution of this fine-tuning strategy unclear.

3. **Unclear Robustness Evaluation**:
While it's beneficial to see the method's robustness against regeneration attacks, Figure 5 does not clearly demonstrate if WMAdapter is more robust to these attacks than other watermarking methods such as Stable Signature or AquaLoRA.

**References**:

[1] Tancik, Matthew, Ben Mildenhall, and Ren Ng. "Stegastamp: Invisible hyperlinks in physical photographs." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[2] Zhang, Kevin Alex, et al. "Robust invisible video watermarking with attention." arXiv preprint arXiv:1909.01285 (2019).

[3] Nguyen, Quang, et al. "Stable Messenger: Steganography for Message-Concealed Image Generation." arXiv preprint arXiv:2312.01284 (2023).

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: - I believe the experimental setup for the FID metric needs to be reconsidered. FID is used to measure the quality of generated images, and in related work, it is typically calculated between watermarked images and real images from the dataset to assess the impact of the watermark on the model's generation quality. This approach would be more reasonable.
- In section 4.3, "Robustness to More Attacks," this paper only presents results for WMAdapter-F against the Regeneration attack and the Surrogate detector attack. To better evaluate the robustness of WMAdapter, the results for WMAdapter-I should also be included.
- The Hybrid Finetuning strategy is a highlight of this paper; however, the final implementation uses the original VAE decoder instead of the finetuned VAE decoder. While Figure 6 intuitively demonstrates the effectiveness of this approach, I hope the authors can further investigate the reasons behind the occurrence of lens flare artifacts.
soundness: 2
strengths: - The paper is well-written and clearly explained, making it easy for readers to understand.
- A Hybrid Finetuning Strategy is proposed, which further reduces artifacts in the watermark through multiple rounds of fine-tuning.
- WMAdapter is lightweight and easy to deploy, with low training overhead.
summary: Watermarking serves as a prominent means for actively identifying generated content, extensively applied in diffusion model image detection and attribution. Existing methods integrate watermark embedding with image generation but have not considered the relationship between watermark information and image semantics. This paper proposes a watermarking method for diffusion models called WMAdapter, which designs a Contextual Adapter to fuse watermark information with the VAE features in the diffusion model. To further improve the visual quality of watermarked images, the paper also proposes a Hybrid Finetuning strategy. This strategy first fine-tunes the adapter, then simultaneously fine-tunes both the adapter and the VAE decoder, using the fine-tuned adapter with the original VAE decoder during final image generation. Experimental results show that WMAdapter can effectively reduce artifacts.
weaknesses: - The WMAdapter integrates features from the VAE with watermark information, representing a contextual watermark. However, the authors state in line 413 that WADIFF "uses a context-less structure to encode the watermark." I do not believe that WADIFF is a context-less watermark, as it combines latent representations with watermark information and passes them through a trainable UNet layer. Therefore, both WMAdapter and WADIFF can be considered contextual watermarks. I encourage the authors to provide a more comprehensive comparison between the two methods.
- Based on the experimental results (Table.2 and Figure.8), compared to Stable Signature, this paper sacrifices some robustness in order to achieve better visual quality.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. I do not think changing the VAE decoder's parameters and changing the internal embedding of the image are mathematically different. Both of them will affect the quality of the generated images. Given the results shown in Table 2 in which WMAdapter achieves worse robustness results compared to other methods, does it mean that the better image quality of WMAdapter is just simply a trade-off between image quality and robustness?
2. In Figure 5, WMAdapter achieves bad robustness results against adversarial perturbations. Specifically, the bitwise accuracy decreases to almost 50% when the PSNR is around 30, which means the watermark can be easily removed by the attacker in the real-world scenario.
3. For hybrid finetuning, it is confusing that finetuning both the VAE decoder and the watermark module and using only the watermark module during the inference achieves better results than  other variants. I think detailed explanations and more experiments should be included for this part.
4. For Figure 8 in Appendix, the proposed method also shows worse robustness compared to Stable Signature, which also shows the improvement in image quality is only a trade-off.
soundness: 2
strengths: 1. The proposed method maintains better image quality than the existing ones.
2. The robustness of the proposed watermarking method is evaluated on some commonly seen image processing methods.
3. The contextual watermarking idea is interesting.
summary: This paper proposed a method called WMAdapter which introduced an external component to the latent diffusion model to add a contextual watermark onto the generated images. The experiments showed that WMAdapter achieved similar bitwise accuracy to other methods while maintaining better image quality. The robustness of the proposed method is also evaluated on some image post-processing method.
weaknesses: 1. Though the authors claim that they keep the diffusion model intact to get better image quality, their watermarking method changes the internal embedding within the VAE decoder. I think both changing VAE decoder's parameters and changing the internal embedding will hugely affect the quality of the generated image.
2. The robustness of the proposed method against adversarial attacks is worrying.
3. The reason why hybrid finetuning works is not clarified.

Meta-review: This paper proposes a watermarking method for latent diffusion models, which is denoted as WMAdapter with a contextual Adapter to fuse watermark information with the VAE features in the diffusion model. The paper proposes a hybrid finetuning strategy to further improve the visual quality of watermarked images. This strategy first fine-tunes the adapter, then fine-tunes both the adapter and the VAE decoder, using the fine-tuned adapter with the original VAE decoder during final image generation. Experimental results show that WMAdapter can effectively reduce artifacts. The reviewers agree that: (1) The paper is well-written and easy to follow. (2) The proposed wMAdapter is compatible with various diffusion models and watermarking decoders. (3) The hybrid fine-tuning scheme can reduce artifacts to some extent. However, Reviewers still concerned about the trade-off between image quality and watermark robustness: the proposed method demonstrates worse robustness compared to Stable Signature, although it preserves better image quality.  The work does not adequately demonstrate the advantages of the proposed method. One reviewer raised the concern that other reviewers with positive scores also acknowledged during the discussion.  Meanwhile, modifying the pre-trained VAE is also not explained well. We have to reject this work due to these concerns and encourage the resubmission with improved version.

Decision: Reject