WMAdapter: Adding WaterMark Control to Latent Diffusion Models
Authors: Hai Ci
Yiren Song
Pei Yang
Jinheng Xie
Mike Zheng Shou
Venue: ICML 2025 poster (2025)
Keywords: watermark
latent diffusion model
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: Watermark LDM via a plugin without compromising image quality

Abstract: Watermarking is essential for protecting the copyright of AI-generated images. We propose WMAdapter, a diffusion model watermark plugin that embeds user-specified watermark information seamlessly during the diffusion generation process. Unlike previous methods that modify diffusion modules to incorporate watermarks, WMAdapter is designed to keep all diffusion components intact, resulting in sharp, artifact-free images. To achieve this, we introduce two key innovations: (1) We develop a contextual adapter that conditions on the content of the cover image to generate adaptive watermark embeddings. (2) We implement an additional finetuning step and a hybrid finetuning strategy that suppresses noticeable artifacts while preserving the integrity of the diffusion components. Empirical results show that WMAdapter provides strong flexibility, superior image quality, and competitive watermark robustness.

WMAdapter:
Adding WaterMark Control to Latent Diffusion Models
Hai Ci' Yiren Song
Pei Yang'
Jinheng Xie Mike Zheng Shou
Abstract
ages in post-hoc way through frequency domain transfor- mations (Cox et al., 2007; Lin et al., 2001; Xia et al., 1998) or encoder-decoder networks (Zhu et al., 2018; Tancik et al., 2020; Zhang et al., 2019). However, in the context of wa- termarking diffusion images, post-hoc methods introduce additional workflows and unable to fully leverage the rich latent space provided by the image generation process. Re- cently, more efforts (Zhao et al., 2023b; Fernandez et al., 2023; Min al., 2024; Xiong al., 2023; Lei et al., 2024; Meng et al., 2024; Yang et al., 2024b; Ci et al., 2024) have focused on leveraging the characteristics of the diffusion process to seamlessly integrate watermarking into the dif- fusion pipeline, known as diffusion- watermarking. Among these, Stable Signature (Fernandez et al., 2023) pro- posed a method that fine-tunes the VAE decoder of a latent diffusion model (Rombach et al., 2022) using a pretrained watermark decoder (Zhu et al., 2018) 8). This approach has shown promising results. However, it requires fine-tuning a separate VAE decoder for each unique watermark, making it difficult to scale to millions of keys as required in large-scale commercial scenarios where each user may need a unique key. Additionally, the tuning of VAE decoder on small amount of data results in blurry and lens flare-like artifacts (see Fig. 7).
Watermarking is essential for protecting the copy- right of Al-generated images. We propose WMAdapter, a diffusion model watermark plu- gin that embeds user-specified watermark infor- mation seamlessly during the diffusion genera- tion process. Unlike previous methods that mod- ify diffusion modules to incorporate watermarks, WMAdapter is designed to keep all diffusion com- ponents intact, resulting in sharp, artifact-free im- ages. To achieve this, we introduce two key in- novations: (1) We develop a contextual adapter that conditions on the content of the cover image to generate adaptive watermark embeddings. (2) We implement an additional finetuning step and hybrid finetuning strategy that suppresses no- ticeable artifacts while preserving the integrity of the diffusion components. Empirical results show that WMAdapter provides strong flexibility, su- perior image quality, and competitive watermark robustness. Code: https: : //gi //gg/tttuu thub. com/ Showlab/MMAAaptre
1. Introduction
Recent works (Bui et al., 2023; Xiong et al., 2023; Min et al., 2024; Meng et al., 2024; Zhang et al., 2024; Kim et al., 2023; Nguyen et al., 2023) have explored watermark plugins for diffusion models. These plugins accept arbitrary watermark keys and generate watermark embeddings with- out requiring per-watermark finetuning, thereby addressing the scalability issue. However, these methods typically gen- erate watermark embeddings without considering the image content (Kim et al., 2023; Xiong et al., 2023; Bui et al., 2023) (i.c., they are context-less) and often require fine- tuning or modifying diffusion modules to incorporate the watermark embeddings (Kim al., 2023; Xiong et al., 2023; Feng al., 2024) Tab. compares several watermarking methods. Unfortunately, finetuning the original diffusion pipeline or making intrusive modifications often leads to significant drop in image quality, resulting in blurriness or noticeable artifacts. Fig. illustrates the image quality of different methods, where artifacts introduced by other methods are evident. Find more examples in Fig. 12.
With the widespread adoption of difffusion models (Ho et al., 2020; Podell al., 2023; Song et al., 2020; Rombach et al., 2022; Ci et al., 2023; Zhang et al., 2023a; Wang et al., 2024), diffusion-generraee images are proliferating across media and the internet. While these models meet the demand for high-quality creative content, their misuse raises signifi- cant concerns about copyright protection and the security of images against deepfakes (Westerlund, 2019; Song et al., 2024b;a). Watermarking technology (Cox et al., 2007) pro- vides a tailored solution for resolving copyright disputes and identifying the sources of forgeries.
Previous watermarking methods added watermarks to im-
'Show Lab, National University of Singapore, Singapore. Correspondence to: Mike Zheng Shou <mike.zheng shou @ gmail.com>.
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
We propose innovative watermark plugin solution

---

WMAdapter: Watermarking Latent Diffusion Models
AquaLoRA
WOUAF
RoSteALS
SSL
WMAdapter
1.0
Better
0.9
Bit Accuracy
X
0.8
HiDDeN
Better
SSL
0.7
StableSig
WOUAF
0.6
RoSteALS
WMAdapter-F
0.5
WMAdapter-
2
4
FID
Figure 1: WMAdapter introduces minimal artifacts, providing better caturacy-qualii, tradeoff.
Table 1: Comparison of several diffusion watermarking methods. They all tend to introduce noticeable artifacts or produce
blurry images.
Modified Diffusion Modules
Scalable
Imperceptible
AquaLoRA (Feng et al., 2024)
UNet Backbone
StableSig (Femandez et al., 2023)
VAE Decoder
XXL X
WOUAF (Kim al., 2023)
VAE Decoder
RoSteALS (Bui et al., 2023)
No
Ours
No
WMAdapter (Fig. 2). Its core design philosophy focuses on preserving the integrity of the original diffusion pipeline to produce high-quality images. We do not modify any pa- rameters of the pretrained diffusion modules. So how do we conceal the watermark information and ensure its robust- ness? We introduce two key innovations: (1) We propose a novel Contextual Adapter structure that conditions on the cover image features generate content-aware watermark embeddings (hence "contextual"). Intuitively, this allows the adapter to better identify areas of the image that are more suitable for hiding the watermark, enhancing concealment and robustness. To fully leverage diffusion features while reducing computational overhead, our Contextual Adapter extracts image features from the intermediate layers of the diffusion VAE decoder. Unlike ControlNet plugins (Zhang al., 2023b; Min al., 2024), which use heavy UNet structure (Ronneberger al., 2015), the Contextual Adapter is lightweight, totaling only 1.3MB in parameters, and ables watermarking image in just 30ms. (2) We introduce an additional finetuning stage with novel Hybrid Finetun- ing strategy to further enhance image quality. To preserve the original diffusion modules, our Hybrid Finetuning strat- egy involves jointly finetuning the adapter and the diffusion VAE decoder during training for alignment, and then using the original VAE decoder during inference. This approach effectively suppresses noticeable artifacts and significantly improves image sharpness. We summarize our contributions
as follows:
1. We introduce WMAdapter, novel diffusion water- marking solution with an innovative design philosophy. embeds watermarks non-intrusively during the dif- fusion process, thereby preserving the integrity of the diffusion pipeline and producing high-quality images.
Methodologically, we propose Contextual Adapter and Hybrid Finetuning to achieve non-intrusive wa- termarking, ensuring both watermark robustness and generation quality.
Experimental results demonstrate that WMAdapter ef- fectively suppresses noticeable artifacts and offers bet- ter accuracy-qualit tradcoff's compared to prior post- hoc and diffusion-native watermarking methods.
2. Related Work
2.1. Post-hoc Watermarking
Post-hoc methods include traditional frequency domain transformation methods (Cox al., 2007), optimization- based methods (Fernandez et al., 2022b; Kishore et al., 2021), and encoder-decoder methods (Zhu et al., 2018; Tan- cik et al., 2020; Jia et al., 2021; ; Sander et al., 2024). Dif- ferent methods have different aims. For instance, Kishore et al. (2021) emphasizes hiding more bits, Zhu al. (2018)
2

---

WMAdapter: Watermarking Latent Diffusion Models
User input
1
1
0
Context-less
Adapter
Watermark bits
Image feature
Pretrained watermark decoder
Noise
Diffusion UNet
VAE decoder
Decoded bits
Figure 2: Framework overview. WMAdapter plugged onto the VAE decoder. takes user input watermark bits and image features from the VAE decoder, imprinting the watermark on-the-fly during VAE decoding. In contrast, traditional context-less adapters take only watermark conditions as input. The image and icons credit to (Freepik-Flaticon, 2024).
and Jia et al. (2021) prioritizes robustness against JPEG compression.
3.1. Framework Overview
Fig.2 illustrates the overall framework of WMAdapter. WMAdapter is a plug-and-play watermark module that can be directly attached to the VAE decoder of a latent diffusion model (Rombach et al., 2022). It imprints the watermark during image generation, seamlessly integrating into the dif- fusion generation workflow. WMAdapter employs novel contextual adapter structure, which takes both watermark bits and image features from the VAE decoder as input and outputs feature residuals containing watermark information. Watermarked images can be directly fed into pretrained watermark decoder, such as HiDDeN (Zhu et al., 2018), to retrieve the watermark information.
2.2. Diffusion Native Watermarking
According to the location of the watermark, we classify diffusion-native watermarking methods into two categories. Adding to initial noise: Tree-Ring (Wen et al., 2023) adds watermarks to the frequency of initial noise, achieving re- markable robustness. Subsequent methods (Yang et al., 2024b; Ci et al., 2024; Lei et al., 2024) improves its multi- key identification capabilities. However, these methods significantly alter the layout of the generated images, which is not desirable in some production scenarios. Adding to latent space: Other methods leverage the latent space of the VAE (Bui al., 2023; Meng al., 2024; Zhang al., 2024; Xiong al., 2023; Kim al., 2023; Fernandez al., 2023) diffusion backbone (Feng et al., 2024). However, they either generate content-agnostic watermark embeddings modify the original diffusion modules, often resulting in lower image quality. contrast, WMAdapter prioritizes image quality through novel contextual designs while pre- serving the integrity the entire diffusion pipeline. Stable Messenger (Nguyen al., 2023) is recent method that also generates content-aware watermarks. However, they mainly focus on improving message accuracy and their model de- sign is different from ours.
The training of WMAdapter consists of two stages: large- scale training and fast finetuning. In the training stage, we freeze the VAE decoder and the watermark decoder and train only the Adapter on a large scale dataset. We then finetune the Adapter and VAE decoder on a small amount of data. Specifically, we present novel hybrid finetuning strategy that is able suppress tiny artifacts and significantly enhance generation quality. We also discuss several different strategies concerning different tradeoffs between robustness and quality.
3.2. Contextual Adapters
In this section, we provide detailed overview of the con- textual structure of WMAdapter. Fig. 3 (Left) illustrates the internal structure of WMAdapter, which comprises a series of independent Fuser modules. Each Fuser ox(-) is attached before a corresponding VAE decoder block It receives both VAE feature fa and watermark bits w inputs, and
3. Method
In this section, we will introduce the framework of WMAdapter, detail its contextual structure, and discuss the training and fine-tuning strategies.
3

---

WMAdapter: Watermarking Latent Diffusion Models
Embedding
[48, h, w]
11001...
W MAdapter
[48,1,1]
Fuser
Fuser
Fuser
11001
MLP
expand
residual
Conv
Mid BIk
Conv
Blk
Features
Ic, h, w]
[c+48, h, w]
[c, h, w]
VAE decoder
Fusing
Figure 3: The architecture of WMAdapter. Left: The structure of WMAdapter. It comprises several independent Fusers with identical structures. Right: The structure of Fuser. It consists of a watermark Embedding module and a Fusing module.
outputs feature residual y/ to update fi. Formally,
converge in just 1-2 epochs. Note that this will not bring se- rious security risks, because there are hundreds of different open-source decoders. We use two types of losses objective: the consistency loss between the watermarked image The and the unwatermarked image and the accuracy of decoded bits. The total loss function is defined as:
y: di wft, =
We put a total of 6 Fusers before the Conv Block, Middle Block and four Up Blocks in the kI-f8 VAE decoder used by Stable Diffusion (Rombach et al., 2022).
Fig. 3 (Right) illustrates the internal structure of an Fuser. An Fuser consists of two main components: the Embedding module and the Fusing module. The Embedding module maps the 01 bit sequence into a 48-dimensional watermark feature vector. This feature vector is then expanded along the width and height dimensions to produce watermark feature map with the same dimensions the image feature. The image feature and watermark feature are concatenated along the channel dimension and fed into the Fusing mod- ule, which outputs the image feature residuals. Keeping lightweight in mind, we use two MLPs with 256 intermedi- ate feature channels for the Embedding module, and two convolutions with half the image feature channels as in- termediates for the Fusing module. We employ LeakyReLU as the non-linearity. The total parameters of WMAdapter are only 1.3M, making it a small and efficient plugin.
C Xn Lmac (Ir, + A2 Sipips (Ir, Fww) + Xahung (1, + da Lthn (w, w')
(2)
where the first three terms represent image consistency losses. We use MAE and LPIPS loss (Zhang et al., 2018) to maintain consistency with VAE pretraining (Rombach et al., 2022). Additionally, loss (Czolbe et al., 2020) similar to Stable Signature (Fernan- dez et al., 2023) to enhance human visual preference. For watermark decoding accuracy, we use binary cross-entropy loss bewteen decoded bits w' and input bits w. We empiri- cally set ' A2 Ag. A4 to 0.2, 0.2, 0.08, 1.0, respectively.
include Watson-VGG
3.4. Hybrid Finetuning
After the training stage, we obtain a watermark adapter that performs well in both accuracy and image quality (Sec. 4.4.2). However, when zoom in the gener- ated images, grid-like artifacts can sometimes be observed (Fig, 6). To further improve image quality and eliminate these tiny artifacts, we introduce a fine-tuning stage on small amount of data, On top of the first stage training losses, we incorporate an additional total variation loss (et al, 2024) on the watermarked images to enhance smoothness, setting its weight 0.02.
3.3. Training
In the training stage, we use a pretrained watermark de- coder to decode watermark bits from the watermarked im- ages. We freeze the watermark decoder and the VAE de- coder, and only train the Adapter. Why do we use pre- trained decoder instead of training a watermark decoder from scratch along with the Adapter? We observe that training an encoder/decoder pair from scratch, as post-hoc methods do, typically requires significant training effort. For example, HiDDeN takes 300 epochs to converge on the COCO dataset. The situation gets worse when trained with a diffusion pipeline. WOUAF (Kim et al., 2023) takes about 10 days. Using a pretrained post-hoc decoder facili- tates efficient knowledge transfer, allowing WMAdapter to
Further, present novel Hybrid Finetuning strategy. Con- cretely, we finetune both the Adapter and the VAE decoder, but use the fine-tuned Adapter and the original VAE decoder for inference. Fig. 4 distinguishes this strategy from two other classic finetuning strategics: Fixed and Joint Finetun- ing. The Fixed Finetuning strategy uses the same training

---

WMAdapter: Watermarking Latent Diffusion Models
Adapter
F Fixed Finetuning
0
Adopter
oint Finetuning
1
Addapter
Hybrid Finetuning
Latent
VAE decoder
P retrained waternark decoder
Latent
VAE decoder Pretrained watermark decoder
Latent
VAE decoder
P Pretrained
Finetuning Inference waternak decoder
Figure 4: Illustration of different finetunig strategies. They differ in how to treat the VAE decoder.
approach as in the first stage, fixing the VAE decoder and quickly finetuning the Adapter with a high learning rate. The Joint Finetuning strategy jointly finetunes the Adapter and the VAE decoder, using both finetuned copies for inference.
nandez et al., 2023). ALL training and finetuning steps are performed on MS-COCO 2017 (Lin et al., 2014) training set. Validation is performed on COCO 2017 validation set. We train and evaluate our adapters on images resolution 512 512. For images smaller than this size, we resize their shorter edge 512, then center crop get 512 512 image.
Sec. 4.4.2 will give side-by-side comparison between these three finetuning strategies. In short, Hybrid Finetuning can effectively suppress noticeable artifacts and, by keeping the VAE intact, produces the sharpest and clearest images while maintaining the plug-and-play advantage, making it ideal for commercial image generation products which require high image quality.
Training strategies For the first stage training, we adopt x NVIDIA A,5000 GPUs of 24 GB memory, with per-GPU batchsize of 2, AdamW optimizer (Loshchilov & Hutter, 2017), learning rate of 5e-4. We train the model for 2 epochs, taking about 5 hours. For the second stage finetun- ing, we use a single A5000 GPU. We set the mini-batch to 2. We also use the AdamW optimizer and a start learning rate of 5c-4. However, we adopt a per-step cosine learning rate decay with 20 warm-up steps. Unless otherwise spec- ified, the total fine-tuning process defaults to 2,000 steps, lasting for about 50 minutes. Different finetuning strate- gies result in several different adapter variants. We use Adapter-B. B, Adapter-F, Adapter- V, and Adapter-/ to denote the adapters obtained by No Finetuning, Fixed Finetuning, Joint Finetuning and Hybrid Finetuning, respectively.
3.5. Discussion
WMAdapter is designed with a strong emphasis on image quality, particularly in suppressing noticeable artifacts in generated images. We introduce the Contextual Adapter and the Hybrid Finetuning, non-intrusive watermarking methods that achieve this goal by preserving the integrity of the diffusion pipeline. This fundamentally distinguishes our approach from other diffusion watermarking methods that embed watermarks at the expense of image quality and introduce noticeable artifacts. We want to highlight the importance of high-quality, artifact-fres watermarked im- user wants receive
ages for generative products, images with visible flaws. The Experiment Section demon- strates that our method successfully combines scalability, high-quality image generation, and watermark robustness.
Evaluation metric Following previous conventions (Zhu et al., 2018; Fernandez et al., 2022b; 2023), we use average bit accuracy to evaluate the watermarking performance of our adapter. Bit accuracy is defined by the ratio of correctly decoded bits in 48-bit watermark sequence. Apart from the bit accuracy, we also report the tracing accuracy among different numbers of users following concurrent works (Min et al., 2024; Ci et al., 2024). We adopt the evaluation pro- tocol of (Min et al., 2024). Concretely, we construct user pools of different sizes, ranging from 10* to 10 to evaluate the accuracy of user tracing at different scales. Each user assigned unique key. For each user pool. randomly se- lect 1,000 users and watermark 5 images per user, resulting in 5,000 watermarked images. For each of the 5,000 images, we find the best match among the user pool and check it's a correct match. Tracing accuracy is then averaged over all 5,000 images. To evaluate the detection performance,
4. Experiments
4.1. Experimental Setup
Model and dataset We experiment with a popular latent diffusion model Stable Diffusson 2.1 (Rombach et al., 2022) and its associated kI-18 VAE, We adopt the pretrained water- mark decoder from HiDDeN (Zhu et al., 2018). The check- point we use was pretrained by (Fernandez et al., 2023), encoding 48-bits watermark information. This checkpoint also used to finetune Stable Signature (Fernandez al., 2023). Thus, our adapter can be directly compared with (Fer-
5

---

WMAdapter: Watermarking Latent Diffusion Models
we report TPR @ @FPR00 Concretely, we assume the bits decoded from the natural images following Bernoulli distri- bution with parameter 0.5. Then the number of matched bits M follows a binomial distribution with parameters (48, 0.5). So we have the false detection rate as a function of threshold
ing the top-performing methods by only 0.01 and 0.03, re- spectively, while still maintaining competitive robustness. Overall, WMAdapter achieves better robustness-qualit tradeoff. which can be seen in Fig. 1 (right).
T: (T) = P (M > -) = Io.s (T + 1, 48 - r), where is the incomplete beta function. We control FPR = 10-6 and calculate the corresponding T, then we evaluate TPR with this threshold.
Tracing accuracy Since certain watermarking methods, such as Wen et al. (2023), don't incorporate the concept of bits or use tracing accuracy as an alternative evaluation protocol (Min et al., 2024), we further compare the tracing accuracy in Tab. 3. We can see that our adapters achieve nearly perfect tracing accuracy with different scales of users. Tree-Ring (Wen et al., 2023) achieves zero tracing accuracy due to its design flaws uncovered by Ci et al. (2024). WAD- IFF (Min al., 2024) is a concurrent effort, which employs HiDDeN decoder to finetune UNet watermark plugin for diffusion models. We can see that its tracing accuracy grad- ually drops as the scale grows despite they employ a heavier adapter (~9900MB params). Both ours and Stable Signature perform consistently at different user scales. Notably, Stable Signature has higher average bit accuracy but gets slightly worse tracing accuracy than ours. We attribute this to its larger performance variance among different keys.
In addition to accuracy measurements, we are also inter- ested in the watermark's s invisibility and image generation quality. We report the Peak Signal-Noise-RRnn Ratio (PSNR) between images before and after watermarking and Fréchet Inception Distance (FID) (Heusel et al., 2017) between wa- termarked images and images from coco val set. Typically, higher PSNR leading to sharper and clearer images. While lower FID means the watermarked images have higher fi- delity and more closely resemble the real images in terms of appearance and variety.
4.2. Comparison With Other Methods
Accuracy and image quality We compare our method with three post-hoc watermarking methods SSL (Fernandez et al., 2022b), StegaStamp (Tancik et al., 2020), and HiD- DeN (Zhu et al., 2018). SSL. bases on iterative optimization to get the watermark, while StegaStamp and HiDDeN are encoder-decoder based methods. For HiDDeN, we use the model provided by (Fernandez al., 2023), which is hanced with a JND mask (Fernandez et al., 2022a) for better image quality. We also compare with three recent diffusion- native watermarking methods RoSteALS (Bui et al., 2023), WOUAF (Kim et al., 2023) and Stable Signature (Fernandez et al., 2023). Note that all these methods do not alter the image layout during watermarking.
Summary Unlike other methods with significant draw- backs-such as RoSteALS, SSL, and WOUAF, which introduce noticeable artifacts and result in significantly lower FID scores, or StableSignature, which lacks scala- bility-our our approach delivers high image quality, scalability, and competitive accuracy simultaneously. In all three as- pects, WMAdapter-/ consistently outperforms HiDDeN, providing a better overall tradeoff.
4.3. Robustness to More Attacks
Other transformations and intensities Fig. 8 evaluates against more image transformations and intensities. Our adapters achieve comparable performance to the baseline Stable Signature under various levels of attacks, while offer- ing flexibility, scalability and higher image quality.
As shown in Tab. 2, WMAdapter-/ achieves the best im- age quality among all methods, excelling in both PSNR and FID. Its PSNR and FID improve over the baseline, Stable Signature, by approximately 17% and 22%, respec- tively. In contrast, Stable Signature produces blurrier im- ages with lens flare artifacts (Sec. 4.5) due to fine-tuning of the VAE decoder, resulting in lower PSNR and FID scores. WMAdapter-/ shows even greater improvements compared to SSL (5% and 83%), RoSteALS (14% and 55%), and WOUAF (38% and 81%), as these methods introduce larger artifacts greatly degrading quality metrics (See Fig. 12 for artifacts).
Regeneration attack Recent work (Zhao al., 2023a; Liu et al., 2024) has demonstrated the potential of regen- eration attacks in watermark removal. We evaluate the robustness of WMAdapter against three different regen- eration methods introduced in Zhao et al. (2023a): one diffusion- based (Zhao et al., 2023a) and two VAE-based methods (Ballé et al., 2018: Cheng et al., 2020). For Ballé et al. (2018); Cheng al. (2020), we assess performance at compression rates of 1-6 and 1-8, respectively. Fig. 5 presents the Accuracy-PSNe curve. We observe that the three regeneration attacks require a PSNR drop of 4-6 dB to successfully remove our watermark. In contrast, only a 2 dB reduction in image quality is needed to remove the watermark of Stable Signature. This demonstrates that our
In terms of watermark detection performance, our meth- ods achieve perfect TPR, outperforming HiDDeN, WOUAF, and Stable Signature. For bit accuracy, while SSL excels in single attack scenario, it is more sensitive to combined attacks. Both WMAdapter-F and WMAdapter-/ surpass SSL, HiDDeN and RoSteALS under combined attacks, trail-
6

---

WMAdapter: Watermarking Latent Diffusion Models
Table 2: Comparison with other watermarking methods on generation quality and robustness. All methods are evaluated on COCO 2017 val set (Lin et al., 2014) with image size 512 x 512. Since Stable Signature (Femnandez et al., 2023) requires finetuning of separate VAE decoders to embed different keys, we report its average results on 10 randomly sampled keys. We report TPR@ FFR10 6 for detection performance. For robustness, we use Crop 0.3, JPEG 80, Brightness 1.5.
Bit Accuracy
Method
PSNR 1
FID
TPR
None
JPEG
Crop
Bright
Comb
SSL.
33.0
14.8
1.00
1.00
0.99
0.97
0.98
0.88
Post
HiDDeN
34.1
3.1
0.99
0.98
0.84
0.97
0.98
0.85
StegaStamp
29.3
9.9
1.00
0.96
0.96
0.49
0.94
0.49
RoSteALS
30.4
5.5
1.00
0.99
0.99
0.50
0.96
0.50
Native
WOUAF
25.3
13.5
0.97
0.99
0.99
0,94
0.97
0.93
Stable Signature
29.7
3.2
0.99
0.99
0.93
0.99
0.99
0.93
WMAdapter--F
33.1
2.7
1.00
0.99
0.92
0.99
0.99
0.92
WMAdapter-1
34.8
2.5
1.00
0.98
0.90
0.97
0.97
0.90
Table Accuracy of tracing different numbers of keys. All methods are evaluated on COCO dataset (Lin et al., 2014). For WADIFF (Min et al., 2024), the number is reported by its original paper.
Query-based attack Another common black-box attack is the query-based attack, which defines a blending process that transitions from random image to a given watermarked image. During this process, it repeatedly queries the water- mark decoder API to determine whether the current blended image contains watermark, aiming identify the image with the minimal perturbation that successfully removes the watermark. We adopt the WEvade-B-O approach from Jiang et al. (2023) and set the detection threshold r to control FPR 10-6 Our observations show that the query-based attack can successfully evade watermark detection for both WMAdapter and Stable Signature, achieving a success rate of 1.0 (i.e., TPR 0). However, this method results in significant image quality degradation, with the final attacked images averaging a PSNR of approximately 8 dB.
Method
Trace 104
Trace 10a
Trace 106
WADIFF
0.982
0.968
0.934
Tree-Ring
0.000
0.000
0,000
Stable Signature
0.999
0.999
0.998
WMAdapter-F 1.000
1.000
1.000
WMAdapter-/
1.000
0.999
0.999
method exhibits better robustness against regeneration at- tacks.
Steganalysis attack Yang et al. (2024a) propose averag- ing multiple watermarked images to extract aontent-agnostic watermark patterns for removal or forgery. However, the contextual adapter in WMAdapter adapts watermark pat- terns based on image layout, making it naturally robust to this type of srtucck-ccheving no bit accuracy drop on a 5k image averaging evaluation.
Adversarial attack Adversarial attack relies on PGD (Madry, 2017) optimization to generate adversarial noise targeting the watermark decoder. Based on access to the watermark decoder, these attacks are categorized as white-box and black-box. In blxck-box settings, a binary classifier is trained to identify watermarked images, and adversarial noise is then optimized mislead this classifier, disrupting the watermark. This is commonly referred to as surrogate detector attack (Saberi et al., 2023; Jiang et al., 2023; An et al., 2024; Lukas et al., 2023). We follow the implementation of An et al. (2024) and demonstrate our method's robustness against both white-box (An' '2-wwb A) and black-box attacks (An'24 V) in Fig. 5. Notably, both WMAdapter and Stable Signature exhibit strong robustness against black-box adversarial attacks, with a bit accuracy drop of about 0.02 and TPR drop less than 0.01. In white-box scenarios, where attackers have full access to the watermark decoders, the watermarks can be casily disrupted with minimal impact on image quality.
4.4. Ablation Study
4.4.1. WHY CONTEXTUAL ADAPTER?
Tab. 4 compares different adapter variants after the first stage training. We can find that using the contextual adapter structure is crucial for both watermark accuracy and image quality, improving bit accuracy by 0.02 and PSNR by significant number of 4.1 db compared with the context-
We did not include this method in Fig. 5 because the resulting image quality is far outside the scope of the comparisons shown the figure.

---

WMAdapter: Watermarking Latent Diffusion Models
Table 5: Comparison between different finetuning strategies. "Adapter-B" means no extra finetuning. Bit Acc is evaluated under combined attacks.
Stortwng 201 $5dh te
1.0
SSichwr 23)
S5ian 24)
Salan edo
0.9
Wee- ww--Ficheryy22l 2.11 We-ributeeta)
Bit Accuracy
Michao
Bit Acc
PSNR
SSIM
FIID
0.8
Adapter-B 0.92
32.8
0.94
2.7
0.7
Adapter-F 0.92
33.1
0.95
2.7
Adapter-I 0.90
34.8
0.96
2.5
0.6
Adapter-V 0.92
29.9
0.87
3.1
0.5
A
35
34
33
32
31
30
29
28
27
26
results in smoother but blurrier images. It also introduces noticeable lens flare artifacts, which are commonly observed in methods such as Stable Signature (Fernandez et al., 2023), FSW (Xiong et al., 2023), AquaLoRA (Feng et al., 2024), and WOUAF (Kim et al., 2023), as they all modify diffu- sion components to embed the watermark. This observation supports our core idea that preserving the integrity of the original diffusion pipeline crucial for high-quality genera- tion.
PSNR
Figure 5: Against various regeneration and adversarial at- tacks. SS: Stable Signature.
less structure. This result well supports our motivation that the watermark encoder should aware the cover image content to generate high quality embedding. Note that SOTA watermarking methods still use the context-less structure to encode watermark (Xiong al., 2023; Kim et al., 2023; Bui et al., 2023). Contextual adapter provides a simple yet promising approach for further improvement. Another key design is to use Ixl conv in the adapter, because we found that 3x3 conv suffers from unstable training.
Considering both numerical results and visual artifacts, Adapter-F and and Adapter-/ offer better sacuracy-quality trade- offs. Therefore, we adopt these two as our default choices. Note that all adapter variants incorporate an additional total variation loss during the second stage finetuning. While this loss helps produce visually smoother images and provides 0.1 PSNR improvement, it does not reduce artifacts (Fig. 6). Applying it during the first stage training can lead to overly smoothed images.
Table 4: Comparison between adapter structures.
Contextual
Context-less
Conv 3 x 3
4.4.3. RESULTS ON DIFFERENT VAES
Bit Ace
0.99
0.97
0.49
PSNR
32.8
28.7
12.0
We train several watermark adapters for VAEs used by SD1.5&2.1 (Rombach al., 2022), SDXL (Podell al., 2023) and DiT (Peebles & Xie, 2023) (kl-f8-mse) at res- olution 512 512. We compare the adapters before the finetuning stage. Tab. 6 shows the results. We observe that WMAdapter consistently performs well across various VAES, making it applicable in a wide range of contexts. The PSNR of SDXL adapter is lower compared to SD2.1 1 and DiT VAE. This may be caused by the resolution mismatch.
4.4.2. ROLE OF FINETUNING
Tab. 5 and Fig. 6 compare different finetuning strategies quantitatively and qualitatively. From Tab. 5, we can see that Adapter-B achieves good numerical results. However, upon closer inspection of the generated images, subtle grid-like artifacts become noticeable. If we freeze the VAE decoder and perform a quick fine-tuning for 2k steps using a large learning rate, resulting in Adapter-F. We find that PSNR and SSIM metrics further improve, though the artifacts persisted.
We further evaluate the zero-shot transferability of WMAdapter across different VAEs. Specifically, di- rectly apply the adapter trained on SD2.1 to the SD1.5 VAE and observe that it effectively handles SD1.5 image latents with minimal performance degradation. This empiri- cal result highlights the zero-shot generalization potential of WMAdapter to various eustomized Stable Diffusion VAEs.
Hybrid Finetuning (Adapter-/) further suppresses artifacts. Since the VAE remains unaltered during inference, pro- duces the sharpest and most visually appealing images, with PSNR improving significantly to 34.8 dB. This improve- ment comes at the minor cost of 0.02 decrease in bit accuracy under combined attacks.
4.5. Qualitative Results
Joint Finetuning (Adapter- V) significantly degrades all im- age quality metrics. As shown in Fig. 6, Joint Finetuning
We qualitatively compare WMAdapter with the baseline method, Stable Signature (Fernandez al., 2023) in Fig. 7.
8

---

WMAdapter: Watermarking Latent Diffusion Models
Orieinal
Adapter-B
AdapieesFF
Adapter-
Adapter-V
Original
Stabie Skg
TA3A
Ous-1 Oursp
TATA
Figure 6: Qualitative comparison between different fine- tuning strategies. Adapter-B and Adapter-F produces tiny grid-like artifacts. Finetuning with VAE (Adapter-/ and -V) alleviates this issuc. Using fintuned VAE at inference time (Adapter-V V) leads to lens flare artifact. Using original VAE (Adapter-/) achieves the most visually appealing results. Zoom in for best view.
TATA
Figure Comparison between WMAdapter and StableS- ignature (Fernandez al., 2023). Yellow arrows point to the generated artifacts. (B)(D)(F show the difference after watermarking. View in color and zoom in.
Table 6:
Evaluation on VAEs from different models.
Impact Statement
SD1.5
SD2.1
SDXL
DiT
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Bit Acc
0.99
0.99
0.99
0.99
PSNR
32.1
32.8
31.2
32.4
A Acknowledgment
We can observe that Stable Signature tends to produce lens flare artifacts, as indicated by the yellow arrows. We at- tribute this issue to the modification of VAE decoder. In contrast, A and Adapter-/ greatly suppress this noticeable artifact by preserving the integrity of all dif- fusion components. As shown in columns (C)D), our adapters produce sharper images with clearer text edges, which is also supported by the higher PSNR metric. In short, compared to StbaleSignature, WMAdapter produces higher quality images with fewer noticeable artifacts. Ap- pendices A.6, A.7, A.8 provide additional comparisons across more datasets.
This project is supported by the Ministry of Education, Sin- gapore, under the Academic Research Fund Tier 1 (FY2022) Award 22-5406-A0001.
References
An, B.. Ding, M.. Rabbani, T., Agrawal, A., Xu, Y., Deng, C., Zhu, S., Mohamed, A., Wen, Y., Goldstein, T., et al. Waves: Benchmarking the robustness of image water- marks. In Forty-first International Conference on Ma- chine Learning, 2024.
Ballé J., Minnen, D., Singh, S., Hwang, S. J., and Johnston, N. Variational image compression with a scale hyperprior. 6th International Conference on Learning Representa- tions, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
5. Conclusion and Limitation
In this paper, we introduce WMAdapter, a plug-and-pley watermarking plugin that enables latent diffusion models to embed arbitrary bit information during image genera- tion. Our adapter lightweight, easy to train, and of- fers superior accuracy-quality trade-off with significantly fewer noticeable artifacts compared to previous post-hoc and diffusion-nation watermarking methods. One limitation is that the Adapter-F variant occasionally produces giid-like artifacts that become visible upon zooming in. In summary, WMAdapter provides a simple yet powerful baseline for further exploration on diffusion watermarking.
Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z.,, Voleti, V.. Letts, A., et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023.
Bui, T., Agarwal, S., Yu, N., and Collomosse, Rosteals: Robust steganography using autoencoder latent space. In

---

WMAdapter: Watermarking Latent Diffusion Models
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 933-942, 2023.
Guo, Y., Yang, C., Rao, A., Wang, Y., Qiao, Y., Lin, D., and Dai, B. Animatediff: Animate your personalized texl-to-image diffusion models without specific tuning. arXiv preprint S.XXi:2337..47225 2023.
Cheng, Z., Sun, H., Takeuchi, M., and Katto, Learned image compression with discretized gaussian mixture likelihoods and attention modules. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Ci, H., Wu, M., Zhu, W., Ma. X., Dong, H., Zhong, F., and Wang, Y. Gfpose: Learning 3d human pose prior with gradient fields. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pp. 4800-48111 2023.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Ideogram.at ai. Ideogram. https / htt///ddeee htt///ddoogammmmmm a login, 2024. Accessed: 2024-05-22.
Ci, H., Yang, P., Song, Y., and Shou, M. Z. Ringid: Re- thinking tree-ring watermarking for enhanced multi-key identification. arXiv preprint arXiv:2404. 240.::2240444055,,, 14055, 2024.
Jia, Z., Fang, H., and Zhang, W. Mbrs: Enhancing robust- ness of dnn-based watermarking by mini-batch of real and simulated jpeg compression. In Proceedings of the 29th ACM international conference on multimedia, pp- 41-49, 2021.
Cox, I., Miller, M., Bloom, J., Fridrich, J., and Kalker, T. Digital watermarking and steganography. Morgan kaufmann, 2007.
Jiang. Z.,, Zhang, J., and Gong, N. Z. Evading watermark based detection of ai- ai-generated content. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pp. 1168-1181, 2023.
Czolbe, S., Krause, O., Cox, and Igel, C. A loss function for generative neural networks based watson's 's percep- tual model. Advances in Neural Information Processing Systems, 33:2051 11-2051-2000 10611, 2020.
Kim, C., Min, K., Patel, M., Cheng, S., and Yang. Y. Wouaf: Weight modulation for user attribution and fingerprint- ing text-to-image diffusion models. arXiv preprint arXiv:2306. 04744, 2023.
et al, L. Total variation denoising. //en. wikipedia. wwtp//ggg//w./ooooooott/eeeeeeoo variation variation_denois ing 2024. Accessed: 2024-05- 22.
https:
Kishore, V., Chen, X., Wang, Y., Li, B.. and Weinberger, K. Q. Fixed neural network steganography: Train the images, not the network. In International Conference on Learning Representations, 2021.
Feng, W., Zhou, W. He, J., Zhang, J., Wei, T., Li, G., Zhang, T., Zhang, W., and Yu, N. Aqualora: Toward white-box protection for customized stable diffusion models via watermark lora. arXiv preprint arXiv:2405.11135. 2024.
Lei, L., Gai, K., Yu, J., and Zhu, L. Diffusetrace: A transpar- ent and flexible watermarking scheme for latent diffusion model. arXiv preprint arXiv:2405.0266
Fernandez, P., Douze, M., Jégou, H., and Furon, T. Active image indexing. arXiv preprint arXiv:2210.10620 2022a.
2024.
Lin, C.-Y., Wu, M., Bloom, J. A., Cox, I. J., Miller, M. L., and Lui, Y. M. Rotation, scale, and translation resilient watermarking for images. IEEE Transactions on image processing, 10(5):767-782. 2001.
Fernandez, P., Sablayrolles, A., Furon, T., Jégou, H., and Douze, M. Watermarking images in self-supervised la- tent spaces. In ICASSP 2022-2022 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3054-3058. IEEE, 2022b.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. I.. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Fernandez, P., Couairon, G., Jégou, H., Douze, M., and Furon, T. The stable signature: Rooting watermarks in latent diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22466 22477, 2023.
Liu, Y., Song. Y., Ci, H., Zhang, Y., Wang, H., Shou, M. Z., and Bu, Image watermarks are removable using con- trollable regeneration from clean noise. arXiv preprint 1rX5v:2400.05470, 2024.
Freepik-Flation Flat icons. https: /www. flaticon. com/ free- icons/ snow, 2024. Accessed: 2024-05- 21.
10

---

WMAdapter: Watermarking Latent Diffusion Models
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. arXiv preprint
Song, Y., Lou, S., Liu, X., Ci, H., Yang, P., Liu, J., and Shou, M. Z., Anti-reference: Universal and immediate de- fense against reference-based generation. arXiv preprint arXiv: arXiv:2412.05980 2024a.
2017.
Lukas, N., Diaa, A., Fenaux, L.., and Kerschbaum, F. Lever- aging optimization for adaptive attacks on image water- marks. arXiv preprint arXiv:2309. 16952, 2023.
Song, Y., Yang. P., Ci, H., and Shou, M. Z. Idprotector: An adversarial noise encoder to protect against id- id-preserving image generation. arXiv preprint arXiv:2412.11638. 2024b.
Madry, A. Towards deep learning models resistant adver- sarial attacks. arXiv preprint arXiv: 700.06087660088,,,, 2017.
Meng, Z., Peng, B., and Dong. J. Latent watermark: Inject and detect watermarks in latent diffusion space. arXiv preprint arXiv:2404.002.. 2024.
Tancik, M., Mildenhall, B., and Ng, R. Stegastamp: Invisi- ble hyperlinks in physical photographs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2117-2126, 2020.
Min, R., Li, S., Chen, H., and Cheng, M. A watermark- conditioned diffusion model for ip protection. arXiv preprint arXiv:2403. 10893, 2024.
Wang. T., Zhang, Y., Qi. S., Zhao, R., Xia, Z., and Weng, J. Security and privacy on generative data in aige: A survey. ACM Computing Surveys, 57(4):1-34, 2024.
Nguyen, Q. Vu, T., Pham, C., Tran, A., and Nguyen, K. Sta- ble messenger: Steganography for message-concealed im- age generation. arXiv preprint arXiv:23/2.01228, 2023.
Wen, Y., Kirchenbauer, J., Geiping, J., and Goldstein, T. Tree-ring watermarks: Fingerprints for diffusion images that are invisible and robust. arXiv preprint arXiv:2305.20030, 2023.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195-4205, 2023.
Westerlund, M.
The emergence of deepfake technology: A review. Technology innovation management review, (11), 2019.
Podell, D., English, Z.,, Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for high-resolution image synthesis. arXiv preprint wxXiv:230..01952 2023.
Xia, X.-G., Boncelet, C. G., and Arce, G. Wavelet transform based watermark for digital images. Optics Express, 3(12):977551, 1998.
Xiong, C., Qin, C., Feng, G., and Zhang, X. Flexible and secure watermarking for latent diffusion model. In Pro- ceedings of the s ACM International Conference on Multimedia, pp. 1668-1676, 2023.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022.
Yang, P., Ci, H., Song, Y., and Shou, M. Z. Can simple averaging defeat modern watermarks? Advances in Neu- rul Information Processing Systems, 37:56644-56673, 2024a.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmenta- tion. In Medical image computing and computer-assitted intervention-MICCAN 2015: 18th international confer- ence, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234-241. Springer, 2015.
Yang, Z., Zeng, K., Chen, K., Fang, H., Zhang, W., and Yu, N. Gaussian shading: Provable performance-
image watermarking for diffusion models. arXiv preprint urXiv:2404.044 2024b.
Saberi, M., Sadasivan, V. S., Rezaei, K., Kumar, A., Chegini, A. Wang, W., and Feizi, Robustness of ai-image detectors: Fundamental limits and practical attacks. arXiv preprint arXiv:2310.000 00076, 2023.
Zhang, D. J., Wu, J. Z., Liu, J.-W., Zhao, R., Ran, L., Gu, Y., Gao, D., and Shou, M. Z. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. arXiv preprint arXiv:2309. 15818, 2023a.
Sander, T., Fernandez, P., Durmus, A., Furon, T., and Douze, M. Watermark anything with localized messages. arXiv preprint arXiv:2411.0722. 2024.
Zhang, G., Wang, I.., Su, Y., and Liu, A.-A. A training-free glug-and-play watermark framework for stable diffusion. arXiv preprint arXiv:2404. 05607, 2024.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score- based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 13456, 2020.
Zhang, K. A., Xu, L., Cuesta-Infaant, A., and Veeramacha- neni, K. Robust invisible video watermarking with atten- tion. arXiv preprint arXiv: 1909 /90..//995....... 2019.

---

WMAdapter: Watermarking Latent Diffusion Models
Zhang, L.., Rao, A., and Agrawala, M. Adding conditional control text-to-imaae diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023b.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-5955 2018.
Zhao, X., Zhang, K., Su, Z.., Vasan, S., Grishchenko, I,, Kruegel, C., Vigna, G., Wang, Y.-X., and Li, L. Invisible image watermarks are provably removable using genera- tive ai. arXiv preprint wrXiv:2306.01955 2023a.
Zhao, Y., Pang, T., Du, C., Yang, X., Cheung, N.-M., and Lin, M. A recipe for watermarking diffusion models. arXiv preprint arXiv:2303. 10137, 2023b.
Zhu, J., Kaplan, R., Johnson, J., and Fei-Fei, L. Hidden: Hiding data with deep networks. In Proceedings of the European conference computer vision (ECCV), pp- 657-672, 2018.
12

---

WMAdapter: Watermarking Latent Diffusion Models
A. Appendix
A.1. Experiment Statistical Significance
For the first training stage, we ran independent training and found the standard deviation of average validation bit accuracy across 3 runs to be 0.0006, and the standard deviation of validation PSNR be 0.03 dB.
For the second finetuning stage, we also 3 independent trials. The standard deviation of average validation bit accuracy across 3 runs was also 0.0006, and the std of validation PSNR was 0.04 db. The small standard deviation at both stages demonstrates the stability of our method. Since the standard deviation is too small to be clearly viewed in Fig. 8, we report the numbers in text.
A.2. Broader Impacts
The proposed diffusion watermarking technique offers significant positive societal impacts, such as enhancing copyright protection for digital creators and helping to prevent the spread of fake news by enabling the authentication of images. However, it also poses potential negative impacts, including privacy concerns, the risk of misuse for malicious purposes, technical challenges that may disadvantage smaller creators, and possible degradation of image quality. Balancing these benefits and drawbacks is crucial to ensure the responsible and effective use of this technology.
In terms of applications, our proposed WMAdapter also be directly applied to video generation models such AnimateDiff (Guo et al., 2023) and as image Diffusion models. We leave further exploration on video to the future work.
(Blattmann et al., 2023), which share the same VAE architecture
A.3. Evaluation on Various Distortion Intensities
Fig. 8 evaluates our method under larger ranges of distortion intensities and more attacks. We can that adapters remain comparable robustness to Stable Signature (Fernandez et al., 2023) over range of attack intensities. Note that all three methods exhibit limited robustness to significant Gaussian noise and Rotation. This limitation arises because the pretrained HiDDeN decoder (Fernandez et al., 2023) was not specifically trained to handle such attacks. To further enhance robustness under such attacks, WMAdapter would need to be built upon a watermark decoder that is pretrained with rotation and noise augmentation.
A.4. Visualization of Distortions
Fig. 9 shows different image distortions evaluated in the paper.
A.5. Evaluation Against Other Adaptive Attacks
We also evaluate WMAdapter- against another adversarial attack Lukas et al. (2023), which propose to train a stronger surrogate detector. We reproduce the adversarial noising method described in the paper. Specifically, we implemented their approach using the reported hyperparameters. We found that the suggested c-ball of 2/255 produced negligible attack effects. We increased the e-ball to 8/255, reducing PSNR from 34.8 30.3 (similar drop other attacks our Fig. 5), while the bit accuracy dropped moderately from 0.98 to 0.93. This suggests that method demonstrates resilience to such attacks.
A.6. More Qualitative Results on COCO Dataset
Fig. 10 shows the watermarked images and their difference with the original images. We find that both WMAdapter-F and WMAdapter- can adaptively embed watermark information into regions with significant color variations and richer textures in the images, significantly enhancing their invisibility.
A.7. Generalization to Ideogram Dataset
Fig. 11 shows our results images generated by Ideogram (Ideogram.at ai, 2024). These images exhibit completely different styles. However, our WMAdapter, trained on COCO, transfers seamlessly to them.
13

---

WMAdapter: Watermarking Latent Diffusion Models
A.8. Comparison With Other Watermarking Methods
Fig. 12 compares various watermarking methods. We observe that our method introduces minimal noticeable artifacts the images. Thanks to the dedicated design of the contextual adapter, the modifications adapt more effectively to the cover image content.
While the JND enhancement (Fermandez et al., 2022a) used by HiDDeN* can also adapt the watermark post-hoc. However, such post-hoc methods compromises robustness and tends to alter the background. In contrast, our contextual adapter is trained snd-to-end, offering better robustness-quality tradeoff (see Tab. and Fig. 1).
14

---

WMAdapter: Watermarking Latent Diffusion Models
Crop
JPEG
Brightness
Constrast
1.0
0.6
StableSignature WMadapter_F
0.2
WMadapter__
0.9
0.7
0.5
0.3
0.1
90
70
30
1.0
1
5
Crop Ratio
JPEG Quality
Brightness Faictor
Constrast Factor
Saturation
Sharpness
Gaussian Noise
Resize
0.8
Accuracy
0.6
0.4
0.2
2
5
0.02
0.04
0,06
0.08
0.10
0.9
0.7
0.5
0.3
0.1
Saturation Factor
Sharpness Factor
Noise Std
Scale Factor
Rotation
Erase
1.0
0.8
0.4
0.0
Degree
Scale
Figure 8: Robustness comparison over various distortion intensities.
15

---

WMAdapter: Watermarking Latent Diffusion Models
Rotate 20
Crop 0.3
PEG 50
Brightness 3.0
Contrast 3.0
Erase 0.25
Noise 0.1
Resize 0.5
Saturation 3.0
Sharpness 3.0
Figure 9: Visualization of different augmentations.
16

---

WMAdapter: Watermarking Latent Diffusion Models
Original
Adapter-
Difference
Adapter-
Difference
TRUCKS
EAST
EAST
EAST
Queens Bronx
Queens Bronx
Queens Bronx
11
Figure 10: Qualitative results on COCO dataset at resolution 512.
17

---

WMAdapter: Watermarking Latent Diffusion Models
Original
Adapter-
Difference
Adapler-
Difference
AE
BAY
WAST
METTO
PEACEUL GRAZING
PEACEUL GRAZING
PEACEUL GRAZING OREAMS! BLAZING QUIE
DREAMS BLAZING QUIE
DREAMS, BLAZING QUIE CHASE
CHASE
Figure 11: Qualitative results Ideogram (Ideogram.at 2024) at resolution 512.
18

---

WMAdapter: Watermarking Latent Diffusion Models
Original
AquaLoRA
WOUAF
RoSteALS
SSL
StableSig
HiDDeN*
WMAdalper-1 I
"Downtown Beijing at sunrise. detailed ink wash" "
"a cat sitting on a box with a drawing of an elephant on it"
"robots meditating in vipassana retreat"
"a tree growing out of the middle of an intersection"
Figure 12: Watermarking images generated with given prompts. For HiDDeN (Zhu et al., 2018), we use a post-hoe just noticeable difference (JND) mask to enhance invisibility (Fernandez al., 2022a). Zoom for best view.
19


Reviews:
[{"review_id": "xI2zUfX6Rz", "reviewer": "Reviewer_ZSgp", "summary": "This paper proposes the **WMAdapter**, which generates content-aware watermark embeddings using the contextual adapter and embeds watermarks with a hybrid fine-tuning strategy. Specifically, the contextual adapter comprises a series of fuser modules, each of which is attached before a corresponding VAE decoder block. In addition to training the adapter, **WMAdapter** adopts a hybrid fine-tuning strategy that jointly optimizes the adapter and VAE decoder while using the original VAE decoder for inference. Experimental results show that the **WMAdapter** retains effective watermark-embedding capability while maintaining generation quality.\n\n### Update after rebuttal\nI've no further questions regarding this paper after rebuttal and keep my original score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: Is it possible to select a better $\\lambda$ or or find better checkpoints to enhance image quality?**\n\nThank you for your insightful question. In practice, selecting a better $\\lambda$ or checkpoint to enhance image quality proves very challenging. During our joint training experiments (Adapter-V), we extensively explored different $\\lambda$ values and checkpoints. However, none could achieve a better trade-off between image quality and robustness compared to Hybrid Finetuning. Specifically, increasing the weight of the image quality loss significantly deteriorated bit accuracy, sometimes even leading to training collapse. Moreover, merely raising the quality loss weight does not effectively suppress the lens flare artifacts shown in Figure 6; it only slightly improves the quantitative metrics.\n\nWe believe this issue arises not only from the choice of loss weights or checkpoints but also from modifications to the VAE decoder itself. Joint finetuning significantly differs from the large-scale pretraining originally conducted for the VAE. It adjusts full VAE decoder parameters based on much smaller datasets with distinct distributions, batch sizes, and training objectives (bit acc + quality). Consequently, this easily disrupts the inherent knowledge encoded in the pretrained VAE decoder, leading to reduced image quality or the emergence of artifacts.\n\nIn other words, adjusting the loss weight or selecting different checkpoints during joint finetuning cannot effectively resolve the substantial deviations from the pretrained parameter landscape. The most reliable solution remains using the original pretrained VAE decoder directly, as employed in our proposed Hybrid Finetuning method. \n\nRegarding the loss, lower values of $\\mathcal{L}(\\mathcal{A}(w), \\mathcal{V}^{new})$ achieved during joint fine-tuning do not necessarily indicate genuine improvements, as the VAE decoder $\\mathcal{V}^{new}$ tends to overfit on the limited training data.\n\n&nbsp;\n\n**Q: Relation To Broader Scientific Literature**\n\nThank you for highlighting this point. We will clarify and discuss the relation of our work to the broader scientific literature in the revised manuscript.", "claims_and_evidence": "The claims are well supported.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense. Below, I list one concern about hybrid fine-tuning.\n\n1. I'm confused about the design of hybrid fine-tuning, and my concern is explained as follows. The better generation performance is achieved at the cost of detection capability. When jointly fine-tuning the adapter $\\mathcal{A}$ and the VAE decoder $\\mathcal{V}^{\\textrm{new}}$, after several updating steps we have the loss $\\mathcal{L}(\\mathcal{A}(w), \\mathcal{V}^{\\textrm{new}}(f))$ which should be lower than $\\mathcal{L}(\\mathcal{A}(w), \\mathcal{V}^{\\textrm{ori}}(f))$, where $\\mathcal{V}^{\\textrm{ori}}$ indicates the original VAE decoder. This aligns with the experimental results where Adapter-I compromises parts of the watermarking capability to enhance the image quality. If so, my concern is whether it is possible to select a set of better $\\lambda$ to achieve this without hybrid fine-tuning or find better checkpoints during the training process.", "theoretical_claims": "No theoretical proofs to check.", "experimental_designs_or_analyses": "The experiments in this paper seem sufficient.", "relation_to_broader_scientific_literature": "WMAdapter can improve traceability within LDMs and reduce the misuse of AIGC contents [1].\n\n[1] Security and Privacy on Generative Data in AIGC: A Survey", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The overall paper is well-written, and the method design is well supported by the motivation and experiments.", "other_comments_or_suggestions": "Please see above.", "questions_for_authors": "Please see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0BYzCWuSiH", "reviewer": "Reviewer_BZ2P", "summary": "This paper introduces WMAdapter, a watermarking plugin for AI-generated images that seamlessly embeds user-specified watermark information during the diffusion generation process. Unlike previous methods that modify diffusion modules to embed watermarks, WMAdapter preserves the integrity of diffusion components, resulting in sharp images with no noticeable artifacts. The contributions of this paper include a contextual adapter for generating adaptive watermark embeddings based on the image content and a hybrid finetuning strategy to suppress artifacts while preserving the integrity of the diffusion process. Experimental results show that WMAdapter provides superior image quality, flexibility, and competitive watermark robustness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: Essential References Not Discussed: Gaussian Shading and FSW**\n\nThank you for the suggestion. In fact, we have already referenced and discussed both works in the Introduction section. Specifically, they are cited as [Yang et al., 2024] and [Xiong et al., 2023], corresponding to Gaussian Shading and FSW, respectively.\n\n&nbsp;\n\n**Q:  “It is recommended to include a comparison of watermark effects and potential artifacts under both hybrid training and standalone training modes.”**\n\nThank you for the suggestion. We kindly refer the reviewer to Figure 6, which provides a visual comparison of watermark effects and potential artifacts under different training modes. We will include more examples in the revised manuscript to improve clarity.\n\n&nbsp;\n\n**Q:  “The motivation lacks clarity — how do the observations from existing methods logically lead to the need for fine-tuning the VAE decoder and the inclusion of associated modules?”**\n\nWe would like to further clarify our motivation. Existing methods typically watermark diffusion models by modifying the backbone or VAE decoder parameters, which often results in visible artifacts and degraded visual quality. To address this, we propose a **non-intrusive framework** that preserves all original diffusion parameters. Specifically, we design a **Contextual Adapter** that is attached to the VAE decoder, avoiding direct modification of its internal structure. In addition, we introduce a **Hybrid Finetuning Strategy** to train the Contextual Adapter in a similarly non-intrusive fashion.  \n\nIf there are any remaining questions regarding our motivation, we would be happy to provide further clarification.", "claims_and_evidence": "The authors claim that WMAdapter is a novel diffusion watermarking solution with an innovative design philosophy, which is appropriate. They also claim that WMAdapter outperforms previous post-hoc and diffusion-native watermarking methods, and the experimental results support this claim.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the diffusion model watermarking.", "theoretical_claims": "This paper does not provide any theoretical claims.", "experimental_designs_or_analyses": "The experimental and analyses of this paper is sound and valid. It compared some important model watermarking methods, such as Stable Signature.", "relation_to_broader_scientific_literature": "None", "essential_references_not_discussed": "[1] Yang Z, Zeng K, Chen K, et al. Gaussian shading: Provable performance-lossless image watermarking for diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 12162-12171.\n[2] Xiong C, Qin C, Feng G, et al. Flexible and secure watermarking for latent diffusion model. Proceedings of the 31st ACM International Conference on Multimedia. 2023: 1668-1676.", "other_strengths_and_weaknesses": "**Strength**\n\n(1) The paper proposes a watermarking method for image generative models that demonstrates robustness against various types of attacks, outperforming existing approaches.\n\n(2) It explores a hybrid training strategy involving the VAE decoder, which effectively mitigates the degradation of image quality caused by watermark embedding.\n\n(3) Comprehensive ablation studies are conducted to evaluate the contribution of the adapter and hybrid training strategy, showing (i) improved watermark decoding accuracy and (ii) no compromise in the performance of the generative model.\n\n**Weakness**\n\n(1) It is recommended to include a comparison of watermark effects and potential artifacts under both hybrid training and standalone training modes.\n\n(2) The motivation lacks clarity — how do the observations from existing methods logically lead to the need for fine-tuning the VAE decoder and the inclusion of associated modules?", "other_comments_or_suggestions": "Please refer to the weakness.", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D6JHiucHLz", "reviewer": "Reviewer_5xGq", "summary": "This paper introduces WMAdapter, a plug-and-play watermarking solution for latent diffusion models that embeds watermarks during the image generation process without modifying the original diffusion components. The authors propose two key innovations: (1) a contextual adapter that conditions on the content of the cover image to generate adaptive watermark embeddings, and (2) a hybrid finetuning strategy that preserves the integrity of diffusion components while suppressing artifacts.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: Discrepancy between qualitative and quantitative evaluation in Figure 1.**\n\nDifferent types of watermarks introduce different types of artifacts. Although FID is one of the most widely used quantitative metrics for image quality, it often struggles to accurately reflect the visual impact of diverse artifacts. Bridging the gap between quantitative metrics and human perception remains an open research problem and an active area of investigation.\n\n&nbsp;\n\n**Q: Why weren't all distortions in Figure 8 quantitatively presented in Table 2**\n\nDue to space limitations, we had to make a trade-off in presentation. In Table 2, our goal was to provide a balanced overview of both image quality and robustness metrics. Including all attack types and quality metrics in a single table would have been impractical. This design choice is similar to that of Table 1 in *Stable Signature*. To complement Table 2, we included Figure 8, which provides a more comprehensive evaluation by illustrating robustness under a wider range of attack types and intensities.\n\n&nbsp;\n\n**Q: How does WMAdapter perform against rotation attacks?**\n\nPlease refer to this anonymous [link](https://ibb.co/FjJWm3c) for our evaluation of rotation robustness. It is important to note that WMAdapter is built upon the public checkpoint provided by *Stable Signature*, which was not pretrained with rotation augmentation. Despite this, WMAdapter demonstrates robustness to moderate rotation (up to 15 degrees) and performs comparably to *Stable Signature*. To further enhance robustness under stronger rotation, WMAdapter would need to be built upon a watermark decoder that is pretrained with rotation-augmented data.\n\n&nbsp;\n\n**Q: “The paper does not evaluate against state-of-the-art adaptive removal methods using optimization or generative models”**\n\nWe would like to clarify that our paper **does evaluate against SOTA opensource** adaptive removal methods, including both optimization-based and generative approaches. Specifically, for **optimization-based methods**, we include evaluations against the adversarial attack proposed in [An’24] under both black-box and white-box settings. For **generative models**, we assess performance against VAE-based methods ([Cheng’20], [Balle’18]) as well as a diffusion-based method ([Zhao’23]). Please refer to Sec. 4.3 and Fig. 5 for detailed results.\n\n&nbsp;\n\n**Q: Lack evaluation against \"Leveraging Optimization for Adaptive Attacks on Image Watermarks”**\n\nAs the code for the referenced paper is **not publicly available**, we have made our best effort to reproduce the adversarial noising method described in the paper. Specifically, we implemented their approach using the reported hyperparameters. We found that the suggested $\\epsilon$-ball of 2/255 produced negligible attack effects. We increased the $\\epsilon$-ball to 8/255, reducing PSNR from 34.8 to 30.3 (similar drop to other attacks in our Fig.5), while the bit accuracy dropped moderately from 0.98 to 0.93. This suggests that our method demonstrates resilience to such attacks. We'll include the result in updated version.\n\n&nbsp;\n\n**Q: Discuss the design choice and the explicit trade-offs made between visual quality and robustness”**\n\nOur goal is to develop a high-quality, artifact-free watermarking technique. To this end, we propose a **non-intrusive watermarking framework** that avoids modifying the pretrained diffusion pipeline. So we design **Contextual Adapter** as a plugin avoiding direct modification of diffusion models. Also we develop **Hybrid Finetuning** to conduct non-intrusive training.\n\nThe relative importance of robustness versus visual quality is indeed a nuanced and context-dependent question. From the perspective of watermarking as a standalone technique, **robustness** is often considered the primary metric. However, when we consider broader deployment scenarios—such as integrating watermarking into GenAI products, which is a key motivation of our work—the priorities shift. In these contexts, **visual quality becomes paramount, as it directly impacts user experience and product adoption**. There is a strong practical demand for artifact-free watermarking that preserves the high visual fidelity expected from GenAI products. Therefore, our design values visual quality without compromising essential robustness, striking a balance suitable for real-world applications.\n\n&nbsp;\n\n**Q: “How does the approach scale to larger or smaller watermark sizes beyond the 48-bit watermark?”**\n\nOur approach is built on a pretrained watermark decoder. To scale to different watermark sizes, one can simply replace the decoder with a pretrained version of the desired size and retrain only the adapter—the rest of the system remains unchanged. Alternatively, users may train a watermark decoder of arbitrary size themselves.\n\n&nbsp;\n\n**Q: Robustness to significant editing**\n\nThanks for your suggestion. We randomly edit 80% of the image by inpainting, the bit accuracy remains 0.91.", "claims_and_evidence": "The claim of superior image quality is backed by quantitative metrics (PSNR, FID) and qualitative visual comparisons showing fewer artifacts than other methods.\n\nThe comparison with AquaLoRA seems to focus on visual quality differences without acknowledging that AquaLoRA was designed for more stringent white-box attack scenarios.", "methods_and_evaluation_criteria": "The watermarking architecture (using contextual adapters attached to VAE decoder) is well-motivated and explained clearly.\n\nThe hybrid finetuning strategy addresses a real issue (artifacts) observed in preliminary experiments.\n\nHowever, the evaluation has some notable limitations:\n\n- The paper acknowledges the quality-robustness trade-off but appears to prioritize visual quality over robustness in their design choices. This is a legitimate approach, but it would benefit from more explicit discussion, as robustness is generally considered more critical for watermarking applications.\n- The paper lacks some important adaptive evaluations. In particular, one important recent works are not addressed: \"Leveraging Optimization for Adaptive Attacks on Image Watermarks\", These studies present important challenges to watermarking methods that should be evaluated.", "theoretical_claims": "The paper is primarily empirical rather than theoretical.", "experimental_designs_or_analyses": "The experimental designs are generally sound.", "relation_to_broader_scientific_literature": "The paper effectively positions WMAdapter within the broader context of: 1) Traditional post-hoc watermarking methods (frequency domain, optimization-based, encoder-decoder) and 2) Diffusion-native watermarking approaches (initial noise-based vs. latent space-based).", "essential_references_not_discussed": "Leveraging Optimization for Adaptive Attacks on Image Watermarks", "other_strengths_and_weaknesses": "Strengths:\n\n* The paper addresses a clear practical need for high-quality watermarking in diffusion models\n* The solution is elegant, lightweight (1.3MB) and computationally efficient (30ms per image)\n* The hybrid finetuning strategy is novel and effective at preserving image quality\n* The approach is practical and implementable, with clear explanations of design choices\n\nWeaknesses:\n\n* The work prioritizes visual quality over robustness, which may limit its practical application in scenarios where security is paramount\n* The evaluation lacks assessment against rotation attacks, which are common in real-world scenarios\n* The paper does not evaluate against state-of-the-art adaptive removal methods using optimization or generative models\n* There is inconsistency between the methods shown in qualitative and quantitative evaluations (Figure 1)\n* Some attacks demonstrated in Figure 8 are not thoroughly quantified in the main results (Table 2)", "other_comments_or_suggestions": "* Why is there a discrepancy between the methods shown in the qualitative demonstration (left side of Figure 1) and the quantitative evaluation (right side of Figure 1)?\n* Figure 8 demonstrates robustness against many additional distortions, but Table 2 only presents a limited subset. Why weren't all these distortions quantitatively evaluated in the main results?\n* The paper lacks evaluation of rotation robustness, which is a critical transformation for watermark assessment. How does WMAdapter perform against rotation attacks, and why was this evaluation omitted?\n* Recent works such as \"Leveraging Optimization for Adaptive Attacks on Image Watermarks\". How does WMAdapter resist these targeted attacks?\n* The paper appears to prioritize visual quality over robustness. Given that robustness is generally considered more critical for watermarking applications, can you discuss this design choice and the explicit trade-offs made?\n* How does the approach scale to larger or smaller watermark sizes beyond the 48-bit watermark used in experiments? Would it require retraining the entire system or just the adapter?\n* Would the watermark still be recoverable after significant editing?\n* Some minor typos, like in L400, \"StbaleSignature\" instead of \"StableSignature\"", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel watermarking technique for diffusion models, more concretely an adapter (the WMAdapter) allowing to keep the components of the diffusion model intact and a fine-tuning strategy for training the adapter equipped diffusion model. Experimental results show that the WMAdapter retains effective watermark-embedding capability while maintaining generation quality.

The rebuttal clarified the open questions of the two reviewers and all voted for acceptance.
Decision: Accept (poster)