Watermark Anything With Localized Messages

## Abstract

Abstract Image watermarking methods are not tailored to handle small watermarked areas.
This restricts applications in real-world scenarios where parts of the image may come from different sources or have been edited.
We introduce a deep-learning model for localized image watermarking, dubbed the Watermark Anything Model (WAM).
The WAM embedder imperceptibly modifies the input image, while the extractor segments the received image into watermarked and non-watermarked areas and recovers one or several hidden messages from the areas found to be watermarked.
The models are jointly trained at low resolution and without perceptual constraints, then post-trained for imperceptibility and multiple watermarks.
Experiments show that WAM is competitive with state-of-the art methods in terms of imperceptibility and robustness, especially against inpainting and splicing, even on high-resolution images.
Moreover, it offers new capabilities: WAM can locate watermarked areas in spliced images and extract distinct 32-bit messages with less than 1 bit error from multiple small regions – no larger than 10% of the image surface – even for small 256 × 256 256 256 256\times 256 256 × 256 images.

## 1 Introduction

Invisible image watermarking embeds information into image pixels in a way that is imperceptible to the human eye and yet robust.
It was initially developed for intellectual property and copy protection, such as by Hollywood studios for DVDs.
However, the applications of watermarking are evolving, particularly in light of the recent development of generative AI models (Kušen and Strembeck, ).
Regulatory acts such as the White House executive order (USA, ), the Californian bill, the EU AI Act (Parliament and Council, ), and Chinese AI governance rules (of the People’s Republic of China, ) require AI-generated content to be easily identifiable.
They all cite watermarking as either compulsory or a recommended measure to detect and label AI-generated images.

Image splicing is one of the most common manipulations, whether applied for benign or malicious purposes (Christlein et al., ; Tralic et al., ).
Splicing involves adding text or memes on a large portion of the image or extracting parts of images and overlaying them on others (Douze et al., ).
It can bypass the state-of-the-art watermarking techniques, which take one global decision per image under scrutiny.
Indeed, in traditional watermarking, the watermark signal fades away and is no longer detected as the surface of the watermarked area decreases.
Besides, these techniques poorly answer the paradoxical question of deciding whether an image should be considered watermarked if only a small part carries the watermark.
A positive decision triggered by a small area might be unfair to artists who use AI models for inpainting or outpainting.
On the other hand, not being robust enough to splicing opens the door to easy removal.

Figure: Figure 1: Overview. (a) The embedder creates an imperceptible image modification. (b) Traditional transformations (cropping, JPEG compression, etc.) and/or advanced manipulations (mixing watermarked and non-watermarked images, inpainting, etc.) may be applied to the image. (c) The extraction creates a segmentation map of watermarked parts and retrieves one or several messages.
Refer to caption: x1.png

To address these issues, this paper redefines watermarking as a segmentation task, giving birth to the Watermark Anything Models (WAM).
Our motivation is to disentangle the strength of the watermark signal from its pixel surface, in contrast to traditional watermarking.
More precisely, the WAM extractor detects if the watermark is present and extracts a binary string *for every pixel* rather than predicting a message for the whole image.
These outputs are post-processed according to the final task.
For global detection, the image is deemed watermarked if the proportion of watermarked pixels exceeds a user-defined threshold.
For global decoding, a majority vote recovers the hidden message.
A new application, out of the reach of traditional robust watermarking, is the localization of watermarked areas and the extraction of multiple hidden messages.
For that purpose, we choose to apply the DBSCAN clustering algorithm over the pixel-level binary strings because it does not require any prior on the number of watermarks (or centroids).
This is detailed in Sec. 3.

These new functionalities require a training with new objectives, that is split into two phases.
The first phase pre-trains the embedder and extractor models for low-resolution images. It essentially targets the robustness criterion.
The embedder encodes a $n_{\text{bits}}$-bit message into a watermark signal that is added to the original image.
The augmenter randomly masks the watermark in parts of the image
and augments the result with common processing techniques (e.g., cropping, resizing, compression).
The extractor then outputs a ($1+n_{\text{bits}}$)-dimensional vector per pixel to predict the parts of the image that are watermarked and decode the corresponding messages.
Detection and decoding losses are used as training objectives.
The second training phase targets the following new objectives:
(1) minimize the watermark’s visibility in alignment with the human visual system,
(2) allow for multiple messages within the same image.
This two-stage training is less prone to instability, compared to previous use of adversarial networks and divergent objectives (Zhu et al., ).
It also trains the extractor on both watermarked and non-watermarked images, for the first time in the literature.
This increases the performance and the robustness of the detection.

We first compare WAM with state-of-the-art methods for regular tasks of watermark detection and decoding on low and high-resolution images.
Our results show that WAM achieves competitive performance in terms of imperceptibility and robustness.
To further highlight the advantages of WAM, we then evaluate its performance on tasks that are not considered in the literature.
Namely, we evaluate the localization accuracy between the predicted watermarked areas and the original mask and assess the ability to detect and decode multiple watermarks in a single image.
For instance, when hiding five 32-bit messages, each in a 10% area of the image, detection of watermarked areas achieves more than 85% mIoU, even after images are horizontally flipped and the contrast adjusted, and bit accuracy (for a total of 160 bits) achieves more than 95% under the same augmentation (Sec. 5.5).

In summary, our contributions are:

- •
the definition of watermarking as a segmentation task;
- •
a two-stage training able to strike a good trade-off between invisibility and robustness even for multiple watermarks and high-resolution images;
- •
WAM, an embedder/extractor model competitive with state-of-the-art methods;
- •
the highlight of new capabilities, localization of watermarks and extraction of multiple messages as depicted in Fig. 1, together with specially designed evaluations.

## 2 Related Work

#### Semantic segmentation

aims to predict a category label for every pixel in an image.
FCN (Long et al., ) employs a convolutional network predicting pixel-wise classification logits.
More recent works (Zheng et al., ; Strudel et al., ) aggregate hierarchical context and are based on a ViT encoder (Dosovitskiy, ), followed by a decoder that generates the pixel-level predictions – similar to our extractor’s architecture.
On the other hand, instance segmentation identifies individual objects within an image (He et al., ; Carion et al., ).
Recent literature, such as MaskFormer (Cheng et al., ) and Mask2former (Cheng et al., ), unifies semantic and instance segmentation.
Segment Anything (Kirillov et al., ) and follow-ups (Zhang et al., ; Zhao et al., ; Ke et al., ; Ma et al., ; Ravi et al., ) go one step further.
They now allow users to segment any object in images or videos using prompts, which are points, bounding boxes, or natural language.

#### Robust watermarking

was first designed for copyright protection (Cox et al., ).
Traditional methods operate either in the spatial domain (Van Schyndel et al., ; Nikolaidis and Pitas, ) or in the frequency domain modifying components through an invertible transform (e.g., Fourier, Wavelet) (Cox et al., ; Urvoy et al., ).
These methods were progressively replaced by deep-learning ones which remove the expert need to crafting the transforms such that the embedding is imperceptible and robust.
The first approach (Vukotić et al., , ; Fernandez et al., ; Kishore et al., ; Chen et al., ) embeds the watermark into the representations of an off-the-shelf pre-trained model.
Our method is inspired by the second approach, which jointly trains deep learning-based architectures to embed and extract watermarks while being robust to augmentations seen during training, such as HiDDeN (Zhu et al., ) and followers (Zhang et al., ; Luo et al., ; Tancik et al., ; Ma et al., ; Bui et al., , ; Pan et al., ).

The two tasks of watermarking, detection and decoding, are rarely performed together.
Detection of the watermark, a.k.a., zero-bit watermarking, distinguishes watermarked content from original images.
Decoding of a hidden message, a.k.a., multi-bit watermarking, implicitly assumes that the content under scrutiny is watermarked.
One possibility for combining the two tasks is to reserve $n_{\text{det}}$ bits of the message as a detection segment – that should match a fixed pattern – with the remaining bits carrying the actual payload.
Assuming that the message decoded from a non-watermarked image is random, the False Positive Rate equals $2^{-n_{\text{det}}}$.
However, a recent study (Fernandez et al., , App. B.5) shows that the decoded bits are neither equiprobable nor independent.
In other words, it is difficult to decode a hidden message while being confident in detecting a watermark.

Very few papers in the literature deal with watermarking objects in picture (Bas et al., ; Barni et al., ).
This trend was abandoned together with the object-oriented MPEG-4 video codec.

#### Active tamper localization

relies on *semi-fragile* watermarking.
Areas where the watermark is not recovered are deemed tampered.
This idea appears early in the literature (Kundur and Hatzinakos, ; Lin et al., ), although deep-learning methods offer significant improvements (Asnani et al., ).
There is a trade-off between the granularity of the localization of the tampered areas and the semi-fragility of the watermark.
The biggest difficulty is to design a watermark robust to benign transformations (e.g., image compression) but fragile to malicious editing.

Our method combines detection and decoding together with a precise segmentation of the watermarked areas.
This natively answers applications ranging from copyright protection, detection of AI-generated objects in images and tampering localization.
EditGuard (Zhang et al., ) sequentially embeds a fragile watermark for localization and a robust watermark for copyright protection.
Although the approach shares similarities with ours, it is not robust to geometric augmentations such as cropping or perspective changes.
In contrast, our approach is conceptually simpler since a single embedding is used for detection and decoding.
It offers better robustness, and enables the extraction of multiple watermarks.
The approach most similar to ours is AudioSeal (San Roman et al., ), which introduces localized watermarking in the audio domain, but does not handle multiple messages.

An extended related work with more details on the methods mentioned above is presented in App. 8.

## 3 Detection, Localization, and Message Extraction

Before introducing our method and its training, this section presents several applications, i.e., how to use WAM’s extractor outputs for watermark localization, zero-bit detection, and decoding of multiple messages within the same image.

The key feature is the extractor $\mathrm{ext}_{\theta^{*}}$ ($\theta^{*}$ refers to the weights of the model after training) which outputs a tensor $y=\mathrm{ext}_{\theta^{*}}(x)$ of dimensions $(1+n_{\text{bits}},h,w)$ for an input image $x\in\mathbb{R}^{3\times h\times w}$.
This tensor consists of a watermark detection mask $y^{\text{det}}\in[0,1]^{1\times h\times w}$ and a decoding mask $y^{\text{dec}}\in[0,1]^{n_{\text{bits}}\times h\times w}$.
We denote by $y^{\text{det}}_{i}\in[0,1]$ the predicted detection score for pixel $i$, and by $y^{\text{dec}}_{i}=[y^{\text{dec}}_{i,1}\dots y^{\text{dec}}_{i,n_{\text{bits}
}}]\in[0,1]^{n_{\text{bits}}}$ its decoded message.

#### Localization

(or pixel-level detection) spots watermarked pixels of the image.
A pixel is deemed watermarked if its detection score $y^{\text{det}}_{i}$ exceeds a threshold $\tau$. Typically, we set
$\tau$ empirically, by measuring the False Positive Rate (FPR) on all the pixels of a held-out training set (the FPR is the probability of a non-watermarked pixel being flagged as such).

#### Detection

(or image-level detection) decides if an image is globally watermarked.
From the pixel-level detection score, an image soft detection score can naturally be computed as:

$$ $s_{\text{det}}:=\frac{1}{h\times w}\sum_{i=1}^{h\times w}\mathbbm{1}\{y^{\text {det}}_{i}>\tau\}\in[0,1].$ (1) $$

The image is flagged if $s_{\text{det}}$ is higher than a threshold that is the proportion of watermarked pixels that the user considers enough to deem a content as watermarked.

#### Decoding

of a single message within an image is done by the following weighted average of the pixel-level soft predictions of the hidden message:

$$ $\hat{m}_{k}=\left\{\begin{array}[]{ll}1&\text{if }\left[\frac{1}{\sum_{i} \mathbbm{1}\{y^{\text{det}}_{i}>\tau\}}\left(\sum_{i=1}^{h\times w}\mathbbm{1} \{y^{\text{det}}_{i}>\tau\}\cdot y^{\text{dec}}_{i,k}\right)\right]>0.5\,,\[8 .0pt] 0&\text{otherwise}\,.\end{array}\right.$ (2) $$

Decoding of multiple watermarks within one image uses a hard detection approach instead of the previous soft weighting.
We isolate pixels detected as watermarked, i.e., pixels $i$ with $y^{\text{det}}_{i}>\tau$, and we compute the local decoded message $\tilde{m}_{i}$ such that for any $k\in\{1,\ldots,n_{\text{bits}}\}$, $\tilde{m}_{i,k}=\mathbbm{1}\{y^{\text{dec}}_{i,k}>0.5\}$.
The DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm (Ester et al., ; Schubert et al., ) clusters the set of locally decoded messages.
It outputs some centroids and an assignment for every pixel detected as watermarked.
DBSCAN selects these centroids among the initial set, thus ensuring that the final decoded messages are binary words.

DBSCAN offers the advantage of not requiring a pre-defined number of clusters.
Instead, we need to specify two parameters:
(1) $\text{min}_{\text{samples}}$, the minimum number of points for a cluster to be considered valid,
and (2) $\varepsilon$, the maximum distance between two samples for one to be considered as in the neighborhood of the other.
For further details on the DBSCAN algorithm, see App. 9.1.

Handling multiple messages in a single image makes WAM robust against attacks that involve splicing several watermarked images together, which can compromise the decoding of traditional watermarking schemes in real-world scenarios.
It also enables active object detection (Asnani et al., ), where the objective is to track watermarked objects.
Additionally, it allows for the identification of the use of multiple watermarked AI tools.

## 4 Watermark Anything Models

### 4.1 The model

WAM considers two joint models: a watermark embedder $\mathrm{emb}_{\theta}$ and a watermark extractor $\mathrm{ext}_{\theta}$, where $\theta$ gathers the parameters of these functions.
The embedder defines the watermark procedure for hiding a message into an image, while the extractor detects watermarking and decodes messages.
This section discusses our choices for the different pieces that constitute WAM.
App. 10.1 gives the technical details of the network architectures.

Figure: Figure 2: The first training phase of WAM, as described in Sec. 4.2, jointly trains the *watermark embedder* to predict an additive watermark and the *watermark extractor* to detect watermarked pixels and decode the hidden message. In between, the augmenter 1) splices the watermarked and the original images based on a random mask and 2) applies classical image transformations.
Refer to caption: x2.png

#### The watermark embedder

($\mathrm{emb}_{\theta}$) consists of an encoder, a binary message lookup table, and a decoder.
The encoder represents an image in a latent space.
The lookup table is used to translate the message into a tensor which is concatenated to the image representation.
From that, the decoder outputs a watermark signal, which is added to the original image with adequate scaling.

The autoencoder that we consider is based on the architecture of the variational autoencoder of LDM (Rombach et al., ).
Its encoder, $\mathrm{enc}_{\theta}$, compresses a $h\times w$ input image $x$ into a latent $z\in\mathbb{R}^{d_{\text{z}}\times h^{\prime}\times w^{\prime}}$ (with downsampling factor $f=h/h^{\prime}=w/w^{\prime}$).
The binary message lookup table $\mathcal{T}_{\theta}$ is of shape $(n_{\text{bits}},2,d_{\text{msg}})$.
Each bit of the message $m$ is mapped to the embedding $\mathcal{T}_{\theta}(k,m_{k},\cdot)\in\mathbb{R}^{d_{\text{msg}}}$, depending on its position $k\in\{1,\ldots,n_{\text{bits}}\}$ and its value $m_{k}\in\{0,1\}$.
The $n_{\text{bits}}$ embeddings are averaged, resulting in a vector of size $d_{\text{msg}}$ which is repeated to form a tensor of shape $(d_{\text{msg}},h^{\prime},w^{\prime})$.
The concatenation to the image representation yields an activation of shape $(d_{\text{z}}+d_{\text{msg}})\times h^{\prime}\times w^{\prime}$.
The decoder, $\mathrm{dec}_{\theta}$, maps back this activation to the watermark signal $\delta_{\theta}(x,m)$ of the same shape as the original image.
The range of this signal is $[-1,1]$ because the last layer of $\mathrm{dec}_{\theta}$ is a hyperbolic tangent activation.
It is finally added to the original image to produce the watermarked output $x_{\text{m}}=\mathrm{emb}_{\theta}(x)=x+\alpha\cdot\delta_{\theta}(x,m)$.
Parameter $\alpha\in\mathbb{R}_{+}$ is called the watermark strength as it controls the distortion applied to the original image.

#### The watermark extractor

($\mathrm{ext}_{\theta}$) takes on the dual role of detecting watermarked pixels and decoding their embedded messages.
We use an architecture similar to SETR (Zheng et al., ) and Segment Anything (Kirillov et al., ).
It comprises a ViT encoder paired with a pixel decoder that upsamples the embeddings to the original image size.
The watermark extractor outputs a vector of size $1+n_{\text{bits}}$ for every pixel, as described in Sec. 3.

#### High-resolution.

WAM operates at a fixed resolution of $h\times w$.
To extend it to higher resolutions, an anisotropic scaling resizes the image to $h\times w$ and $\mathrm{emb}_{\theta}$ computes the watermark signal $\delta$ from this resized image.
A bilinear interpolation scales $\delta$ back to the size of the original image.
The extraction also operates at $h\times w$, by resizing all images before feeding them to the network.
Therefore the extraction process remains consistent with the pre-training conditions.
We only use these interpolations at embedding time.
WAM is thus only trained on low resolution images (Sec. 4.2) but can be used for high-resolution images too, which represents an important training compute gain.
A similar approach also appears in the recent literature (Bui et al., ).

### 4.2 Pre-training models for localized message embedding and extraction

The objective of this first training phase is to obtain a robust and localizable watermark hiding a $n_{\text{bits}}$-bit message inside parts of an image, without caring much about imperceptibility.
This stage does not incorporate any perceptual loss: our goal is to achieve perfect localization and decoding even after severe augmentations.
Figure 2 illustrates the detailed process.

#### Augmentations

are two-step processes.
They first splice images $x$ and $x_{\text{m}}$ based on a binary mask and then apply usual image transformations to improve robustness.

The first step randomly samples a mask among full masks, rectangles, irregular shapes, or object segmentation maps (possibly provided by the training dataset).
These masks are inverted with probability $0.5$.
It yields a mask $r_{\text{mask}}\in[0,1]^{h\times w}$.
Figure 2 shows examples of masks, with more samples displayed in Figure 6 of App. 10.2.
The first step then computes the spliced image as
$x_{\text{masked}}=r_{\text{mask}}\odot x_{\text{m}}+(1-r_{\text{mask}})\odot x$, where $\odot$ denotes component-wise multiplication.

The second step of the augmentation applies a processing typical from real-world image editing including geometric transformations (identity, resize, crop, rotate, horizontal flip, perspective) and valuemetric adjustments (JPEG, Gaussian blur, median filter, brightness, contrast, saturation, hue).
The geometric transformation alters the positions of watermarked pixels, so it is also applied to the mask to keep track of the watermarked areas.
This adjusted mask is the ground truth for the localization and it is denoted $y^{\text{det},\star}$ in the sequel.
These augmentations are relatively strong to reflect the transformations found on media sharing websites, as opposed to professional photo publications.
Additional details and discussions on augmentations are postponed to App. 10.4.

When a given image goes through the embedder-augmentation-extractor sequence, the final output depends on the models’ parameters $\theta$ (omitted so far for clarity):

$$ $y(\theta)=[y^{\text{det}}(\theta),y^{\text{dec}}(\theta)]=\mathrm{ext}_{\theta }\ \Big{(}\ \text{transformation}\ \big{(}\ \text{masking}\ \left(\ \mathrm{ emb}_{\theta}(x,m),\,x\ \right)\ \ \big{)}\ \ \Big{)}\ .$ (3) $$

#### Objectives.

The training minimizes the objective function $\ell(\theta)$ which is a linear combination of the detection and decoding losses:
$\ell(\theta)=\lambda_{\text{det}}\cdot\ell_{\text{det}}(\theta)+\lambda_{\text
{dec}}\cdot\ell_{\text{dec}}(\theta)$.

The detection loss is the average of the pixel-wise cross-entropy between $y^{\text{det}}(\theta)\in[0,1]^{h\times w}$ and the ground truth $y^{\text{det},\star}\in\{0,1\}^{h\times w}$ (pixel watermarked or not).
Similarly, the decoding loss is the average of the pixel-wise and bit-wise binary cross-entropy between $y^{\text{dec}}_{i,k}(\theta)$ and $m_{k}$ only over the watermarked pixels, where $m$ is a random message originally embedded in that image.
For a given image and message, $\ell_{\text{det}}$ and $\ell_{\text{dec}}$ are:

$$ $\displaystyle\ell_{\text{det}}(\theta)$ $\displaystyle=\frac{-1}{h\times w}\sum_{i=1}^{h\times w}\left[y^{\text{det}, \star}_{i}\log(y^{\text{det}}_{i}(\theta))+(1-y^{\text{det},\star}_{i})\log(1- y^{\text{det}}_{i}(\theta))\right],$ (4) $\displaystyle\ell_{\text{dec}}(\theta)$ $\displaystyle=\frac{-1}{n_{\text{bits}}\times\sum_{i=1}^{h\times w}y^{\text{ det},\star}_{i}}\sum_{i=1}^{h\times w}y^{\text{det},\star}_{i}\sum_{k=1}^{n_{ \text{bits}}}\left[m_{k}\log(y^{\text{dec}}_{i,k}(\theta))+(1-m_{k})\log(1-y^{ \text{dec}}_{i,k}(\theta))\right].$ (5) $$

### 4.3 Post-training for imperceptibility and multiple watermarks

The model trained in the first phase (Sec. 4.2) produces a too visible watermark.
Moreover, it cannot deal within multiple watermarks in an image. The second training phase addresses these issues.

#### Perceptual heatmap.

Figure: (a) Without JND
Refer to caption: x3.png

The Just-Noticeable-Difference (JND) map is a hand-crafted model of the minimum artifact perceivable by a human at every pixel.
For instance, artifacts on flat, uniform areas are more visible than on textured ones.
The JND map was introduced by Chou and Li () and later used by Wu et al. ().
App. 9.2 gives more details on its computation.

Given an image $x$, we have $\text{JND}(x)\in\mathbb{R}^{3\times h\times w}$.
We modulate the intensity of the watermark per pixel at embedding time:
the final watermarked image is computed as $x_{\text{m}}=x+\alpha_{\mathrm{JND}}\cdot\mathrm{JND}(x)\odot\delta_{\theta}(x
,m)$.
In practice, we found that applying this JND on the model trained in the first phase slightly degrades the detection and decoding performance.
The fine-tuning cancels this degradation.
Applying JND only in the second training phase makes training easier than previous methods relying on “perceptual” or “contradictory” losses (like StegaStamp (Tancik et al., ) or HiDDeN (Zhu et al., )).

#### Multiple watermarks.

Since the first training phase (Sec. 4.2) uses a single message, it tends to decode a constant message across pixels even on image regions with different messages.
The second training phase introduces several masks to address this limitation.
The masks are generated as before with distinct random messages in each of the masked areas.
The number of disjoint masks goes from 1 to 3 with probabilities 0.6, 0.2, and 0.2 respectively (see App. 10.2).

The detection loss $\ell_{\text{det}}$ takes the union of all masks as ground truth $y^{\text{det},\star}_{i}$.
The decoding loss $\ell_{\text{dec}}$ is computed separately for each message and the losses are summed up.
The scenario of multiple watermarks within a single image arises when watermarked images are combined.
Unlike other methods producing a global message, WAM can now distinguish and decode each message separately.

## 5 Experiments & Results

### 5.1 Implementation details

We provide here further implementation details using the notations introduced in Sec. 4.1, and the rest of the architectures are described in App. 10.1.
All experiments are run with $n_{\text{bits}}=32$.
There is a total of $1.1$M parameters for the embedder and $96$M for the extractor (equivalent to a ViT-base).
In our experiments, we found it necessary to increase the size of the extractor compared to the embedder.
This is probably because the extractor has more tasks to handle, such as segmenting and extracting multiple messages, while the embedder only needs to embed one message into an image.
Also, it is important for the embedding process to be quick since it happens on the user-side, so we keep it voluntarily small to be fast and usable at scale.
We train our model on the MS-COCO training set with blurred faces (Lin et al., ), that contains 118,000 images, many of them with segmentation masks.
We train at resolution $h\times w=256\times 256$, and with $f=8$ (so $h^{\prime}\times w^{\prime}=32\times 32$).
The first training phase (Sec. 4.2) is optimized with AdamW (Kingma, ; Loshchilov, ) with a linear warmup of the learning rate in $5$ epochs from $1\times 10^{-6}$ to $1\times 10^{-4}$ and a cosine annealing to $1\times 10^{-6}$.
We set $\lambda_{\text{dec}}=10$, $\lambda_{\text{det}}=1$, $\alpha=0.3$.
We train with a batch size of $16$ per GPU for 300 epochs using 8 V100 GPUs which takes roughly 2 days.
The second training phase (Sec. 4.3) further trains the model with the JND attenuation for 200 epochs, hiding up to 3 messages per image using either randomly sampled rectangles or segmentation masks.
During this phase, $\alpha_{\text{JND}}=2$.
App. 10.2 provides details on the generation of masks used during both training phases.
For the extraction of multiple messages, we use the Scikit-learn DBSCAN implementation (Pedregosa et al., ).

### 5.2 Quality

Table 1 evaluates quantitatively the difference between the watermarked and original images with PSNR, SSIM and LPIPS (Zhang et al., ) metrics.
We adapt the methods so that they are comparable in terms of imperceptibility.
We show qualitative examples for COCO and DIV2k images in App. 12.
The perceptual quality is good: the JND map successfully modulates the watermark signal in areas where the eye is not sensitive.
Nevertheless, repetitive and regular patterns can be observed in some images when the bright or textured areas are big enough (furs of animals for instance).

**Table 1: Evaluation of the watermark imperceptibility. We report the PSNR, SSIM, and LPIPS between watermarked and original images of COCO (low/mid-resolution) and DIV2k (high-resolution).**
|  |  | HiDDeN | DCTDWT | SSL | FNNS | TrustMark | WAM (ours) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| COCO | PSNR ($\uparrow$) | 38.2 | 37.0 | 37.8 | 37.7 | 40.3 | 38.3 |
| SSIM ($\uparrow$) | 0.98 | 0.98 | 0.98 | 0.98 | 0.99 | 0.99 |  |
| LPIPS ($\downarrow$) | 0.05 | 0.02 | 0.07 | 0.06 | 0.01 | 0.04 |  |
| DIV2K | PSNR ($\uparrow$) | 38.4 | 38.7 | 38.2 | 39.0 | 39.1 | 38.8 |
| SSIM ($\uparrow$) | 0.98 | 0.99 | 0.98 | 0.99 | 0.99 | 0.99 |  |
| LPIPS ($\downarrow$) | 0.07 | 0.03 | 0.11 | 0.04 | 0.01 | 0.03 |  |

### 5.3 Detection and decoding

We benchmark WAM’s performance against several watermarking methods:
DCTDWT (Al-Haj, ),
HiDDeN (Zhu et al., ),
TrustMark (Bui et al., ),
SSL (Fernandez et al., ),
FNNS (Kishore et al., ).
We also compare WAM to generation-time watermarking methods (Fernandez et al., ; Wen et al., ) in App. 11.1.
Although this list is not exhaustive, it covers the main types of state-of-the-art methods.
For HiDDeN, we use a model which hides 48 bits.
FNNS, SSL, and DCTDWT can hide messages or arbitrary length, we choose to hide 48 bits as well.
We use the first 16 bits for detection and 32 bits for message decoding.
For detection, an image is flagged as watermarked if at most 1 bit is wrongly decoded.
This corresponds to a theoretical False Positive Rate (FPR) of
$2.6\times 10^{-4}$, i.e., $2.6$ out of 10,000 images are falsely flagged as watermarked on average.
TrustMark (Bui et al., ) directly outputs a boolean (watermarked or not).
We use the version that hides $40$ bits, but only use the first 32 bits.
For all evaluations, we apply the watermark embedding and extraction at the original image resolution as described in the high-resolution paragraph of Sec. 4.1.
Finally, for WAM, an image is flagged if $s_{\text{det}}$ (Eq. 1) is higher than 0.07.
This value empirically delivers an approximately similar FPR on the COCO validation set.

We evaluate the robustness against various transformations: geometric (flip, crop, perspective, and rotation), valuemetric (adjustments in brightness, hue, contrast or saturation, median or Gaussian filtering, and JPEG compression), splicing: scenarios where only 10% of the image is watermarked superimposed onto the original or an other background.
We also evaluate the robustness against inpainting, using the LaMa (Suvorov et al., ) model to inpaint areas of the watermarked image specified by random or segmentation masks.
For COCO, we use the union of all segmentation masks for each image which in average corresponds to 30% of the image, and for DIV2k we use a randomly generated mask (because there are no segmentation masks) that covers around 35% of the image.
We give details on the compared baselines in App. 10.3 and on the transformations in App. 10.4.

2(a) presents the detection and decoding results for low/mid-resolution images, averaged over the first 10k images of the COCO validation set – the detailed results for every augmentation used to form the different groups are in App. 11.3.
2(b) does the same for high-resolution images from the DIV2k (Timofte et al., ) validation set.
For both distributions, WAM is competitive and even shows improved robustness on classical geometric/valuemetric transformations.
Most importantly, WAM performs better on splicing or inpainting where other methods fail to achieve more than 90% TPR and Bit acc.
This is true even if WAM was not explicitly trained on high resolution images or to be robust against inpainting.
We also show examples of WAM’s detection masks after inpainting in Fig. 10 of App. 11, where some modified parts of images are not detected as watermarked anymore.

**Table 2: Detection and decoding after image editing (detailed in Sec. 10.4). We show the bit accuracy (Bit acc.) between the encoded and decoded messages, the proportion of images correctly deemed watermarked (TPR), and the proportion of non-watermarked images falsely detected as watermarked (FPR), in %. Since HiDDeN, DCTDWT, SSL and FNNS do not naturally provide a detection result, we hide 48 bits and reserve 16 bits for detection, and flag an image as watermarked if it has strictly less than two bits incorrectly decoded (these baselines are detailed in Sec. 5.3).**
|  |  | Augmentations |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | None | Geometric | Valuemetric | Inpainting | Splicing |  |  |  |  |  |
| Method | FPR | TPR | Bit acc. | TPR | Bit acc. | TPR | Bit acc. | TPR | Bit acc. | TPR | Bit acc. |
| HiDDeN | 0.08 | 76.9 | 95.5 | 31.2 | 80.1 | 48.4 | 87.2 | 44.7 | 88.7 | 0.9 | 68.0 |
| DWTDCT | 0.02 | 77.1 | 91.4 | 0.0 | 50.5 | 13.6 | 58.1 | 47.8 | 81.7 | 0.8 | 59.9 |
| SSL | 0.00 | 99.9 | 100.0 | 14.3 | 76.5 | 70.5 | 92.1 | 67.7 | 91.1 | 0.4 | 58.9 |
| FNNS | 0.10 | 99.6 | 99.9 | 62.4 | 86.6 | 82.1 | 93.9 | 89.4 | 97.3 | 38.7 | 88.5 |
| TrustMark | 2.88 | 99.8 | 99.9 | 36.3 | 71.4 | 90.1 | 98.2 | 39.4 | 83.2 | 83.2 | 57.1 |
| WAM (ours) | 0.04 | 100.0 | 100.0 | 99.3 | 91.8 | 100.0 | 99.9 | 97.9 | 99.2 | 100.0 | 95.3 |

### 5.4 Localization

WAM and EditGuard (Zhang et al., ) are the only methods that provide watermark localization.
We therefore consider images at fixed resolution $512\times 512$ (unlike in Sec. 5.3) to align with the setup of EditGuard.
We nevertheless observe similar results at different resolutions for WAM.
We focus on the COCO validation set and splice centered watermarked rectangles of various sizes within the images to ensure a well-controlled experiment.
We then either apply no transformation or crop the upper left part of the image (25%), which is then resized to the original size.
By doing so, the proportion of watermarked pixels is still the same as in the spliced image before cropping, which allows us to evaluate the robustness of the localization.
Figure 7 of App. 10.5 illustrates this protocol.

Figure 4 reports the mean Intersection over Union (mIoU) to evaluate the localization, as commonly done in the segmentation literature.
Figure 4 evaluates the bit accuracy – through localization – following Eq. 2.
We observe that WAM accurately predicts both classes, even after cropping and resizing, except when the watermarked area covers 95% of the image, in which case the extractor tends to classify all pixels as watermarked.
In terms of bit accuracy, WAM recovers in average 31 out of 32 bits even when only 10% of the $256\times 256$ image is watermarked, and around 25 bits when only 10% of a 25% crop is watermarked (which corresponds to 2.5% of the overall number of pixels).
For both evaluations, WAM outperforms EditGuard which, in particular, is not robust to cropping.

Figure: Figure 4: Evaluation of the localization on the validation set of COCO, with or without cropping before extraction, following the setup described in Sec. 5.4. (Left) Localization accuracy using intersection over union between the predicted watermarked areas and the ground-truth mask. (Right) Bit accuracy between the ground truth message and the decoded message, computed from Eq. (2).
Refer to caption: x5.png

### 5.5 Multiple watermarks

We compare the detection and decoding of multiple watermarks before and after the second training phase of WAM (Sec. 4.3).
We embed up to five distinct messages into five separate 10% areas of every image (first resized to $256$ to ease the experiments).
This is done by feeding the image several times to WAM’s embedder, then pasting the different watermarked areas onto the original image.
These areas are squares disposed in a checkerboard pattern, which ensures that the watermarked areas do not overlap and have the same size (see Fig. 8 of App. 10.6).
This is an arbitrary choice made to remove confounding factors during evaluation, although the training does not require the watermarked areas to be squared nor to have the same size.
Following the methodology detailed in Sec. 3, we apply the DBSCAN algorithm to the vector outputs $\tilde{m}_{i}$ of the extractor corresponding to all pixels $i$ identified as watermarked ($y^{\text{det}}_{i}>\tau$).
It identifies clusters corresponding to different watermarked regions without requiring the number of hidden messages to be known in advance.
We use $\tau=0.5$ to threshold the watermarked pixels, and we choose a rather strict setup with $\varepsilon=1$ and $\text{min}_{\text{samples}}=1000$ pixels (around $2\%$ of the image) in our evaluations, resulting from an hyperparameter search detailed in Fig. 9 of App. 11.2.
This means that a cluster will be considered only if it has at least 2% of the image’s pixels, and that the maximum distance between two predicted messages to be neighbors is 1.

Figure 5 presents the results.
We compute the bit accuracies by comparing the centroid of each detected cluster (decoded message) with the ground truth message that has the largest overlapping area: the bit accuracy is thus computed only across the clusters that are discovered by DBSCAN.
Without the second phase of training, the messages get mixed up, and WAM predicts the same (wrong) message for all watermarked pixels.
After the training, WAM is able to accurately extract up to five different 32-bit messages from the areas covering 10% of the image each.
Therefore, although the second phase of training embeds between 1 and 3 watermarks per image, WAM generalizes to more watermarks.
It also shows that WAM’s effective capacity is greater than 32 bits (since it can hide multiple messages) if we do not consider heavy crops as a transformation we want the method to be robust against, similarly as (Bui et al., ; Tancik et al., ; Wen et al., ).

This method remains effective after some image manipulations; extended quantitative results are presented in Fig. 9 of App. 11.2. For example, WAM achieves 85% mIoU even when the images have been horizontally flipped and the contrast adjusted. Furthermore, the bit accuracy, averaged across $5$ messages totaling 160 bits, exceeds 95% under the same conditions. However, WAM fails when JPEG compression is also added on top of these two transformations. Illustrative examples of the identified clusters in various scenarios can be found in Fig. 8 of App. 10.6.

Figure: (a) After first training phase (PSNR $\approx 25$ dB)
Refer to caption: x6.png

## 6 Conclusion, Limitations, and Future Work

#### Conclusion.

This work introduces the Watermark Anything Model, which approaches image watermarking as a segmentation task.
WAM is able to predict whether an image is watermarked or not, as well as to localize its watermarked regions.
It thus handles images with a small watermarked part, or where parts of the image have been removed or edited.
Additionally, it is able to detect and extract multiple watermarks within the same image, for the first time in the literature.
Our training which introduces
localization under heavy augmentations also offers state-of-the-art robustness on classical settings considered by current watermark methods, where most of the image is watermarked.

We identify two main limitations that we address in the next paragraphs.

#### Low payload.

In our experiments, WAM’s capacity is limited to 32 bits, and training on larger messages is challenging.
In contrast, other watermarking methods such as EditGuard or RoSteALS can successfully embed more than 100 bits per image, but are not robust to crops or outpainting.
This is a trade-off between capacity and robustness.
Note that in practice, other methods use decoding to match with the original message and conclude if an image is watermarked.
In contrast, WAM’s detection process is separate from the decoding, so 32 bits may be enough for most applications.

#### Perceptual quality.

In spite of the JND weighting, we notice that the watermark can still be visible in some areas of the watermarked images, even at a relatively high PSNR (see examples in App. 12).
This could be due to the JND map focusing only on the cover image and not on the watermark signal itself.
Improvements might be achieved by employing more sophisticated Human Visual System (HVS) models or by regularizing the watermark to eliminate repetitive patterns during training.

## Reproducibility statement

Section 5 provides the specifics of our training and evaluation setup.
Further implementation details, including network architectures, mask designs, transformation parameters for robustness evaluation, and settings for baseline comparisons, are presented in App. 10.
The models and code for inference and training used in this paper will be made available upon publication.
The code is based on PyTorch, the training dataset is publicly available, and the training requires half a week on 8 V100 GPUs, which makes the models reproducible at reasonable cost.

The training and inference code, as well as trained models are available at: [https://github.com/facebookresearch/watermark-anything](https://github.com/facebookresearch/watermark-anything)

## 7 Ethical Statement

### 7.1 Societal Impact

Watermarking in general improves the traceability of content, be it AI-generated, or not.
It can have positive consequences, for example when it is used to trace the origin of fake news or to protect intellectual property.
This traceability can also have negative consequences, for example when it is used to trace political opponents in authoritarian regimes or whistleblowers in secretive companies.
Besides, it is not clear how to disclose watermark detection results, which may foster a closed ecosystem of detection tools.
It may also exacerbate misinformation by placing undue emphasis on content that is either not detected, generated by unknown models, or authentic but used out of context.
We however believe that the benefits of watermarking outweigh the risks, and that the development of robust watermarking methods is a positive step for our society.

### 7.2 Environmental impact

The cost of the experiments and of model training is high, though order of magnitude less than other computer vision fields (Oquab et al., ).
One training with a schedule similar to the one reported in the paper represents $\approx 30$ GPU-days.
We also roughly estimate that the total GPU-days used for running all our experiments to $5000$, or $\approx 120$k GPU-hours.
This amounts to total emissions in the order of 20 tons of CO2eq.
Estimations are conducted using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented by Lacoste et al. ().
We do not consider in this approximation: memory storage, CPU-hours, production cost of GPUs/ CPUs, etc.

We were careful to limit the environmental impact of our research by doing most of our experiments on smaller models, on fewer epochs ($100$) and at lower resolutions ($128\times 128$), before scaling up to larger models, more epochs and higher resolutions.
We also focus on small specialized model that are efficient at inference time and more environmental-friendly (Luccioni et al., ).
At the end of the day, we believe that the environmental cost of this research is justified by the potential benefits of WAM.

## 8 Extended Related Work on Image Watermarking

### 8.1 Traditional watermarking

We call traditional watermarking a technique embedding a digital watermark in a host content.
As far as images are concerned, the first methods are usually classified into two categories depending on the space in which the watermark is embedded.
In spatial domain, the watermark is encoded by directly modifying pixels, such as flipping low-order bits of selected pixels (Van Schyndel et al., ).
For example, Nikolaidis and Pitas () slightly modify the intensity of randomly selected image pixels while taking into account properties of the human visual system, robustly to JPEG compression and lowpass filtering.
Bas et al. () create content descriptors defined by salient points and embed the watermark by adding a pattern on triangles formed by the tessellation of these points.
Ni et al. () use the zero or the minimum points of the histogram of an image and slightly modifies the pixel grayscale values to embed data into the image.
The second category frequency domain watermarking offers better robustness.
It usually spreads a pseudorandom noise sequence across the entire frequency spectrum of the host signal (Cox et al., ).
The first step is a transformation that computes the frequency coefficients.
The watermark is then added to these coefficients, taking into account the human visual system.
The coefficients are mapped back onto the original pixel space through the inverse transformation to generate the watermarked image.
The transform domains include Discrete Fourier Transform (DFT) (Urvoy et al., ),
Quaternion Fourier Transform (QFT) (Bas et al., ; Ouyang et al., ),
Discrete Cosine Transform (DCT) (Bors and Pitas, ; Piva et al., ; Barni et al., ; Li et al., ),
Discrete Wavelet Transform (DWT) (Xia et al., ; Barni et al., ; Furon and Bas, ),
both DWT and DCT (Feng et al., ; Zear et al., ), etc.

Deep learning-based methods have recently emerged as alternatives for traditional watermarking.
The first attempts use neural networks as a fixed transform into a latent space (Vukotić et al., , ; Kishore et al., ; Fernandez et al., )
Since there is no inverse transform, the embedding is done iteratively by gradient descent over the pixels.
The recent approaches are often built as embedder/extractor networks. They are trained end-to-end to invisibly encode information while being resilient to transformations applied during training.
This makes it easier to build robust systems and avoids algorithms hand-crafted for specific transformations.
HiDDeN (Zhu et al., ) is a famous representative of this approach and has been extended in several ways.
Luo et al. () add adversarial training in the attack simulation to bring robustness to unknown transformations.
Zhang et al. (, ); Yu () use an attention filter further improving imperceptibility.
Ahmadi et al. () adds a circular convolutional layer that helps spreading the watermark signal over the image.
Wen and Aydore () use robust optimization with worst-case attack as if an adversary were trying to remove the mark.
Many other approaches focused on improving robustness, imperceptibility, speed, etc. (Jia et al., ; Bui et al., , ; Huang et al., ; Evennou et al., ; Pan et al., ).

Another line of works focuses on steganography (Baluja, ; Wengrowski and Dana, ; Zhang et al., ; Tancik et al., ; Jing et al., ; Ma et al., )
that pursues a different goal.
Steganography hides a message in the image without leaving any statistical traces, but the robustness is null or limited.

### 8.2 Watermarking for generative AI

The most trendy research direction in watermarking is arguably the detection of AI-generated images for transparency and for filtering such results when building new models.
The *Post-generation* or *post-hoc* approach uses traditional watermarking: the content is first generated and then watermarked.
On the contrary, the *generation-time* methods natively generate watermarked images.
Watermarking no longer incurs additional runtime and is more robust and secure than post-hoc.
We broadly categorize generation-time watermarking into the two following categories.

*In-model* methods modify the weights of the generative model.
This allows open-sourcing the model without revealing the watermark.
The earliest methods (Wu et al., ; Yu et al., ; Zhao et al., ) watermark the images of the training set with the hope that the model learns what watermark is during its training.
This is computationally expensive and not scalable.
Alternatively, some proposals (Fei et al., , ) train Generative Adversarial Networks (GAN) with additional watermarking losses such that generated images contain the watermark.
Stable Signature (Fernandez et al., ) focuses on Latent Diffusion Models (LDM) and fine-tunes the latent decoder to embed the watermark, while Feng et al. () fine-tune the U-Net that predicts the latent diffusion noise instead.
To eliminate the need to fine-tune the model for every user,
some papers use a hyper-network predicting the modifications to be applied to the generative model, be it a GAN (Yu et al., ; Fei et al., ) or diffusion-based (Kim et al., ).

*Out-of-model* methods alter the generation process.
This is easier to implement since it does not require training or fine-tuning the model.
Ci et al. () and Rezaei et al. () propose an adapter to the decoder that takes the secret message as input.
A different class of out-of-model methods, specific to diffusion models, embed the watermark by adding patterns to the initial noise (seed).
For example, Tree-Ring (Wen et al., ) adds circular patterns to the initial noise and inverts the diffusion process to extract the watermark.
As follow-up works, Hong et al. () improve the inversion of the diffusion process, Ci et al. () extend the method to multi-bit watermarking, and Lei et al. () deploy the embedder/extractor framework over the noise.

## 9 Algorithms details

### 9.1 DBSCAN

DBSCAN (Density-Based Spatial Clustering of Applications with Noise) (Ester et al., ; Schubert et al., ) is a clustering method that groups points in a dataset based on their density and proximity to each other. In our case, points are locally decoded messages (one per pixel deemed as watermarked).
It works as follows:

- 1.
Initialization: Set the parameters $\varepsilon$ (maximum distance between two samples for one to be considered as in the neighborhood of the other) and $\text{min}_{\text{samples}}$ (minimum number of samples in a cluster).
- 2.
Neighborhood search: For each point $p$ in the dataset, find all points within a distance of $\varepsilon$ from $p$. This forms the neighborhood of $p$.
- 3.
Cluster formation: If the neighborhood of $p$ contains at least $\text{min}_{\text{samples}}$ points, form a cluster around $p$. Otherwise, mark $p$ as noise.
- 4.
Cluster expansion: For each point $q$ in the cluster, find all points within a distance of $\varepsilon$ from $q$.
Add these points to the cluster if they are not already part of it.
- 5.
Repeat steps 3-4: Continue expanding the cluster until no more points can be added.
- 6.
Assign labels: Assign a label to each point in the dataset indicating which cluster it belongs to (if any).

It is important to note that $\varepsilon$ is neither a hard boundary nor a maximum bound on the distances of points within a cluster: points that are further apart than $\varepsilon$ but still within the neighborhood of a core point (a point with at least $\text{min}_{\text{samples}}$ neighbors within $\varepsilon$ distance) can also be part of the same cluster.

One of the significant advantages of DBSCAN is its ability to cluster points without knowing the number of clusters beforehand, unlike some other clustering algorithms like K-Means.
Additionally, it identifies arbitrarily shaped clusters (watermarked area in our case), including those that may be surrounded by a different cluster.
This clustering is also robust to outliers.
However, DBSCAN also has some limitations.
It is sensitive to hyper-parameters $\varepsilon$ and $\text{min}_{\text{samples}}$, and to the choice of the distance metric (bit difference in our case), which may affect the clustering results.
DBSCAN may also be computationally expensive for a large number of data points.
This is less of an issue in our case since it is lower than $256\times 256$.

We use the Scikit-learn (Pedregosa et al., ) implementation of DBSCAN in our experiments.

### 9.2 Just-Noticeable-Difference

The maximum change that the human visual system (HVS) cannot perceive is referred to as the Just-Noticeable-Difference (JND) (Krueger, ).
It is used in image/video watermarking, compression, quality assessment, etc.
JND models in the pixel domain directly calculate the JND at each pixel location (i.e., how much pixel difference is perceivable by the HVS).

This section describes in detail the JND map used in Sec. 4.3.
It is based on the work of Chou and Li ().
We use this model for its simplicity, its efficiency, and its good qualitative results.
More complex HVS models could also be used if higher imperceptibility is needed (Watson, ; Yang et al., ; Zhang et al., ; Jiang et al., ).
The JND map takes into account two characteristics of the HVS, namely the luminance adaptation (LA) and the contrast masking (CM) phenomena.
We follow the same notations as Wu et al. (); Fernandez et al. (), and consider images that are in the range $[0,255]^{3\times 256\times 256}$.

#### Luminance masking.

Luminance masking refers to the phenomenon where the HVS is less sensitive to distortions in bright regions of an image.
We denote by $K_{lum}$ the luminance kernel, and $x$ the input image. The local background luminance of the image is:

$$ $\displaystyle B(x)(i,j)=\frac{1}{32}\sum_{k,l\in[-2,2]^{2}}K_{lum}(k,l)\cdot x (i+k,j+l),\text{ with }K_{lum}=\begin{bmatrix}1&1&1&1&1\\ 1&2&2&2&1\\ 1&2&0&2&1\\ 1&2&2&2&1\\ 1&1&1&1&1\end{bmatrix}.$ (6) $$

These luminance values are then post-processed as follows
to account for the non-linear response of the HVS:

$$ $LA(x)(i,j)=\begin{cases}17\cdot\left(1-\sqrt{\frac{B(x)(i,j)}{127}+\epsilon} \right)+3,&B(x)(i,j)\leq 127\\ \frac{3}{128}\cdot(B(x)(i,j)-127)+3,&B(x)(i,j)>127,\end{cases}$ (7) $$

where $\epsilon$ is a small positive value to ensure differentiability during back-propagation.

#### Contrast masking.

Contrast masking refers to the phenomenon where the HVS is less sensitive to distortions in regions of high contrast.
The gradient magnitude is:

$$ $\displaystyle C(x)(i,j)=\sqrt{\left(\sum_{k,l}K_{X}(k,l)\cdot x(i+k,j+l)\right )^{2}+\left(\sum_{k,l}K_{Y}(k,l)\cdot x(i+k,j+l)\right)^{2}},$ (8) $\displaystyle\text{with }K_{X}=\begin{bmatrix}-1&0&1\\ -2&0&2\\ -1&0&1\end{bmatrix},K_{Y}=\begin{bmatrix}-1&-2&-1\\ 0&0&0\\ 1&2&1\end{bmatrix},$ (9) $$

where $K_{X}$ and $K_{Y}$ are the horizontal and vertical gradient kernels, and $x$ is the input image.
To account for the non-linear response of the HVS, the contrast masking values $CM(x)$ at each pixel location:

$$ $CM(x)=\frac{16\cdot C(x)^{2.4}}{C(x)^{2}+26^{2}}.$ (10) $$

#### Heatmap generation.

The heatmap generation component of the JND model combines the luminance masking and contrast masking values to produce the JND heatmap:

$$ $\displaystyle H(x)=LA(x)+CM(x)-\gamma\cdot\min(LA(x),CM(x)),$ (11) $$

where $\gamma$ is a parameter that controls the trade-off between luminance masking and contrast masking.

For color images, we compute the heatmap from the image’s luminance.
Then we repeat it over the 3 color channels but with a scaling that differs for each channel: $H_{\mathrm{JND}}=[\alpha_{R}H,\alpha_{G}H,\alpha_{B}H]$, where $(\alpha_{R},\alpha_{G},\alpha_{B})=(1,1,2)$, because the human eye is more sensitive to red and green than blue color shifts.
This produces slightly more distortion in the blue channel.

## 10 Implementation details & parameters

### 10.1 Architectures of the watermark embedder/extractor

We hereby detail the modeling choices and architectures of Sec. 4.1.
Our models, in particular the embedder, are kept voluntarily small to be fast and usable at scale.
For instance the embedder and extractor described bellow have respectively $1.1$M and $96$M ($\approx$ ViT-Base) parameters.
Further work could explore how to build larger models with same throughput to improve the results.

In the following we consider an image $x\in\mathbb{R}^{3\times H\times W}$ and a message $m\in\{0,1\}^{n_{\text{bits}}}$
The embedder and extractor operate at resolution $h\times w=256\times 256$ (see Sec. 4.1).

#### Embedder.

Our goal is to embed a message in an image in a way that is imperceptible to the human eye.
This task is similar in many ways to image compression or image-to-image translation.
The most used architectures for this arguably come from the works of Rombach et al. (); Esser et al. (), which strike a very good balance between image quality and efficiency.

The encoder and decoder are described in Tab. 3 (we refer to the VQGAN paper (Esser et al., ) for more details).
They mainly consist of residual blocks optionally followed by upsampling or downsampling blocks.
For the encoder, we use $m=4$ residual blocks with output channels $d=32,32,32,d^{\prime}=64$, downsampling factor of $2$ for the $3$ first blocks (leading to a division by $f=8$ of the edge size of the latent map), and $d_{\text{z}}=4$.
The decoder mirrors the encoder, and we choose $d_{\text{msg}}=n_{\text{bits}}=32$.
The Up block interpolates the activation map to the new size, then applies a 2D convolution with kernel size $3$, stride $1$ and padding $1$.
In particular, we choose not to use deconvolution layers (ConvTranspose2D) because of the checkerboard patterns they introduce (Odena et al., ).
The Down block average-pools the activation map with $2\times 2$ kernels with stride $2$.

Our task does not need a bottleneck, since we are not interested in learning compressed representations.
Therefore the autoencoder predicts a signal $\delta\in[-1,1]^{3\times H\times W}$ – and not directly the final image – which is added to the image with a residual layer.
Another difference is that the message is embedded in the latent space of the encoder, which means that the first layer of the decoder is bigger.
As a reminder, the binary message lookup table is $\mathcal{T}_{\theta}\in\mathbb{R}^{n_{\text{bits}}\times 2\times d_{\text{msg}}}$.
Each bit of the message $m$ is mapped to the embedding $\mathcal{T}_{\theta}(k,m_{k},\cdot)\in\mathbb{R}^{d_{\text{msg}}}$, depending on its position $k\in\{1,\ldots,n_{\text{bits}}\}$ and its value $m_{k}\in\{0,1\}$, then repeated, which yields
$z_{\text{msg}}=\text{repeat}\left(1/n_{\text{bits}}\sum_{k=1}^{n_{\text{bits}}
}\mathcal{T}_{\theta}(k,m_{k},\cdot)\right)\in\mathbb{R}^{d_{\text{msg}}\times
3
2\times 32}$.

**Table 3: High-level architecture of the encoder and decoder of the watermark embedder $\mathrm{emb}_{\theta}$. The design of the networks follows the architecture presented by Ho et al. (); Esser et al. (); Rombach et al. ().**
| Encoder | Decoder |
| --- | --- |
| $x\in\mathbb{R}^{3\times H\times W},m\in\{0,1\}^{n_{\text{bits}}}$ | $(z,z_{\text{msg}})\in\mathbb{R}^{(d_{\text{z}}+d_{\text{msg}})\times 32\times 32}$ |
| Interpolation, Conv2D $\to\mathbb{R}^{d\times 256\times 256}$ | Conv2D $\to\mathbb{R}^{d^{\prime}\times 32\times 32}$ |
| $m\times$ $\{$ Residual Block, Down Block$\}$ $\to\mathbb{R}^{d^{\prime}\times 32\times 32}$ | Residual Block $\to\mathbb{R}^{d^{\prime}\times 32\times 32}$ |
| Residual Block $\to\mathbb{R}^{d^{\prime}\times 32\times 32}$ | Non-Local Block $\to\mathbb{R}^{d^{\prime}\times 32\times 32}$ |
| Non-Local Block $\to\mathbb{R}^{d^{\prime}\times 32\times 32}$ | Residual Block $\to\mathbb{R}^{d^{\prime}\times 32\times 32}$ |
| Residual Block $\to\mathbb{R}^{d^{\prime}\times 32\times 32}$ | $m\times$ $\{$ Residual Block, Up Block$\}$ $\to\mathbb{R}^{d\times 256\times 256}$ |
| GroupNorm, Swish, Conv2D $\to\mathbb{R}^{d_{\text{z}}\times 32\times 32}$ | GroupNorm, Swish, Conv2D $\to\mathbb{R}^{3\times 256\times 256}$ |
| $\mathcal{T}_{\theta}$(m), Repeat $\to\mathbb{R}^{d_{\text{msg}}\times 32\times 32}$ | TanH, Interpolation $\to[-1,1]^{3\times H\times W}$ |

**Table 4: High-level architecture of the encoder and decoder of the watermark extractor $\mathrm{ext}_{\theta}$. The design of the networks follows the architecture presented by Zheng et al. (); Kirillov et al. ().**
| Image encoder (ViT) | Pixel decoder (CNN) |
| --- | --- |
| $x\in\mathbb{R}^{3\times H\times W}$ | $z\in\mathbb{R}^{d^{\prime}\times 16\times 16}$ |
| Interpolation $\to\mathbb{R}^{3\times 256\times 256}$ | $m^{\prime}\times$ $\{$ Residual Block, Up Block$\}$ $\to\mathbb{R}^{d^{\prime\prime}\times 256\times 256}$ |
| Patch Embed (Conv2D), Pos. Embed $\to\mathbb{R}^{d\times 16\times 16}$ | Linear $\to\mathbb{R}^{(1+n_{\text{bits}})\times 256\times 256}$ |
| $m\times$ $\{$ Transformer Block $\}$ $\to\mathbb{R}^{d\times 16\times 16}$ | Sigmoid (optional) $\to\mathbb{R}^{(1+n_{\text{bits}})\times 256\times 256}$ |
| LayerNorm, GELU, Conv2D $\to\mathbb{R}^{d^{\prime}\times 16\times 16}$ | Interpolation $\to\mathbb{R}^{(1+n_{\text{bits}})\times H\times W}$ |

#### Extractor.

Our goal is to detect and extract the message from an image, at the pixel level.
This task is similar to image segmentation.
Our extractor is based on a vision transformer (ViT) (Dosovitskiy, ) followed by a pixel decoder, as commonly done in the literature (Kirillov et al., ; Zheng et al., ; Oquab et al., ).

The architecture of the extractor is detailed in Tab. 4.
The encoder is a ViT which consists of a series of attention blocks to process the image’s patches into a high-dimensional feature space.
We use the ViT-Base architecture ($86$M parameters), with patch size $16$ (Dosovitskiy, ), with $d=d^{\prime}=768$.
The patch embeddings are then upscaled by the pixel decoder to predict the segmentation masks and messages for every pixel.
The latter uses $m^{\prime}=3$ Up blocks which upsample (bilinearly) the activation map by factors $4,2,2$ respectively (total upscale factor is $16$, which is the patch size used in the ViT encoder).
Each Up block is followed by a Conv2D with kernel size of $3$ and stride of $1$, a LayerNorm, and a GELU activation.
The number of channels output by the convolution is its number of input channels divided by the upsampling factor of the Up block that precedes.
We obtain a latent map of shape $(d^{\prime\prime}=d/16,256,256)$, which is mapped to $(1+n_{\text{bits}})$-dimensional pixel features by a linear layer.
Finally, a Sigmoid layer scales the outputs to $[0,1]$ (this is in fact only done at inference, since the training objective implicitly applies it in PyTorch).

### 10.2 Mask generation

We follow the protocol of LaMa (Suvorov et al., ), and generate most of our masks used during the two phases of training by adapting the authors’ code [github.com/advimman/lama](https://github.com/advimman/lama).
Figure 6 shows some qualitative examples.
More specifically, a diverse set of random masks are used:

- •
Box-shaped masks are defined with a margin of 10 pixels and bounding box sizes width and height randomly chosen, ranging from 30 to 100 pixels.
They can be generated multiple times (1 to 3 times) per image.
- •
Full-image masks cover the entire image.
- •
Segmentation-based masks are segments corresponding to object boundaries or specific areas within an image.
We use the masks of the COCO dataset, and either select the union of all masks with probability 0.5, or the union of a random number of objects.
- •
Irregular masks are random brush strokes, characterized by parameters such as maximum and minimum angles (up to 4 degrees), lengths and widths (ranging from 20 to 50 pixels).
The brush strokes can be applied between 1 to 5 times per image.

During the first training phase (described in Sec. 4.2), we use irregular, box-shaped, full-image, and segmentation-based masks, each with a probability of 0.25.
Additionally, there is a 50% chance of inverting any of the masks.

During the second training phase (described in Sec. 4.3), we use the same masks except that:

- •
for “box-shaped masks”, with probability 0.5, instead of outputting a single mask as in the first phase, we output a random number from 1 to 3 of non overlapping rectangles;
- •
for “segmentation-based masks”, with probability 0.5, instead of outputting a single mask as in the first phase, we output a random number from 1 to 3 of segmented objects.

When randomly selected during this post-training phase, both types of multiple masks cannot be inverted and they are used to hide different watermarks within the same image.

Figure: (a) Random masks
Refer to caption: x8.jpg

### 10.3 Watermarking methods

We follow the evaluation setup used by Fernandez et al. ().
For HiDDeN (Zhu et al., ), we use the model available in [github.com/facebookresearch/stable_signature](https://github.com/facebookresearch/stable_signature) and modulate the output with the same JND mask.
For DCTDWT, we use the implementation of [github.com/ShieldMnt/invisible-watermark](https://github.com/ShieldMnt/invisible-watermark) (the one used in Stable Diffusion).
For SSL Watermark (Fernandez et al., ) and FNNS (Kishore et al., ) the watermark is embedded by optimizing the image, such that the output of a pre-trained model is close to the given key.
SSL Watermark uses a model pre-trained with DINO (Caron et al., ), while FNNS uses the HiDDeN extractor used in all our experiments, and not SteganoGan (Zhang et al., ) as in the original paper.
We optimize the distortion image for $10$ iterations, and modulate it with the same JND mask.
This avoids visible artifacts and gives a PSNR comparable to our method ($\approx 38$dB).
For TrustMark (Bui et al., ), we use the “C variant” with $40$ bits in the official implementation [github.com/adobe/trustmark](https://github.com/adobe/trustmark), as it achieves a similar PSNR.

### 10.4 Transformations

Transformations seen at training time and evaluated in Sec. 5 simulate common image processing steps.
We categorize them into different groups:
*valuemetric*, which change the pixel values;
*geometric*, which modify the image’s geometry;
*splicing*, which add watermarked areas inside the image;
and *inpainting*, which fill masked areas with plausible content.
The geometric and valuemetric transformations are displayed in Tab. 5 and detailed in the following table:
Transformation
Type
Parameter
Training
Evaluation
Brightness
Valuemetric
from torchvision
Random between 0.5 and 2.0
1.5 and 2.0
Contrast
Valuemetric
from torchvision
Random between 0.5 and 2.0
1.5 and 2.0
Hue
Valuemetric
from torchvision
Random between -0.1 and 0.1
-0.1 and 0.1
Saturation
Valuemetric
from torchvision
Random between 0.5 and 2.0
1.5 and 2.0
Gaussian blur
Valuemetric
kernel size $k$
Random odd between 3 and 17
3 and 17
Median filter
Valuemetric
kernel size $k$
Random odd between 3 and 7
3 and 7
JPEG
Valuemetric
quality $Q$
Random between 40 and 80
50 and 80
Horizontal flip
Geometric
NA
Crop
Geometric
edge size ratio $r$
Random between 0.33 and 1.0
0.33 and 0.5
Resize
Geometric
edge size ratio $r$
Random between 0.5 and 1.5
0.5
Rotation
Geometric
angle $\alpha$
Random between -10 and 10
-10 and 10
Perspective
Geometric
distortion scale $d$
Random between 0.1 and 0.5
0.1 and 0.5
For crop and resize, each new edge size is selected independently, which means that the aspect ratio can change (because the extractor resizes the image).
Moreover, an edge size ratio of $0.33$ means that the new area of the image is $0.33^{2}\approx 10\%$ times the original area.
For brightness, contrast, saturation, and sharpness, the parameter is the default factor used in the PIL and Torchvision (Marcel and Rodriguez, ) libraries.
For *splicing*, we crop a random area of the image and paste it back at the same location on the original image or on a different background image.

For evaluation (Sec. 5.3) we select some transformations from the list above and apply them with the parameters given in the table.
We chose these parameters high enough to have pronounced effects on the robustness of the watermark.
In practice, they are quite strong and would not not be encountered often in real-world scenarios.
Additionally we evaluate the robustness against *inpainting* which is not seen at training time.
We use LaMa (Suvorov et al., ) to modify masked areas in the image conditioned on the rest of the image.

**Table 5: Illustration of transformations evaluated in Sec. 5.**
| Identity | Contrast 0.5 | Contrast 1.5 | Brightness 0.5 | Brightness 1.5 |
| --- | --- | --- | --- | --- |
|  |  |  |  |  |
| Hue 0.5 | Saturation 1.5 | Median filter 7 | Gaussian blur 17 | JPEG 40 |
|  |  |  |  |  |
| Crop 0.33 | Resize 0.5 | Rotation 10 | Perspective 0.5 | Horizontal flipping |
|  |  |  |  |  |

### 10.5 Localization experiments

Figure: Figure 7: Experimental protocol for the evaluation of watermark localization as performed in Sec. 5.4.
Refer to caption: x12.png

Figure 7 gives an example of how the evaluation for watermark localization is performed in Sec. 5.4, for the “crop 0.25” augmentation.
The mask used to place the watermark (second image) is the same for each image with an area covering from 10% to 100% of the image.
We perform a 25% crop of the upper left corner.
After that, we resize the image to its original size.
Note that after this augmentation, the watermarked area still recovers the same proportion of the crop (by design, as the watermarked area was originally centered).

### 10.6 Multiple Watermarks

Figure 8 shows an example of how the evaluation for multiple watermarks is performed in Sec. 5.5.
After watermarking 5 areas of the image that are 10% each with different messages, the extractor detects several watermarked areas and outputs one message for each.

Figure: (a) After the first phase of the training, evaluation without augmentation
Refer to caption: x13.png

## 11 Additional Results

### 11.1 Comparison with watermarking methods for generative models

**Table 6: Detection results for WAM and generation-time watermarking methods, on 1k negative (non-watermarked), and 1k positive (watermarked), possibly edited, images. AUC refers to the Area Under the ROC curve, TPR@$10^{-2}$ is the TPR at FPR$=10^{-2}$. Stable Signature (Fernandez et al., ) embeds a 48-bit message and uses the bit accuracy as score, while Tree-Ring (Wen et al., ) and WAM output a detection score. WAM is competitive with watermarking methods for generative models (although the latter offer noteworthy advantages).**
|  | AUC | TPR@$10^{-2}$ | TPR@$10^{-4}$ |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Stable Sig. | Tree-Ring | WAM | Stable Sig. | Tree-Ring | WAM | Stable Sig. | Tree-Ring | WAM |
| None | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| Valuemetric | 0.94 | 1.00 | 1.00 | 0.90 | 1.00 | 1.00 | 0.90 | 1.00 | 1.00 |
| Geometric | 1.00 | 0.91 | 1.00 | 0.97 | 0.56 | 1.00 | 0.87 | 0.43 | 1.00 |
| Comb. | 1.00 | 0.75 | 1.00 | 0.99 | 0.05 | 1.00 | 0.99 | 0.01 | 1.00 |
| Splicing | 0.97 | 0.72 | 1.00 | 0.90 | 0.00 | 1.00 | 0.81 | 0.00 | 0.00 |

So far, we only considered general watermarking methods that apply the watermark on existing images.
We now compare WAM to methods specific for LDM, which watermark at generation time.
We generate 1k images from text prompts with Stable Signature (Fernandez et al., ), Tree-Ring (Wen et al., ) and without watermarking.
We also generate a set of watermarked images with WAM from this last set.
For each method, we retrieve a detection score for both the watermarked set of images and the non-watermarked images,
that we use to compute the Receiver Operating Characteristic (ROC) curve and the Area Under the Curve (AUC) for each method.
In the case of Stable Signature, we embed a 48-bit binary message and the score is the bit accuracy with this message, while Tree-Ring and WAM directly output a detection score.

The results are shown in Tab. 6.
We observe that WAM overall obtains better performance.
However, the watermarking methods for generative models are still competitive and offer important advantages,
namely, the possibility to open-source the model for Stable Signature,
and the possibility to watermark with virtually no image degradation as well as very strong robustness against valuemetric transformations for Tree-Ring.
Note that the results are not perfectly apple-to-apple since the efficiency of the extractor is different between methods: Tree-Ring requires to inverse the diffusion noise which is considerably heavier, while Stable Signature uses a very small extractor ($\approx 100$k parameters).

Figure: (a) After horizontal hlip and contrast adjustment of $1.5$. The overall mIoU is at $85\%$.
Refer to caption: x16.png

### 11.2 Hyper parameter search for DBSCAN

In Section 5.5, the DBSCAN algorithm is used to decode multiple messages from each image, with parameters set to $\varepsilon=1$ and $\text{min}_{\text{samples}}=1000$.
Under the same experimental setting, Figure 9 displays the results of a hyperparameter search conducted under two scenarios: images that underwent horizontal flipping and a contrast adjustment with a parameter of $1.5$ on the left, and the same transformations followed by JPEG compression at a quality level of 80 on the right.
In the absence of JPEG compression, WAM effectively recovers the correct number of messages with satisfactory bit accuracies.
However, when JPEG compression is applied on top, WAM encounters difficulties.
For both scenarios, smaller values of $\varepsilon$ and $\text{min}_{\text{samples}}$ yielded the most favorable outcomes.
These findings guided the selection of the aforementioned values for the primary evaluation in sec. 5.5.

### 11.3 Detailed robustness results

Table 7 shows the detailed results for all the transformations.
We observe that WAM handles many of them, although the performance decreases as they get stronger (e.g., for the combination).
Figure 10 shows several examples of segmentation masks predicted after LaMa inpainting applied to both COCO and DIV2K datasets, consistent with the evaluation settings described in Sec. 5.3.

**Table 7: Full decoding results on the COCO validation set, used for the aggregated results presented in Tab. 2(a) of Sec 5.3. “Combination” corresponds to JPEG-80, Brightness 1.5 and Crop 50% all on the same image, and was not part of the evaluation of Sec. 5.3. TPR is for a threshold on the detection score $s_{\text{det}}$ such that FPR $=0.04\%$. The bit accuracy is computed as described in Eq. 2.**
|  | HiDDeN | DCTDWT | SSL | FNNS | TrustMark | WAM |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | TPR / Bit acc. | TPR / Bit acc. | TPR / Bit acc. | TPR / Bit acc. | TPR / Bit acc. | TPR / Bit acc. |  |  |  |  |  |  |
| None | 76.0 | 95.5 | 77.1 | 91.4 | 99.9 | 100.0 | 99.6 | 99.9 | 99.8 | 99.9 | 100.0 | 100.0 |
| Crop (20%) | 66.5 | 93.3 | 0.1 | 50.2 | 3.3 | 68.8 | 98.5 | 99.6 | 2.1 | 50.3 | 99.7 | 88.8 |
| Crop (50%) | 71.0 | 94.6 | 0.0 | 50.4 | 7.0 | 73.4 | 99.2 | 99.8 | 1.9 | 60.9 | 99.6 | 95.9 |
| Rot (10%) | 7.1 | 80.8 | 0.0 | 50.7 | 11.2 | 78.1 | 76.0 | 94.9 | 3.0 | 56.4 | 97.7 | 77.0 |
| Perspective (0.1) | 65.7 | 93.6 | 0.0 | 51.8 | 15.6 | 80.0 | 98.0 | 99.6 | 77.8 | 96.7 | 100.0 | 99.8 |
| Perspective (0.5) | 32.3 | 89.1 | 0.1 | 50.0 | 0.8 | 61.3 | 51.6 | 91.3 | 2.6 | 50.8 | 100.0 | 96.0 |
| Horizontal Flip | 0.1 | 54.3 | 0.0 | 50.2 | 32.7 | 86.2 | 0.1 | 56.2 | 99.8 | 99.9 | 100.0 | 100.0 |
| Brightness (1.5) | 85.6 | 96.9 | 16.1 | 60.9 | 90.9 | 97.7 | 99.1 | 99.7 | 83.2 | 97.1 | 100.0 | 100.0 |
| Brightness (2.0) | 83.2 | 96.2 | 0.1 | 49.4 | 67.8 | 91.9 | 97.2 | 99.0 | 58.8 | 92.2 | 100.0 | 99.9 |
| Contrast (1.5) | 74.1 | 95.4 | 20.4 | 63.4 | 92.9 | 98.2 | 98.9 | 99.7 | 77.1 | 96.2 | 100.0 | 100.0 |
| Contrast (2.0) | 67.3 | 94.3 | 0.6 | 49.4 | 66.0 | 92.4 | 97.0 | 99.4 | 52.1 | 90.8 | 100.0 | 99.9 |
| Hue (-0.1) | 9.0 | 77.4 | 29.5 | 66.7 | 98.0 | 99.4 | 96.7 | 99.2 | 99.3 | 99.9 | 100.0 | 100.0 |
| Hue (+0.1) | 19.3 | 84.0 | 59.4 | 81.2 | 97.7 | 99.3 | 98.5 | 99.5 | 99.5 | 99.9 | 100.0 | 100.0 |
| Saturation (1.5) | 80.6 | 96.2 | 21.3 | 61.7 | 99.7 | 99.9 | 99.5 | 99.9 | 99.0 | 99.8 | 100.0 | 100.0 |
| Saturation (2.0) | 81.7 | 96.5 | 0.3 | 47.4 | 98.3 | 99.5 | 99.4 | 99.8 | 96.8 | 99.5 | 100.0 | 100.0 |
| Median filter (3) | 74.6 | 94.9 | 0.9 | 53.2 | 82.8 | 96.5 | 99.4 | 99.9 | 99.7 | 99.9 | 100.0 | 100.0 |
| Median filter (7) | 23.4 | 83.0 | 0.4 | 53.0 | 20.4 | 80.7 | 61.1 | 90.2 | 99.4 | 99.9 | 100.0 | 100.0 |
| Gaussian Blur (3) | 47.1 | 88.0 | 41.7 | 77.0 | 97.2 | 99.1 | 87.3 | 95.5 | 99.8 | 99.9 | 100.0 | 100.0 |
| Gaussian Blur (17) | 0.1 | 51.2 | 0.0 | 49.8 | 3.8 | 69.6 | 0.0 | 50.5 | 98.7 | 99.8 | 100.0 | 99.8 |
| JPEG (50) | 11.2 | 77.3 | 0.0 | 49.8 | 5.1 | 72.5 | 34.5 | 86.9 | 99.1 | 99.8 | 99.9 | 99.0 |
| JPEG (80) | 32.8 | 86.8 | 0.1 | 50.5 | 66.1 | 92.6 | 80.5 | 95.4 | 99.6 | 99.9 | 100.0 | 99.9 |
| Proportion (10%) | 0.8 | 65.9 | 1.1 | 56.9 | 0.6 | 61.8 | 36.3 | 88.2 | 2.0 | 58.6 | 99.9 | 94.2 |
| Collage (10%) | 0.9 | 70.1 | 0.5 | 62.8 | 0.1 | 55.9 | 41.1 | 88.7 | 1.3 | 55.7 | 100.0 | 96.5 |
| Combination | 44.7 | 88.7 | 0.0 | 49.9 | 1.2 | 62.8 | 87.7 | 96.1 | 1.7 | 58.8 | 99.2 | 87.2 |

## 12 Qualitative Examples

We show examples of images from the COCO dataset in Fig. 11 and for DIV2k in Fig. 12.
We observe that the watermark is imperceptible to the human eye, primarily due to the JND map used to mask the watermark in areas where the human visual system is less sensitive.
With enough attention, it is however possible to see it in some images, especially in very white or very dark areas (e.g., the boat in the last row of Fig. 11) and in the fur of the animals (e.g., the wolf and the penguin of Fig. 12).
We hypothesize that it comes from the fact that the JND map only accounts for the cover image where the watermark signal $\delta$ is added, and not for $\delta$ itself.
However, repetitive and regular patterns are perceptible, and, when the bright or textured areas are big enough, this becomes noticeable.
We believe that further work could improve the imperceptibility of the watermark, for instance by using more complex HVS models (Watson, ) or by using a regularization on the watermark signal to remove repetitive or structured patterns.

Figure: Figure 10: Examples of WAM’s detection after inpainting with LaMa (Suvorov et al., ), with the experimental set-up detailed in Sec. 5.3. The first two lines are with images from the validation set of COCO (using the union of the segmentation masks), while the third line is with an image from the DIV2k dataset, with a random mask.
Refer to caption: x18.png

Figure: Figure 11: Qualitative results on the validation set of MS-COCO, at various resolutions and for a $32$-bits message. The difference image is displayed as $10\times\text{abs}(x_{\text{m}}-x)$.
Refer to caption: extracted/5992812/figs/appendix/coco/10_nw.jpg

Figure: Figure 12: Qualitative results on images from DIV2k, at higher resolution and for a $32$-bits message. The difference image is displayed as $10\times\text{abs}(x_{\text{m}}-x)$.
Refer to caption: extracted/5992812/figs/appendix/DIV2k/00_nw.jpg

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: In the paper, the authors talk about generative AI models, but they do not show how strong the model is against these types of AI, like Stable Diffusion, which is popular in the AI art community. With Stable Diffusion, it’s possible that all the hidden messages could disappear when the image is changed. It’s important to show that the model can handle generative AI modifications, like Stable Diffusion. For example, if I use Stable Diffusion-based super-resolution (a common tool for improving image quality) on the image, can the model still detect the watermark? If not, the watermark becomes easy to remove.

Another useful test would be to see if adding a second watermark to an image removes or covers the first one. For instance, after editing an image, if I use WAM again to add my own watermark, can the original watermark message still be found?

Please do these tests to show how strong the model is, or suggest ways to prevent these possible problems.
soundness: 3
strengths: The method has been tested against various manipulations, showcasing its robustness. It reliably detects watermarks even in image splicing and inpainting, including small or detailed areas. The paper is well-structured, making it clear, easy to read, and accessible to the reader.
summary: This paper proposes a new method, Watermark Anything Model (WAM), for adding watermarks to images. The method demonstrates state-of-the-art performance even under various manipulations, such as splicing, geometric transformations, color changes, and inpainting. It consists of an embedder and an extractor capable of localizing the watermark region and decoding distinct messages in different areas of the image. The paper is well-written and easy to understand. Overall, the proposed approach offers a robust solution for image watermarking, with potential applications across diverse digital media fields.
weaknesses: The watermark model has not yet been tested against attacks from generative AI, which is widely used today. The paper would be strengthened if the watermark remains detectable after processing with super-resolution models (whether CNN-based or Stable Diffusion-based). Otherwise, people could easily remove the watermark without much effort. The model demonstrates a 100% true positive rate in many cases; however, most of these manipulations are included in the training augmentation, which may indicate overfitting rather than true robustness. If a manipulation is not included in the training data, can the watermark still be detected? Please also elaborate on the model performance against VAE attack and diffusion attack.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1.	In the two-stage training process, robustness and imperceptibility were trained separately.  Could the authors clarify how they ensure that this separation does not lead to conflict? For instance, robustness training led to visible artifacts, while imperceptibility training weakened robustness. How is this balance maintained effectively? 
2.	Could the authors discuss the trade-off between the bit capacity, robustness, imperceptibility? It is well-known that there is an inherent trade-off among these three dimensions in watermarking. In this paper, bit capacity was sacrificed, yet imperceptibility was not outstanding than the baseline, and robustness comparisons were limited to a specific scenario: localized watermarked regions. 
3.	While WAM used a JNP map to enhance imperceptibility, its imperceptibility was comparable but not outstanding relative to baseline methods. Is there potential for further improvement in imperceptibility without compromising robustness?
4.	Given that the capacity of WAM was limited, its ability to support multiple distinct watermarks is constrained. As the amount of AIGC grows rapidly, do the authors have suggestions for extending WAM to support larger payload?
soundness: 2
strengths: Originality:
WAM aimed to localize the watermarked region within an image, which is a novel perspective in watermarking. On the other hand, WAM was able to handle multiple distinct watermarks within a single image, which was rarely addressed in prior work. Additionally, WAM has a two-stage training process, first stage trained robustness while the second stage trained imperceptibility,  which is different from the traditional DNN-based methods. 

Quality:
The authors provided comprehensive experiments on standard datasets, which strengthened the validity of the results. 

Clarity:
The paper was well-structured and clearly explained. The use of toy examples and tables enhanced the readability and understanding of the paper.

Significance:
 The paper introduced watermark localization and a two-stage training process, which could open up new possibilities in watermarking technology.
summary: This paper introduced a new approach to image watermarking called the Watermark Anything Model (WAM). It allowed identify and localize the watermarked regions rather than just determined if an image is watermarked. 
WAM used a two-stage training process. In the first stage, based on the traditional LDM encoder-decoder architecture, the watermark messages were embedded into every single pixel of the image. Detection and Decoding were then measured by calculating the portion or weighted average of the detected watermarked pixels. The second stage applied the Just-Noticeable-Difference map to make the watermark more imperceptible. 
 This paper introduced a novel approach that allowed the localization of the watermarked regions; it proposed a two-stage training process rather than traditional DNN-based method that balanced between robustness and imperceptibility; It allowed multiple distinct watermarks to be embedded into a single image.
weaknesses: 1.	Limited capacity
WAM’s capacity was limited to 32 bits, which may be insufficient for many real-world applications. On the other hand, the comparison with the baseline method were not entirely fair when they are designed for larger capacity.
2.	Although utilized the JND map to improve watermark imperceptibility, it was only comparable to, but not outstanding among, baseline methods. On the other hand, since WAM used a two-stage training process, first training for robustness and then for imperceptibility, it was unclear how well WAM balanced the trade-off between robustness and imperceptibility when training these aspects separately.
3.	 The paper lacked discussion of the computational complexity, which would be valuable for understanding its efficiency and scalability in real-world applications.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please clarify the inovitive of WAM accoding to the weakness.
soundness: 2
strengths: 1. WAM redefines watermarking as a pixel-level segmentation task, allowing for precise localization and recovery of watermarks within small image regions. This advancement expands watermarking capabilities beyond global, image-wide methods.
2. WAM’s architecture combines an encoder-decoder structure and clustering techniques, making it well-suited for segmentation. The two-stage training, which introduces imperceptibility and multi-message support, adds flexibility and enhances usability in high-resolution and high-precision tasks.
summary: The paper presents a novel framework, WAM, for localized image watermarking that embeds and extracts watermarks from specific regions within an image. WAM’s key innovation is treating watermarking as a segmentation task, which enables the model to identify and decode watermarked regions even when only small parts of an image are modified. The model operates in two phases: initial training for robustness and a fine-tuning stage to ensure imperceptibility and support for multiple watermarks.
weaknesses: 1. Although the paper compares WAM with several baseline watermarking models, including advanced steganographic methods could provide a more comprehensive assessment of WAM's relative strengths and limitations. Additionally, embedding and extracting multiple watermarks in different regions of an image is not a unique concept, as it has been previously introduced in methods such as [a] and [b].
2. The need for pixel-level detection across high-resolution images could make WAM computationally intensive, especially in scenarios requiring real-time analysis. Further evaluation of computational costs would help contextualize its practical limitations.
3. The paper acknowledges WAM’s limited payload capacity (32 bits per region). While suitable for smaller messages, the model may face challenges in applications requiring larger payloads. Future work could explore optimizing the architecture to balance capacity and robustness.
4. The reliance on DBSCAN parameters for multi-watermark clustering introduces an element of complexity. Parameter tuning may be required across different datasets or conditions, potentially complicating deployment.

[a] Pan, Wenwen, Yanling Yin, Xinchao Wang, Yongcheng Jing, and Mingli Song. "Seek-and-hide: adversarial steganography via deep reinforcement learning." IEEE Transactions on Pattern Analysis and Machine Intelligence 44, no. 11 (2021): 7871-7884.
[b] Meng, Ruohan, Qi Cui, Zhili Zhoul, Chengsheng Yuan, and Xingming Sun. "A Novel Steganography Algorithm Based on Instance Segmentation." Computers, Materials & Continua 63, no. 1 (2020).

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: See weaknesses.
soundness: 3
strengths: 1. The paper introduces an innovative approach by transforming the watermarking task into a segmentation task, enabling the embedding of multiple watermarks at different locations in an image and supporting watermarking of small areas.
2. The proposed method shows broad applicability and can handle high-resolution images and images of various sizes.
3. The design motivations are well-justified, and the implementation details are clear. For improving visual quality, the authors opted for the JND approach instead of common "perceptual" or "contradictory" losses. Extensive experiments validate the effectiveness of using JND for watermarking.
summary: This paper addresses the limitation of current image watermarking methods in handling small watermarked areas. The authors redefine the watermarking task as a segmentation problem, decomposing it into localization and detection tasks. They adopt a two-stage training strategy to separately achieve robustness and imperceptibility. By incorporating localization capabilities, the authors enable embedding multiple watermarks within a single image. Extensive experiments demonstrate the broad applicability of their method.
weaknesses: 1. Although the authors discuss different model architectures, the overall parameter size remains significantly larger compared to prior models such as HiDDeN (454.4K), FIN (737.8K) [2], and CIN (36.01M) [3]. While the authors highlight that their encoder (1.1M parameters) is relatively compact, it is still substantially larger compared to earlier models like HiDDeN (188.93K) and FIN (747.80K). This large parameter size may limit the method's applicability in resource-constrained environments.
2. The comparative experiments are not comprehensive. The authors do not consider more advanced watermarking models, such as MBRS, FIN, and CIN, which significantly outperform HiDDeN.
3. It would be helpful for the authors to provide a relationship graph between the extractor’s parameter size and performance, to justify the necessity of the large parameter size for detection and decoding tasks.

Overall, I am optimistic about the paper's method and the problem it addresses. I am particularly intrigued by the authors' segmentation-based approach for embedding multiple watermarks within a single image. My final score will depend on the authors' rebuttal.


[1] Fang H, Qiu Y, Chen K, et al. Flow-based robust watermarking with invertible noise layer for black-box distortions[C]//Proceedings of the AAAI conference on artificial intelligence. 2023, 37(4): 5054-5061.

[2] Ma R, Guo M, Hou Y, et al. Towards blind watermarking: Combining invertible and non-invertible mechanisms[C]//Proceedings of the 30th ACM International Conference on Multimedia. 2022: 1532-1542.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 3
questions: 1. The paper compares EditGuard's work, but does not compare the work that is very similar to this paper [1], please explain the difference between the two;
2. How to guarantee the imperceptibility of the framework, and did not see the loss module in the mature watermarking framework, such as the MSE loss between the embedded image x_m and x, the loss of discriminative networks, etc.
soundness: 3
strengths: The paper introduces the Watermark Anything Model (WAM), a deep learning-based approach for embedding and extracting localized image watermarks, which excels in efficiently embedding and locating watermarks within small areas, demonstrating resilience against image tampering and splicing. It can extract distinct 32-bit messages from multiple regions, each less than 10% of the image area, with minimal bit error rates. Trained at low resolutions and further optimized for perception invisibility and multiple watermarks, WAM competes with state-of-the-art methods in terms of imperceptibility and robustness, offering new possibilities and enhanced practicality for image watermarking technology.
summary: The paper generalizes for multi-source images and achieves image locality preservation without a priori knowledge. The most innovative is to define watermarking as a segmentation task. Other contributions have been proposed in state-of-the-art papers, not for the first time in this thesis, and are not innovative enough.
weaknesses: 1. Insufficient innovation, little change in the existing image watermarking framework, no analysis for specific modules, still using the traditional training architecture, just increase the template after random clipping, to achieve the application of the scene changes. 
2.Many existing work has been involved in local image watermarking, known as multi-source image watermarking [1]. The arbitrary image watermarking framework proposed in this paper though does not require prior knowledge and can be protected against any image. But only the major part of the image in a real scene has a role to play and there is no need to add watermarking information for any pixel.

[1] Wang G, Ma Z, Liu C, et al. MuST: Robust Image Watermarking for Multi-Source Tracing[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(6): 5364-5371.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Please refer to the weaknesses
soundness: 3
strengths: 1. The proposed method is able to embed multiple watermarks/32-bit messages
2. The method can output a detection/segmentation map that identifies the regions containing watermarks and those that do not
summary: This work proposes a deep-learning-based image watermarking method, dubbed WAM. The encoder of WAM can watermark images with multiple watermarks/32-bit messages and the decoder of WAM can output a detection/segmentation map that identifies the regions containing watermarks and those that do not (called detection) and also a message map that presents the message in each watermarked area (called decoding).
weaknesses: 1. I am unclear on how the model achieves multiple watermark encodings. If I understand correctly, according to Figure 2, the model does not accept a mask input during the inference stage. In that case, how does it inject different messages into different regions? This makes Figure 1 somewhat misleading, as the first two steps suggest that the message can be precisely controlled to be encoded into specific objects. Could the authors provide further clarification on this point?

2. Could you clarify how the mask for the watermark's position is inputted into WAM in Figure 7 of the Appendix? Additionally, could you explain why this specific position was chosen for the localization experiment?

3. In lines 226-232, if I am not mistaken, this method appears to be identical to the method proposed by Bui et al., rather than merely 'a similar approach'. Please provide proper credit.

4. Regarding the claim in lines 304-305, it is unclear why applying JND would make training easier compared to using perceptual loss. Could the authors provide evidence or reasoning to support this claim?   'Applying JND only in the second training phase makes training easier than previous methods relying on “perceptual” or “contradictory” losses.'

5. I believe the reason WAM outperforms the baselines in terms of bit accuracy for splicing is simply because WAM incorporates splicing during training, whereas the other models do not. If the other models were to apply splicing as an augmentation or attack during training, would WAM still outperform them in terms of bit accuracy?

6. Typically, the TPR is reported at a fixed FPR, as seen in Tree Ring. Is the approach in this paper, reporting TPR at varying FPRs, fair and appropriate?

7. I am not entirely convinced of the necessity of embedding multiple messages separately into a single image. If the embedding capacity is large enough (e.g., 100 bits), users could already encode several different messages within a single bit stream. Could the authors provide some practical scenarios where embedding multiple separate messages would be particularly beneficial?

8. What happens if the position masks of multiple watermarks overlap? Additionally, for the watermark to be successfully extracted, what is the minimum proportion the watermark mask should cover relative to the overall image size? How many watermarks can be embedded in a single image at most?

9. A small typo on line 407, 'hide messages or arbitrary length' should be 'hide messages of arbitrary length'?

10. Will the source code and pretrained model be released for the community?

11. Is WAM robust to the new type of regeneration attacks [1], like diffusion-based regeneration?

---
[1] Invisible Image Watermarks Are Provably Removable Using Generative AI

---

I hope I have understood the method correctly. If there are any misunderstandings leading to wrong judgment, I apologize to the authors. I would be open to adjusting my score if the authors address my concerns during the rebuttal.

Meta-review: This paper works on watermarking. Authors proposed a deep-learning model for localized image watermarking, WAM, which could extract different watermarks from different regions. 

This paper was reviewed by 6 reviewers and got mixed scores as three 8, two 5, one 3. 

Strength and weaknesses given by reviewers before rebuttals are as follows (notes that different reviewers has different perspectives of the paper, so conflicts in strength and weaknesses might happen):

The strength of the papers are: 1) paper is well written; 2) proposed method is novel; 3) results are comprehensive.

Weaknesses are: 1) model is not tested against attack from generative AI and generalization capability is not tested; 2) capacity is limited to 32 bits. some comparison is not fair; 3) lacks discussion of computational complexity; 4) embedding and extracting multiple watermarks in different regions was explored before; 5) need more experiments; 6) novelty is limited. 

During rebuttal, Reviewer p14G (rating 8) suggested that their concerns are well addressed by authors. 

Reviewer mbVM (rating 5) suggested that their concerns on capacity are not addressed well. but also mentioned that "And I am leaning towards rejection unless there is a champion for acceptance.".

Reviewer n1mi (rating 5) didn't give comments in the rebuttal discussion.

Reviewer oRgo (rating 8) said they are positive about this paper and maintained the score. But suggested the experimental setup is incomplete and some of the conclusions are not solid. 

Reviewer pZCp (rating 8) gave positive views about this paper. but also suggested several ways to improve the paper.

Reviewer UTj6 (rating 3) thought authors addressed their concerns and would like to raise the score. But at the end mentioned "The rebuttal period has been extended, but the authors have not updated the PDF as they promised." AC believed there was some miscommunication here. 

Given all these discussions, AC found in general reviewers were positive about this paper and didn't have strong opinions to reject this paper. So AC decided to accept this paper.

Decision: Accept (Poster)