Can Simple Averaging Defeat Modern Watermarks?
Authors: Pei Yang
Hai Ci
Yiren Song
Mike Zheng Shou
Venue: NeurIPS 2024 poster (2024)
Keywords: Watermark
Security
Primary area: safety_in_machine_learning
TL;DR: Many content-agnostic digital watermarking techniques are susceptible to simple steganalysis-based watermark removal.

Abstract: Digital watermarking techniques are crucial for copyright protection and source identification of images, especially in the era of generative AI models. However, many existing watermarking methods, particularly content-agnostic approaches that embed fixed patterns regardless of image content, are vulnerable to steganalysis attacks that can extract and remove the watermark with minimal perceptual distortion. In this work, we categorise watermarking algorithms into content-adaptive and content-agnostic ones, and demonstrate how averaging a collection of watermarked images could reveal the underlying watermark pattern. We then leverage this extracted pattern for effective watermark removal under both greybox and blackbox settings, even when the collection of images contains multiple watermark patterns. For some algorithms like Tree-Ring watermarks, the extracted pattern can also forge convincing watermarks on clean images. Our quantitative and qualitative evaluations across twelve watermarking methods highlight the threat posed by steganalysis to content-agnostic watermarks and the importance of designing watermarking techniques resilient to such analytical attacks. We propose security guidelines calling for using content-adaptive watermarking strategies and performing security evaluation against steganalysis. We also suggest multi-key assignments as potential mitigations against steganalysis vulnerabilities. Github page: \url{https://github.com/showlab/watermark-steganalysis}.

Can Simple Averaging Defeat Modern Watermarks?
Pei Yang" Hai Ci Yiren Song Mike Zheng Shou
Show Lab, National University of Singapore yangpei Cu nus edu, cihai com y yirenênus edu. sg, mike. mike.zheng shou@gmail......... com
Abstract
Digital watermarking techniques are crucial for copyright protection and source identification of images, especially in the era of generative Al models. However, many existing watermarking methods, particularly content-agnostic approaches that embed fixed patterns regardless of image content, are vulnerable to steganalysis attacks that can extract and remove the watermark with minimal perceptual distor- tion. In this work, we categorise watermarking algorithms into content-adaptic and content-agnostic ones, and demonstrate how averaging a collection of water- marked images could reveal the underlying watermark pattern. We then leverage this extracted pattern for effective watermark removal under both greybox and blackbox settings, even when the collection of images contains multiple watermark patterns. For some algorithms like Tree-Ring watermarks, the extracted pattern also forge convincing watermarks clean images. Our quantitative and qualitative evaluations across twelve watermarking methods highlight the threat posed by steganalysis content-agnostic watermarks and the importance of de- signing watermarking techniques resilient to such analytical attacks. We propose security guidelines calling for using content-adaptive watermarking strategies and performing security evaluation against steganalysis. We also suggest multi-key assignments as potential mitigations against steganalysis vulnerabilities. Github http:///ww........ooo.//o//aa/aa
Introduction
Digital watermarking hides information within digital media, facilitating copyright protection and source authentication (H3]. With the advances in Al-based image generation and editing (477). robust and secure digital watermarking is crucial for preventing deepfake misuse or manipulations of created contents 8 9).
We categorise digital watermarking methods into two types; content-adaptie and content- aggostt.. Content- addatie methods take images into the watermarking process, dynamically adjusting the watermark's is placement and strength based on the image content, as seen in technologies like HiD- DeN 100 and RivaGAN []] Content-agnostii methods, however, use fixed, predefined watermark patterns independent of or weakly dependent image content. Apart from traditional methods like additive perturba- tions, and Tree-Ring [14] that places ring pattern the initial noise of diffusion generation process. adatevee methods typically offer better robustness against image processing distortions, while content-agnosti methods are computationally lighter and easier to implement.
DwtDetSvd [12). this also includes RoSteALS [33 that adds
fundamental requirement for digital watermarks is robustness, ensuring watermarks cannot be easily removed or tampered with (2] To meet the requirement, existing methods have been improving
"Equal contribution.
Cooresponniigg author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

watermark robustness through design considerations [1] or data augmentation during training [10]. and have demonstrated strong robustness various image distortions like noise perturbations or JPEG compression 15
14 16 Works like Tree-Ring [14 even demonstrated robustness against strong attacks like VAE compression and image re-generation [17 18]. In this paper, however, we reveal that content-agnostic watermarking techniques, including Tree-Ring, are vulnerable to steganalysis attacks, unmasking their hidden fragility.
To the best of our knowledge, our steganalysis is the first successful blackbox attack against Tree-Ring watermarks [14. We discovered content- agnossic ripple pattern in Tree-Ring- watg-waarmrreed images and identified that this component is essential for watermark detection. Subtracting this pattern allows evading watermark detection with minimal impact on image perceptual quality. This raises the question: Do diffusion model watermarking methods that modify initial noise [19] (20 [4] [6]
truly add semantic watermarks, or do they merely propagate low-level content-agnostic patterns to generated images?
Lastly, we propose new security guidelines for the watermarking community, emphasizing the importance of robustness against steganalysis. The guidelines call for performing evaluations against steganalysis when proposing new watermarking methods. It also encourages the development of content-adaptive watermarking methods to enhance resistance to steganalysis. For existing content- agnostic watermarking methods, we suggest assigning multiple watermarks per user as mitigation strategy. In summary, the main contributions of this paper are:
We reveal the vulnerability of content-agnostic watermarking methods to steganalysis removal and forgery.
To the best of our knowledge, we are the first to successfully attack Tree-Ring water- marks in a blackbox setting, providing deeper insights into the essence of diffusion noise watermarking methods.
We propose new security guidelines for future watermarking methods to help defend against steganalysis attacks.
2 Related works
2.1 Digital image watermarking
The field of digital image watermarking has evolved from traditional rule-based approaches more recent deep cerning-basee techniques, with a significant focus on watermarking diffusion-generatee images [4 21 222 We categorise digital watermarking technologies into content-agnostic, which craft modifications based solely on the watermark information, and content-adaptive, which tailor modifications based on both watermark information and the image content.
Content-agnostic watermarking Traditional methods like DwtDetSvv 12 employ fixed wa- termark patterns in transform domains, while more recent approaches modify the initial noise for diffusion- image generation. Tree-Ring watermarks [14] replace the low-frequency Fourier- domain pixels of Gaussian noise with ring pattern before using it for diffusion denoising. Similarly, Gaussian Shading [19] preserves the distribution while sampling the initial noise. Other approaches train encoders generate additive watermark perturbations without conditioning on image features.
13
23
24
Content-adaptive watermarking These techniques leverage image features generate watermarks tailored to the input image content. Early encoder-decoder methods like HiDDeN [10] and StegaStamp [15] employed deep neural networks to imprint watermarks onto the images. SSL (25 leveraged self- supervised networks as feature extractors, while RivaGAN [1] used attention mechanisms to look for appropriate local regions for watermark encoding. Recent approaches like Stable Signature 22] WADiff (2I), and Zhao et al. 26 finetune the diffusion model to enable content-aware watermarking of diffusion-generatee images. WMAdapter (27 designs dedicated contextual adapter.
Through the classification, we highlight the vulnerability of content-agnostic techniques to steganaly- sis attacks, they employ fixed or weakly content-dependent watermark patterns.

---

2.2 Attacks on watermarking
Traditional attacks applied distortions disrupt watermarks, performing signal-level distortions like image compression, noise perturbation, blurring colour adjustment, and geometric transformations like rotation or cropping (0) T0 12). In terms of videos, codecs may also distort invisible watermarks [i 32
These attacks fool watermark detectors at the cost of significant image quality degradation. To resist such attacks, training-bass based methods have been simulating the distortions via
"attack layers" during training [10 35, while training- frreingg-eeeeeeeeo methods have been employing design considerations such as watermarking only wwwfrequency components [14].
Recent attacks base on deep models: regeneration attacks with diffusion models 36 and VAEs can provably remove pixel-level invisible watermarks [37 [8]. But such attacks are shown to be ineffective [17] for Tree-Ring [14] that alters the image lot. When attackers can access the watermarking algorithm, they may also perform adversarial attacks 38). The downside is that both regeneration and adversarial attacks are computationally expensive. In contrast, we propose new type of blackbox steganalysis attack, which efficient and works for content-agnostic watermarks. Steganalysis can extract meaningful watermark patterns, thus promoting further applications like forgery or explainability.
Watermark steganalysis
3.1 Notations
Let denote the original digital image, w the watermark information (e.g., bit sequence or geometric pattern), and the watermark encoder that imprints w into Zs, yielding the watermarked image Tw E(re, w). The embedding constraint ensures that Te and zw are perceptually indistinguishabl.. A watermark decoder recovers the embedded information v D(x..) ) for authentication purposes.
3.2 Threat model
The adversary aims fool D by manipulating Tw using a strategy denoted T(+) such that D(T(xw)) + w (watermark removal) or manipulating ig such that ((T(x))) = w (watermark forgery). Formally, the adversary solves:
Watermark Removal:
max D(T() - w|l,
T
(I)
Watermark Forgery:
min D(T(ze)) wil.
T
subject to the constraint that the original image and the manipulated image T'x) are perceptually indistinguishable. Rather than applying strong distortions as T, we demonstrate that the adversary can take steganalysis approach to fool D.
3.3 Steganalysis: watermark extraction, removal and forgery
Figure iillutraate our watermark removal/forgery strategy T', which assumes that E perturbs an additive pattern & agnostic image content, such that T., Ze dw. This assumption can be refined based on a detailed understanding of specific watermarking algorithms (as will be showcased in Section 4.4.1p. Under this additive assumption, to either remove or forge watermarks, we can approximate 8n Tw Te. To improve approximation and reduce randomness, we propose averaging over n images during pattern extraction:
1 n
n
8
Fengi
F0,i
With the approximated 8... the adversary can perform greybox watermark removal (&s T(xw) Ja d.w) or forgery (f.,. T(*Ø) = De + 8.)) on given image x. Even without paired Ta. the adversary can perform blackbox removal/forgery by approximating z through averaging any

---

Greybox Pattern Extraction
Watermark Removal
Watermarked
Paired clean
Extraccted 8,r
W Watermarked
Extracted 6y
Removed fg
Blackbox Pattern Extraction
Watermark Forgery
Σ
Watermarked Xw
Unpaired clean Ao
Extracted 6.r
Unpaired clean X
Extracted By
Forged &wr
Figure 1: Watermark pattern extraction, removal and forgery under the Simple Linear Assumption Two groups paired (greybox) or unpaired (blackbox) images are first averaged and then subtracted to reveal the watermark pattern. The pattern extracted is then used for watermark removal/forgery.
collection of clean images from the Internet. There is practical scenario where the adversary's watermarked image collection contains multiple different watermarks, we show in Section 4.4.4thha they can still use Equation for pattern extraction.
4 Experiments
4.1 Experimental setup
Image experiments
We evaluate our proposed steganalysis on 12 existing image watermarking
methods: Tree-Ring 14), Ringll> [16]. RAWatermark (23 DwtDetSvd [12. RoSteALS [3]]
Gaussian Shading 19), Stable Signature [22]. WmAdapter 27), RivaGAN [I]] SSL 225 HiDDeN
[0], and DwtDet 12]. For the greybox setting, we use the COCO2017
39 validation set for
WmAdapter 22 and Stable Signature 222. Stable Diffusion Prompts 40 for Tree- Ring 14 and
RingID 16 prompts, and DiffusionDB 41 for the remaining methods as the non-watermarked images (Iz) The corresponding watermarked images (Iw) generated using the respective watermarking methods. In the blackbox setting with no access to paired images, we substitute z with ImageNet [42] test set. The selection of images within the datasets is random. The datasets resized to x56x 256 for RoSteALS, S, SSL,, and HiDDeN, and 512x512 for other methods.
We assess watermark removal under different n (number of images averaged) during watermark pattern extraction, and test on 100 imagesPduring during watermark removal. We report detection AUC for Tree-Ring [4] and RAWatermark [23]. and watermark decoding bit accuracy for the other methods. Additionally, we evaluate the image quality between Fwr and its non-watermarked counterpart, reporting PSNR in the main text and SSIM, LPIPS 43 and SIFID 44 in the appendix.
Audio experiments We then extend the experiments to audio watermark removal on AudioSeal 445 and WavMark (46), using the zh-CN subset of the Common Voice dataset [47. Each audio segment is preprocessed to 16 kHz mono format, with only the first two seconds retained. We use paired audio for greybox removal, and unpaired audio for blackbox removal. We report the watermark detection accuracy for AudioScal [45), and watermark decoding bit accuracy for WayMark [46]. To quantify the audio quality after watermark removal, calculate the Ccale-Invariaa Signal-to-Noise Ratio (SI-SNR) between the senermark--emovve audio and its non wotermarrarreed counterpart.
Computing resources The experiments were conducted on AMD EPYC 7413 24-Core Processor and an Nvidia RTX 3090 GPU, requiring around 200GB of disk space. The execution time for each experiment ranges from around 10 minutes (HiDDeN) around 10 hours (Tree-Ring).
'Each configuration is tested on 100 images or audio segments to minimise computational cost during repeated ablation studies watermark removal and forgery.

---

4.2 Quantitative analysis on watermark removal
As shown in Figure 2 (left column), our steganalysis- watermark removal method effectively degrades the detection performance of RAWatermark (0.5744 AUC), DwtDetSvd (0.5722 accuracy), Tree-Ring (0.2407 AUC), RoSteALS (0.2444 bit accuracy), and Gaussian Shading (0.5615 bit accuracy). The results highlight two key findings: (1) the aforementioned methods embed content- agnostic watermarks, and (2) content-agnostic watermarks are susceptible to steganalysis- removal.
The effectiveness of our method increases as n decreases, albeit at the cost of increased image distortion (smaller PSNR). In contrast, content-adaptin watermarking methods (Figure 2 right column) demonstrate robust resistance this attack, maintaining high detection accuracy (> 0.95) upon convergence. This resilience underscores the importance of content- content-adaptivity adaptivity in watermark design to thwart steganalysis-based removal attacks.
4.3 Qualitative analysis
In this qualitative analysis, we first examine the patterns extracted from various watermarking methods, then discuss how the removal of these watermarks affects image quality.
Extracted patterns Figure 9 displays patterns extracted from content-agnostic methods, while T shoshwws those from content-adaptive methods. Content-agnostic watermarks tend to exhibit unique patterns, like fingerprints. For example, RingID [66 patterns are concentric rings with bright spots in the centre, DwtDetSvd [12] patterns resemble vertical lines like barcodes, and RoSteALS [3] patterns appear as grid-like patches with non-uniform illumination. In contrast, patterns extracted from content-adaptive watermarks are less discernible. Notably, under the greybox setting, the eextteN-extracted (T) pattern converges to zero, indicating completely no discernible pattern. Furthermore, patterns extracted in the greybox setting contain fewer visual artifacts than those in the blackbox setting. As more images are averaged, the extracted watermark pattern becomes clearer and more precise, with fewer residual artefacts from image contents. A more detailed analysis is presented in Appendix A.3
Visual quality degradation Figures 12 and 13 show the visual impact of removing aggnont- and content-adaptive watermarks, respectively. For all methods but Gaussian Shading [19], under the greybox setting, when more than 50 images are averaged, virtually no visual artefacts remain after watermark removal. In the blackbox setting, averaging over 100 images is necessary to eliminate most artifacts. The exception is Gaussian Shading [99, which consistently produces visible artefacts because the pattern extracted has large magnitude. Subtracting this pattern significantly distorts the image. Appendix A.44 provides a more detailed analysis.
4.4
Case study: Tree-Ring watermarks
To further reveal how steganalysis be threat to content-agnostec watermarking algorithms, conduct case study on Tree-Ring watermarks [4]. Tree-Ring is sophisticated Liffusion-basee watermarking algorithm that injects a frequency-domaii ring pattern into Gaussian noise signal before using this watermarked noise for diffusion-denoisinn image generation. During detection, performs DDIM inversion recover the injected ring pattern from the initial noise and compares it to reference pattern. In the following experiments, we demonstrate that with minimal modifications, we can both remove and forge Tree-Ring watermarks under different scenarios.
4.4.1 Low-level content-agnostic pattern in Tree-Ring
This section focuses revealing the low-level content-agnostic component of Tree-Ring watermarks 14]. First, we curate a specific steganalysis for Tree-Ring's detection algorithm, demonstrating how tailored steganalysis more accurately extracts watermark patterns than generic averaging. We then compare the extracted watermarks with the ground truth to showcase this low-level component.
Our steganalysis incorporates the DDIM inversion steps from Tree-Ring's detection process. By inverting watermarked images to the DDIM-inverted latent space and averaging them, we obtain patterns (Figure 3 second row). As more images averaged, these patterns closely resemble those
5

---

Tree- Tree-Ring
Ring
Stable Signature
20
13 35
0.8
0.96
20
PSNR
RingID
W mA dapter
14
0.8
12.5
0.8
12
0.2
RAWa Watermarkk
RivaGAN
10
DwtDctSvd
SSL
30
28
0.2
10
0.2
5
20
100
300
25
NR
10
20
50
100
200
500
1k
3k.
3k
RoSteALS
HiDDeN
8 18 81 86
0.6
0.4
0.4
t:
0.2
0.2
8
10
20
100
500
24
NR.
10
20
50
100
200
500
1k
Gaussian Shading
DwtDct
10
0.6
R.6
28
0.4
9.6
0.2
9.2
0.98
NR
10
20
50
100
200
300
12
20
100
200
500
18
22
3k
All horizontal axes: n (owmber images averaged during pattern extraction) Performance Metric (Greybox) d PSNR (Blackbox)
Performance Metric (Blackbux)
da PSNR (Greybux)
Figure Performance of watermark detectors under steganalysis-basee removal. Performance metrics include AUC (watermark verification AUC), Bit Acc (bit accuracy, percentage of correctly decoded bits), and Det Ace (detection accuracy, accuracy of fully recovered watermark). The plots also illustrate the corresponding PSNR measure of image quality degradation. The left/right columns show content-ggnosiicccomee--ddd
methods, respectively. NR denotes the case without removal, reflecting the decoder's inherent performance.

---

Number of Watermarked Images Averaged
Ground Truth
10
50
100
500
5000
Fier Spectium
Avg Lnverted Latents
Spaiial
Gryybox
Avgrot Images
BHladkbbx
Figure Visualisation of Tree-Ring-extractee watermark patterns. Top: Pattern extracted from DDIM-inverted latents without subtracting The first and second rows are Fourier transform pairs. Bottom: Pattern extracted in image space under greybox and blackbox settings, akin to Figure 9
Steganalysis
Noise
Blur
Brightness
1.0
1.0
1.0
1.0
AUC
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
15
10
5
0.50
0.25
0.5
1.0
10 SIFID
PSNR (dB)
SSIM
LPIPS
Figure 4: Tree-Ring detection AUC against quality metrics for steganalysis-based removals (n 5000) and image distortions. Steganalysis-based removals (blue) cluster bottom-left, indicating effective watermark removal with comparatively low quality degradation.
extracted under greybox or blackbox settings, manifesting as ripples spreading from the corners and forming aliasing patterns in the centre, reminiscent of superpositioned 2D sinc functions.
In the Fourier domain (Figure 3 first row), these patterns display a clear ring structure nearly identical to the ground truth. The high similarity between the ground truth and the patterns extracted from both the image and EDDIM-inverted latent domains indicates that Tree-Ring likely propagates content-agnostic ripple pattern throughout the image generation process, slightly but directly revealing the generated images. This insight enables us fool Rine-Ring's 's watermark detector by simply subtracting this ripple pattern, effectively removing the watermark information.
4.4.2 Comparison with distortion-based removal techniques
To compare perceptual quality degradation between our method and distortion- based ones, in Figure we plot Tree- Free-Ring's AUC versus qualitative metrics varying signal strengths during watermark subtraction. In all four plots, the steganalysis-bant watermark removal curves clustered in bottom-left corner, indicating that effective steganalysis can remove watermarks with significantly less image quality degradation compared to distortion-based methods. Note that although Section the excess performance degradation under small n is believed be caused by excess distortions
7

---

Table 1: Tree-Ring 14
detection accuracy at 1% FPR for watermark removal and forgery. NRmv
represents "no removal".
# Imgs Avged
NRmv
20
100
200
500
1000
2000
5000
Removal 1.00
0.08
0.09
0.13
0.13
0.13
0.14
0.14
0.14
0.14
Forgery
1.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Forgery Removal
5 images
10 images
50 images
100 images
500 images 5000 images
Count Count
25
25
50
100
50
100
100
50
100
50
100
100
I--Distance to Reference
Tree-Ring Ring watermarked images
Clean images
Tree-Ring TPR@1 9FPR threshold
Tampered images (watermarked images after removal/clean images after forgery)
Figure 5: Histograms distance to reference watermarking pattern for Tree-Ring watermark removal (top) and forgery (bottom). For removal, averaging more images pushes the watermark-removed images (green) away from the true watermarked images (orange). For forgery, oppositely, this increases the similarity of forged images (green) to true watermarked images (orange). Red dashed lines are thresholds T at 1% FPR.
introduced due to imperfect pattern extraction, in this section, we demonstrate that distortions generally do not help remove watermarks. Appendix visualise images under these distortions.
4.4.3 Watermark forgery
We demonstrate the ability to forge Tree-Ring watermarks (f Zg ).w) non-watermarked images, in addition watermark removal (itg Ew - 8..). Table shows the forged watermarks completely deceive Tree-Ring's detection. 5 forged watermarks exhibit slightly larger distances compared to authentic watermarked images. However, when large (500 images), the forged images overlap with true watermarked images in the histogram, precluding threshold- baased separation, thereby demonstrating Sice-Ring's vulnerability steganalysis-baa watermark forgery.
4.4.4 Effectiveness of removal under multiple watermarks
We study heterogeneous scenario where the adversary's 's image collection contains multiple different watermark patterns. When there are more patterns in the adversary's image collection, the detection AUC rises while the PSNR drops, indicating decreased steganalysis removal efficacy. Mixing three different watermark patterns increases Tree-Ring's 's detection AUC from below 0.2 to above 0.7 in both greybox and blackbox settings with n 5000, indicating that mixing watermarking keys could
Blackbox
Greybox
16
16
0.8
15
0.8
15
0.6
14
14
0.4
0.4
0.2
13
0.2
13
12
12
2
3
4
5
2
4
3
4
5
6
4
5
6
Horizontal Axis: Number of Heterogeneous Tree-Ring Ring Keys Mixed in the Adversary's Collection
# Tmgs Avged During Extraction: No Removal
-
-0110
-0
-0
++100
-0200
5000
++1000
-02000
5000
Figure 6: Ablation study on watermark removal performance with Tree-Ring when the adver- sary's image collection contains multiple watermark patterns.

---

A udioSeal
WavMark
60
0.8
30
22 SASS ((R)
0.8
40
SI-SNIGG
0.6
20
0.6
20
0.4
10
0.4
0.2
-10
-20
NR
5
10
20 50
100
200
500 1k
2k
5k
NR
10
20
50
100
200
500
1k 2k 5k
All horizontal axes: n (number audio segments averaged during pattern extraction) Performance Metric (Blackbox) Performance Metric (Greybox) -á- SI-SNR (Blackbox) -a- SI-SNR (Greybox)
Figure Impact of steganalysis-barg based removal on audio watermark detection. The plot shows watermark detection accuracy (Det Acc) for AudioSeal 455 and bit accuracy (Bit Acc) for WavMark
46
SI-SNR values indicate audio quality changes post-removal. NR represents the baseline
performance without any removal attempts.
improve security against simple steganalysis- basedd watermark removals. Nevertheless, the remaining 0.3 gap to perfectness still demonstrates the vulnerability of content-agnostic watermarking. We highlight that assigning multiple watermarks serves as mitigation and cannot fundamentally address the steganalysis vulnerability. (Appendix giveg
gives more cases).
4.4.5 Summary
In this case study, we rooted Tree-Ring's security vulnerabilities in its use low-level content- agnostic ripple patterns watermarks, rather than solely from semantic watermarking. This enables us to successfully fool Tree-Ring watermark detection with minimal impact on perceptual quality. Although it exhibits strong robustness distortions [14] and regeneration attacks 17), through steganalysis ceeennllyss-bbsssee based removal, we are the first to effectively remove Tree-Ring watermarks without access to the algorithm.
4.5
Audio watermark steganalysis
The distinction between content-agnostic and content-adaptive watermarks extends beyond images, applying equally other media like audio. To test the generality of steganalysis approach, extend the experiments to two audio watermarking methods: AudioSeal [45) and WavMark [46].
Following the methodology outlined in Section 3.3 we extract audio watermark patterns by averaging in the time domain. Aigure/iillusraann the efficacy this approach on audio watermarks. Similar to content-agnosti -agnostc image watermarks, our steganalysis-but based removal significantly impairs the performance of AudioSeal, reducing its detection accuracy from a perfect 1.0 around 0.75 in both greybox and blackbox settings. This decline underscores AudioSeals's vulnerability to our simple averaging- steganalysis.
Interestingly. for WavMark, subtracting the averaged pattern counterintuitively improves its bet accuracy from below 0.8 to a perfect 1.0 when n large. While the complexity of WavMark's algorithm precludes definitive conclusions from this experiment, the pattern extracted under large showcased the existence the systematic bias and its correlation with the watermark information. Although our method does not directly "remove" WavMarks' watermark in the traditional sense, the observed behaviour raises questions about its resilience to more sophisticated steganalysis attacks.
In both cases, smaller n values lead to lower watermark detection rates and lower SI-SNR values, demonstrating coarsely extracted patterns further degrading detection performance with additional audio quality distortions. This mirrors our finding in the image domain, highlighting the importance of sufficient large for accurate watermark pattern extraction across different media types.
Guidelines towards steganalysis-secure watermarking
Our analysis demonstrates that content-agnostic watermarking methods like Tree-Ring 14 are
vulnerable to steganalysis-based attacks. Although these methods claim robustness by demonstrating strong resistance against distortions (c.g.. blurring or noise perturbations), adversaries may still

---

remove the watermark through steganalysis. thus compromising their robustness. Our experiments reveal that even complex, highly nonlinear methods based on deep neural networks are susceptible to steganalysisbbasel watermark removal.
To avoid such threat, future watermarking algorithms shall content-adaptive watermarking methods. One approach is to incorporate image features while encoding watermark information. For example, HiDDeN [T] and RivaGAN [1I introduce image features into the watermark encoder through concatenation and attention, respectively. For existing content-agnostic methods, assigning multiple watermarks per user can mitigate but not fundamentally solve steganalysis threat.
While we performed simple averaging steganalysis on RGB images and 16 kHz monophonic audios, watermarking methods should resist more complex techniques, such as steganalysis in various colour spaces or different transform domains. Evaluating security against diverse steganalysis models, analogous robustness tests against distortions, is crucial for developing secure watermarking algorithms. These two aspects form our security guidelines:
1. Ensure watermarks are content-adaptive to resist steganalysis attacks.
2. Evaluate watermark against steganalysis to ensure robustness.
6
Conclusions
This work has revealed the vulnerability of content-agnostic watermarking algorithms to steganalysis attacks. We have demonstrated effective watermark removal and forgery techniques under both greybox and blackbox settings across twelve watermarking methods, including recent deep-learning approaches. Our findings extend to audio watermarking methods as well. To address these threats, we propose security guidelines that encourage exploring content-adaptive watermarking methods and evaluating them against steganalysis attacks. We have also proposed temporary mitigations for existing content-agnostic methods. Only by addressing the vulnerability to steganalysis can we develop secure and robust digital watermarking systems capable of safeguarding the integrity of digital content in the era of generative AI.
Limitations and ethical considerations effective against content-agnostic watermarking, not content-adaptive techniques. Responsible devel- opment and deployment of steganalysis technologies are crucial, adhering to fairness, accountability, and transparency principles to prevent misuse for unwarranted surveillance or privacy violations.
Steganalysis-based watermark removal/forgery is only
Broader impacts The proposed steganalysis attack and security guidelines extend beyond image watermarking. They apply to watermarking other media like video [48), audio [49], 3D models [50]- 52). and to other domains. Our proposed guidelines and mitigation strategies strengthen watermarking security, contributing to safer digital environment.
7
Acknowledgements
This research is supported by the Ministry Education, Singapore, under the Academic Research Fund Tier 1 (FY2022) Award 22-5406-A0001.
References
[1] R. C. Gonzalez and R. E. Woods, Digital image processing, 3rd ed. Upper Saddle River. :.r:lo: Pearson/Prentice Hall, 2008
[2] M. Asikuzzaman and M. R. Pickering, "An overview of digital video watermarking." IEEE Transactions Circuits and Systems for Video Technology, vol. 28, 9, pp. 2131-2153, 2018.
[3] Cox, M. Miller, Bloom, and C. Honsinger, "Digital watermarking." Journal of Electronic Imaging. vol. 11, no. 3, pp. 414 414, 2002.
[4] J. Ho, A. Jain, and P. Abbeel, "Denoising diffusion probabilistic models." Advances in neural information processing systems, vol. 33, pp. 6840 68016 2020.
10

---

[5] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, "Score-based generative modeling through stochastic differential equations." arXiv preprint vrXiv:2011.13455 2020.
[6] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, "High-resolution image synthesis with latent diffusion models," in Proceedings of the IEEE/CVF Conference on Compuier Vision and Pattern Recognition (CVPR), June 2022. pp. 10 006684-01 1965.
[7] C. Meng. Y. He, Y. Song. J. Song, J. Wu, J.-Y. Zhu, and S. Ermon, "SDEdit: Guided image synthesis and editing with stochastic differential equations," " International Conference on Learning Representations, 2022.
[8] Y. Mirsky and W. Lee, "The creation and detection of deepfakes: A survey"" ACM computing surveys (CSUR), vol. 54, no. 1. pp. 1-41, 2021.
[9] L. Verdoliva, "Media forensics and deepfakes: an overview." IEEE Journal of Selected Topics in Signal Processing, vol. 14, no. 5. pp. 910-932, 2020.
[10] Zhu, R. Kaplan, Johnson, and Fei-Fei, "Hidden: Hiding data with deep networks." in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 657-672.
[11] K. A. Zhang, I.. Xu, A. Cuesta--nfannte and Vceramachaneni, "Robust invisible video watermarking with attention," " arXiv preprint arXiv:1909.0123 2019.
[12] Cox, M. Miller, J. Bloom, J. Fridrich, and Kalker, Digital Watermarking and Steganography, 2nd ed. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 2007.
[13] T. Bui, S. Agarwal, N. Yu, and J. Collomosse, "Rostcals: Robust steganography using autoencoder latent space," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 933-942.
[14] Y. Wen, J. Kirchenhauer, J. Geiping, and T. Goldstein, "Tree-ring watermarks: Fingerprints for diffusion images that are invisible and robust," arXiv preprint 2:X.v:2305.200300 2023.
[15] M. Tancik, B. Mildenhall, and R. Ng, "Stegastamp:: Invisible hyperlinks in physical photographs," in Proceedings of the IEEEJCVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[16] H. Ci, P. Yang, Y. Song, and M. Z. Shou, "Ringid: Rethinking tree-ring watermarking for enhaneed multi-key identification," " 2024.
[17] B. An, M. Ding, T. Rabhani, A. Agrawal, Y. Xu, C. Deng, S. Zhu, A. Mohamed, Y. Wen, T. Goldstein et al., "Benchmarking the robustness of image watermarks." " arXiv preprint arXiv:2401.00 105573, 2024.
[18] W. Nie, B. Guo, Y. Huang, C. Xiao, A. Vahdat, and A. Anandkumar, "Diffusion models for adversarial purification," arXiv preprint X2x::2005.00766 2022.
[19] Z. Yang. K. Zeng. K. Chen, H. Fang. W. Zhang, and N. Yu, "Gaussian shading: Provable performance- lossless image watermarking for diffusion models," in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024.
[20] L. Lei, Gai, J. Yu, and Zhu, "Diffusetrace: transparent and flexible watermarking scheme for latent diffusion model," arXiv preprint arXiv:2405.02696, 2024.
[21] R. Min, Li, H. Chen, and M. Cheng, "A watermark-conditionee diffusion model for ip protection." arXiv preprint or Xiv:2403. 10893, 2024.
[22] P. Fernandez, G. Couairon, H. Jégou, M. Douze, and T. Furon, "The stable signature: Rooting watermarks in latent diffusion models," in Proceedingy of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 22 466-22477.
[23] X. Xian, G. Wang. X. Bi, J. Srinivasa, A. Kundu, M. Hong, and J. Ding, "Raw: A robust and agile alug-and-play watermark framework for ai ai-generated images with provable guarantees," 2024.
[24] R. Ma. M. Guo, L. Yuming, H. Zhang, C. Ma. Y. Li, X. Xie, and S. Zhang, "Pigw: A plug-in generative watermarking framework," arXiv preprint 2rx:::2403. /2053, 2024.
[25] P. Fermandez, A. Sablayrolles, T. Furon, H. Jégou, and M. Douze, "Watermarking images in supervuu latent spaces," in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp.

---

[26] Y. Zhao, T. Pang. C. Du, X. Yang. N.-M. Cheung, and M. Lin, "A recipe for watermarking diffusion models," arXiv preprint arXiv:2303. 10137, 2023.
[27] H. Ci, Y. Song, P. Yang, J. Xie, and M. Z. Shou, "Wmadapter: Adding watermark control to latent diffusion models," arXiv preprint arXiv:2006008337, 2024.
[28] F. Hartung and M. Kutter, "Multimedia watermarking techniques," Proceedings of the IEEE, vol. 87, no. 7, pp. 1079-1107, 1999.
[29] F. A. Petitcolas, R. J. Anderson, and M. G. Kuhn, "Information hiding-a survey." Proceedings of the IEEE, vol. 87, no. 7. pp. 1062-1078, 1999.
[30] A. A. Ukasha, R. M. Alabid, I. A. Nasir, and F. A. Alwafi, "A robust color image watermarking improvement scheme based image normalization." " in 2021 IEEE 1st International Maghreb Meeting of the Conference on Sciences and Techniques of Automatic Control and Computer Engineering MI-STA, 2021, pp. 356 362.
[31] Celik, J. Talstra, A. Lemma, and S. Katzenbeisser, "Camcorder capture robust low-complexity wa- termarking of mppe-2 bit-streams," in 2007 IEEE International Conference on Image Processing, vol. 5. IEEE, 2007, pp. V-489.
[32] T. Yu-min, Q. Yun-hui et al., "Adaptive video watermarking algorithm based on mpeg-4 streams," in 2008 10th International Conference on Control, Automation, Robotics and Vision. IEEE, 2008, pp. 1084-1088.
[33] M. Ahmadi, A. Norouzi, N. Karimi, S. Samavi, and A. Emami, "Redmark: Framework for residual diffusion watermarking based on deep networks," Expert Systems with Applications, vol. 146, 113157, 2020.
[34] X. Luo, Y. H. Chang, C. Liu, P. Milanfar, and F, Yang, "Dvmark: deep multiscale framework for video watermarking." IEEE Trunsactions Image Processing. 2023.
[35] K. A. Zhang, I.. Xu, A. Cuesta-Infante, and K. Veeramachaneni, "Robust invisible video watermarking with attention," " arXiv preprint arXiv: 1909.01285, 2019.
[36] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, "High-resolution image synthesis with latent diffusion models," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10 6965-
[37] X. Zhao, K. Zhang. Su, S. Vasan, Grishchenko. C. Kruegel, G. Vigna, Y.-.. Wang, and Li, "Invisible image watermarks are provably removable using generative ai," arXiv preprint 2r36v22306.01953 2023.
[38] N. Lukas, A. Diaa, L. Fenaux, and F. Kerschbaum, "Leveraging optimization for adaptive attacks on image watermarks," " arXiv preprint XXX.:23309.19955 2023.
[39] T.-Y. Lin, M. Maire, S. Belongie, Hays, Perona, D. Ramanan, Dollár, and C. L. Zitnick, "Microsoft coco: Common objects in context," in Computer VCCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer, 2014, pp. 740-755.
[40] Gustavosta,
"Sabbe-diffusion-roompp""
Difiee-Difuuso--PPnnee Prompts 2023.
[41] Z. J. Wang, E. Montoya, D. Munechika, H. Yang, B. Hoover, and D. H. Chau, "Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models," arXiv preprint 1.XXx::210014896 2022.
[42] J. Deng, W. Dong, R. Socher, I..J. K. Li, and I.. Fei-Fei, "Imagenet: A large-scale hicrarchical image datahase," in 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248-255.
[43] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, "The unreasonable effectiveness of deep features as perceptual metrie," in CVPR, 2018.
[44] T. Shaham, T. Dekel, and T. Michaeli, "Singan: Learning generative model from a single natural image." in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 4570-4580.
[45] R. S. Roman, P. Fernandez, A. Défossez, T. Furon, T. Tran, and H. Elsahar, "Proactive detection of voice cloning with localized watermarking." 2024.
[46] G. Chen, Y. Wu, S. Liu, T. Liu, X. Du, and F. Wei, "Wavmark: Watermarking for audio generation." 2024.
[47] R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers. and G. Weber, "Common voice: A mnasiiely-multilingue speech corpus," 2020.
12

---

[48] A. Blattmann, T. Dockhom, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz. Y. Levi, Z. English, V. Voleti,
A. Letts al., "Stable video diffusion: Scaling latent video diffuusion models to large datasets," arXiv preprint 1511 2023.
[49] Anonymous, "Stable audio," "
hutps//sabblllyy//stbbbaaauo 2024, accessed: 2024-06-12 12.
[50] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall, "Dreamfusion: Text-to-3d using 2d diffusion," arXiv, 2022.
[51] M. Deitke, R. Liu, M. Wallingford, H. Ngo, O. Michel, A. Kusupati, A. Fan, C. Laforte, V. Voleti, S. Y. Gadre al., "Objaverse-xl: A universe of 10m+ 3d objects," Advances in Neural Information Processing Systems, vol. 36, 2024.
[52] H. Ci, M. Wu, W. Zhu, X. Ma, H. Dong, F. Zhong, and Y. Wang, "Gfpose: Learning 3d human pose prior with gradient fields." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 4800-880018 4810.
13

---

A Supplementary Materials
A.1 Watermark implementation details
Tree-Ring watermarks [4] are implemented following the corrected version proposed by RingID Tree-Ring Ring watermarks exhibited an issue where the watermark pattern actually injected into the initial noise pattern was inconsistent with the pattern claimed by Tree-Ring [14].
[]6] under the verification setting. In comparison, the original Tree-
Our implementation adopts the original ring pattern proposed by Tree-Ring 14 with radii from to 10, but the ring pattern is centred at the origin in the Fourier domain, following the correction proposed by RingID [16.. To ensure lossless watermark injection, we discard the imaginary part of the sampled watermark pattern during watermark detection. This lossless injection ensures that the watermark pattern actually injected matches the reference pattern used during detection.
RingID [16] is implemented using patterns with radii ranging from to 14 in the identification setting.
A.2 Perceptual quality of images under distortions
The distortions applied in plotting Figure 4 are violent, as visualised in Figure 8 Blurring with radius of 13 renders the cat's eyes invisible. Perturbing with a 100 noise eliminates the cat's pattern. Upscaling the brightness to eight times overexposes the entire background. However, as shown in Figure none of these distortions can defeat Tree-Ring's detector, but steganalysis-base watermark removal can. This is possibly due to Tree-Ring's large-scale content-agnostic ripple-like
No Removal
x1
x2
x3
Ours
No Removal
r=1
r=5
r=9
r=
Gaussian Blur
No Removal
== 10
=
= 50
= 100
Gaussian Noise
No Removal
x2
x4
x6
x8
Brightness
Figure 8: Visualisation of images under different strengths of steganalysis- watermark removal (blackbox, n = 5000) and image distortions. We amplify the signal strength of the extracted pattern by multiplication with factor.
14

---

patterns according to our analysis in the main paper. This demonstrates that steganalysis- based watermark removal effectively captures the vulnerability of the watermarking method through the extracted pattern, even without knowledge about the watermarking algorithm. Subsequently, it highlights that Tree-Ring's robustness to distortions is a Maginot Line that can be breached with steganalysis attacks.
A.3 More visualisations of patterns extracted
In the main text, we have visualised the patterns extracted from content-agnostic watermarking methods. In this section, we present more comprehensive visualisation of patterns extracted from content-agnostic methods in Figure | along with visualisations of the patterns extracted from content-adaptive watermarking techniques, as depicted in Figure I0
While subtracting these patterns from watermarked images may not effectively circumvent the watermark detection process, the extracted patterns still exhibit regularities. By averaging 5000 images, the patterns extracted from both Stable Signature [22] and WmAdapter (27] display mild repetitive grid-like structures, suggesting that images watermarked with this technique may share commonalities at specific frequency components. For such repetitive patterns, it would be worth exploring whether they are caused by the patching mechanism of the transformer architecture. Patterns extracted by RivaGAN [T] exhibit a greenish tint, indicating RivaGAN introduces a systematic bias during the watermarking king process despite addaptt- components. The patterns extracted by SSL 25]. similar to RAWatermark [23], are biased towards a specific noise distribution. However, while RAWatermark (3 is vulnerable to steganalysis-basce
removal, SSL [25] demonstrates robustness against such attacks. As the parameter n. increases, the noise components in the patterns extracted by HiDDeN [T0] transition towards greyish hue and eventually diminish. By the gray-world assumption principle, this observation suggests that HiDDeN best incorporates content- -onteit-adaptive watermarks into the images. Under spatial domain averaging, DwtDet [2] reveals nothing but tiny systematic bias, which aligns with the fact
We also visualise audio patterns extracted from AudioSeal [45] and WavMark [46] in Figure When n = 5000, the greybox AudioSeal-extratter pattern is highly imperceptible, with a signal strength below -45dB relative to the cover media. However, directly subtracting this pattern in the time domain significantly reduces the watermark detection accuracy by 30%. In contrast, WavMark- extracted patterns have larger amplitudes, indicating that WavMark also introduces systematic bias during the watermarking process. Moreover, greybox-extractee WavMark patterns concentrate within the first second, which aligns with WavMark's 's method of adding watermark patterns at one-second intervals. This exposes WavMark's 's watermarking locations, allowing an adversary to potentially remove the watermark by simply clipping out these segments. The spectrograms further show that WavMark mainly adds watermarks below 6 kHz, which could represent the robustness threshold it was trained to withstand against low-pass filtering. In summary, although both methods claim robustness against various distortion- based attacks, their vulnerabilities can be casily exposed through simple steganalysis techniques.
A.4 Perceptual quality of images after watermark removal
In the main text, we analyzed the perceptual quality of Tree-Ring watermarked images after watermark removal. We now visualise content- aggnoiiainn watermarked images after watermark removal in
Figure 12] and content-adaptively watermarked images after watermark removal in Figure [3
From these two figures, we can observe that except for Gaussian Shading [19] images carrying different watermarks, after watermark removal, have nearly identical visual quality. The visual quality is predominantly related n, the number of images averaged during pattern extraction. In the greybox setting, when n >> 50, there are no visually apparent artefacts. In the blackbox setting, n > 100 is generally required for eliminating significant artefact patterns. For images with Gaussian Shading [19] watermarks, due to the large magnitude of the extracted watermark pattern itself, steganalysis- based removal inevitably causes significant changes to the image content, thereby reducing perceptual quality. This leads to two insights:
For adversaries, averaging more images allows for a better approximation of an effective content-agnostic watermark pattern that can be used for watermark removal;
15

---

Number of Watermarked Images Averaged
5
10
50
100
500
1000
5000
Greyboxx
Troo-King
Blackbox
Greybox
RingiD
Blackbox
Greybox
RAWatemmar
Blackbok
Greybox
DutDossd
Blackbox
Greyyoo
RosteAL
Blackbox
Greybox
Carrodan Shadlog
Blackbok
Figure 9: Visualisation of watermark pattern extracted from content-agnostic methods. All patterns adaptively normalised before visualisation.
16

---

Number of Watermarked Images Averaged
5
10
100
500
1000
5000
Graybox
Stabll SSigrttung
Blackbox
Graybox
Winddpptrr
Blackbox
Graybox
RIaGANN
Blackbox
Graybox
Blackbox
Graybox
HIDDON
Blackbox
Graybox
Dwttxt
Blackbox
Figure 10: Visualisation of patterns extracted from content-adaptive watermarking methods. All patterns are adaptively normalised before visualisation.
17

---

AudioSeal Blackbox
AudioSeal Greybox
WavMark Blackbox
WavMark Greybox
Rro bet anp
n n=5
F R
Time
Timee
n=
0.0
0.5
Time
Thme rit
TI 50
Fre (40) A0e
Me
ING PMM
Fre (w)
Fox
F
0.5
Time
Thrre [a]
TI ==
meeltin Ame
8
0.5
1.3
0.5
15
58
15-000
Dai Amp
3.5
155
0.5
1.5
0.5
1.5
TI 550000
Fre (oe am1
Fres (4G
2.5
2.5
1.5
0.5
1.5
05
Figure 11: Visualisation of audio patterns extracted. Each subplot contains a time-domain audio
signal (top) and its spectrogram (bottom). Amp stands for (time-domain) audio signal amplitude. The amplitudes are normalised to within [-1, 1].
2.
For watermark distributors, reducing the count of distributing the same watermark could lower the security risk posed by steganalysis.
A.5 More results on quantitative analysis on watermark removal
In Section 4.2 we primarily discussed the Peak Signal-to-Noise Ratio (PSNR) as measure of image
distortion resulting from our steganalysis-bart based watermark removal method. To provide more comprehensive assessment of the impact on visual quality, we also evaluated three additional metrics: Structural Similarity Index (SSIM), Learned Perceptual Image Patch Similarity (LPIPS), and Single Image Fréchet Inception Distance (SIFID).
Tables 2 13 present the performance of various watermarking methods under our steganalysis-basse removal attack, along with the corresponding image quality metrics. The trends observed in PSNR are largely mirrored in the additional metrics:
Content-Agnostie Methods For methods like Tree-Ring [4), RingID [16). RAWatermark (23). DwtDetSvd vd [12), and RoSteALS [3], all metrics show a consistent trend: as the number of images averaged (n) decreases, watermark removal becomes more effective (lower detection rates), but at the cost of increased image distortion (lower PSNR, SSIM, and higher LPIPS, SIFID).
For example, in the case of RAWatermark (Table 4 Blackbox setting), as decreases from 5000 to 5, the AUC drops from 0.574 to 0. 133, indicating more effective watermark removal. However, this comes at the cost of image quality: PSNR drops from 27.98 to 17.92, SSIM from 0.964 to 0.528, while LPIPS increases from 0.020 to 0.424, and SIFID from 0.022 to 1.401, all indicating significant visual degradation.
18

---

Number of Watermarked Images Averaged
No Removal
5
10
50
100
500
5000
Greyyoox
Thee
Blockbox
Graybox
Ringh
Blackdoo
Grayhox
RAWatemark
Blackbox
Graybox
Doundssd
Hackboo
Graybox
RobteALS
Bladkbox
Gresbox
Gauodan Shoelin
Blackbox
Figure 12: Visualisation content- agnosticly watermarked images after watermark removal.
19

---

Number of Watermarked Images Averaged
No Removal
5
10
50
100
500
5000
Grayboo
Slaable Skg satuur
Blackbox
Graybox
Wondlpper
Blarkóoo
Graybox
RioGAN
Blackbox
Crayboo
Hlacbbo
Graybon
HIDDN
Bladkbox
Greybox
DAtDEt
Blackbox
Figure 13: Visualisation of cdntent-adaptively watermarked images after watermark removal.
20

---

Bladkbox
Greybox
16
16
0.8
(dB)
15
0.8
0.6
0.6
14
0.4
0.4
0.2
13
13
12
2 3
2
3
5
6
5
Horizontal Axis: Number of Heterogeneous Tree- Free-inng Keys Mixed in the Adversary's 's Collection
# imgs Avged Durin Extraction: - No Removal -+-5 ++10
20
-+50
++1100
-00
+0000
++1000
-002000
+ 5000
Figure 14:
Ablation study on watermark removal performance with DwtDetSvd adversary's s image collection contains multiple watermark patterns.
12 when the
Content-Adaptive Methods In contrast, content-adaptive methods like Stable Signature [22). WmAdapter [27]. RivaGAN (T]] and SSL (25 maintain good detection performance (bit accuracies 0.95) even decreases, with the image quality metries also show minimal degradation. For instance, SSL (Table [T] greybox setting) maintains a high bit accuracy (0.895 at n = 5) while preserving image quality: PSNR (34.26), SSIM (0.920), LPIPS (0.048), and SIFID (0.048) all close to the no-removal (NRmv) values. This could be due to that the systematic biases averaged out from content-adaptive methods mild, so subtracting such mild patterns from the original images would not lead to significant perturbations.
A.6 Effectiveness of removal under multiple watermarks
In Section 4.4.4 our case study on Tree-Ring [14] demonstrated that when an adversary's 's image collection contains a mix of several different watermark patterns, the effectiveness of watermark removal is significantly reduced. Based on this finding, we proposed assigning multiple keys mitigation strategy against steganalysis threats. However, in this section, we caution that this approach not universal solution and should be applied judiciously. The high complexity of watermarking algorithms means that this method cannot guarantee enhanced watermark security. To illustrate this, replicate our experiment using the DwtDetSvd [12) algorithm.
Figure 14 shows that in the greybox scenario, DwtDeetSvd behaves similarly to Tree-Ring: as the adversary's image collection incorporates more diverse watermarks, watermark removal becomes less effective without significantly impacting image quality. Interestingly, this strategy fails in the blackbox setting. When the adversary's collection mixes more than 4 watermarks, DwtDetSvd's detection accuracy surprisingly drops below O.I. While the reasons behind this phenomenon warrant further investigation, this observation underscores merely a temporary workaround to bolster current angeott- watermarking algorithms. It does not address the fundamental vulnerabilities of these algorithms and may even be counterproductive in certain scenarios. Therefore, watermark distributors should employ this method cautiously and at
main point: assigning multiple watermarks per user is
Table 2: Performance (AUC) of Tree-Ring Line-Ring 14] ing image quality degradations. NRmv stands for no removal.
under steganalysis based removal and the correspond-
Images Averaged
NRmv
5
10
20
50
100
200
500
1000
2000
5000
AUC
1.000
0.293
0.267
0.314
0.275
0.214
0.228
0.211
0.224
0.224
0.241
PSNR
15.62
12.91
14.00
14.43
14.99
15.24
15.33
15.42
15.43
15.46
15.47
Blackbox
SSIM
0.555
0.298
0.355
0.431
0.482
0.512
0.528
0.540
0.545
0.547
0.548
LPIPS
0.411
0.609
0.566
0.514
0.472
0.451
0.440
0.430
0.427
0.425
0.425
SIFTD
0.375
1.205
0.751
0.461
0.409
0.392
0.396
0.402
0.404
0.404
0.404
AUC
1.000
0.141
0.179
0.246
0.260
0.213
0.251
0.237
0.230
0.241
0.259
PSNR
15.62
14.64
14.97
15.33
15.53
15.58
15.63
15.65
15.66
15.66
15.67
greybox
SSIM
0.555
0.346
0.401
0.473
0.511
0.531
0.542
0.549
0.551
0.552
0.553
LPIPS
0.411
0.554
0.523
0.479
0.449
0.433
0.422
0.415
0.413
0.412
0.412
SIFID
0.375
0.955
0.588
0.402
0.367
0.364
0.367
0.371
0.372
0.373
0.373
21

---

Table 3: Performance (decoding accuracy) of RinglD [16] under steganalysis- removal and the
corresponding image quality degradations. NRmv stands for no removal.
Images Averaged
NRmv
5
10
20
50
100
200
500
1000
2000
5000
Dec Acc
1.000
0.180
0.100
0.140
0.110
0.110
0.130
0.130
0.130
0.130
0.130
PSNR
13.06
11.70
12.21
12.62
13.01
13.12
13.19
13.24
13.26
13.28
13.28
Blackbox
SSTM
0.459
0.240
0.276
0.345
0.392
0.418
0.430
0.441
0.446
0.448
0.449
LPIPS
0.460
0,595
0.564
0.523
0.495
0.483
0.479
0.477
0.475
0.475
0.475
SIFID
0.250
0.613
0.380
0.266
0.258
0,255
0,259
0.260
0.260
0,259
0.259
Dec Ace
1.000
0.070
0.050
0,100
0.090
0.110
0.130
0.120
0.130
0.140
0.140
PSNR
13.06
12.19
12.72
13.08
13.28
13.31
13.34
13.36
13.37
13.37
13.37
greybox
SSIM
0.459
0.245
0.293
0.368
0,410
0.430
0.440
0.446
0.449
0.450
0.451
LPIPS 0.460
0.574
0.545
0.509
0.486
0.477
0.472
0.470
0.469
0.469
0.469
SIFID
0.250
0.591
0.352
0.253
0.239
0.241
0.243
0.244
0.245
0.245
0.246
Table 4: Performance (AUC) of RAWatermark 23 corresponding image quality degradations. NRmv stands for removal.
under steganalysis-based removal and the
Images Averaged
NRmv
5
10
20
50
100
200
500
1000
2000
5000
AUC
0.714
0.133
0.205
0.219
0.315
0.379
0.394
0.494
0,540
0,566
0.574
PSNR
28.83
17.92
20.67
22.38
24.86
27.21
28.14
29.15
27.81
27.86
27.98
Blackbox
SSIM
0.928
0.528
0.659
0.764
0.862
0.916
0.939
0.960
0.959
0.961
0.964
LPIPS
0.028
0.424
0.327
0.236
0.149
0.096
0.065
0.036
0.028
0.022
0.020
SIFID
0.017
1.401
0.696
0,286
0.102
0.047
0.039
0.022
0.024
0.023
0.022
AUC
0.714
0,500
0,501
0,501
0.501
0.501
0.502
0,502
0,502
0.502
0.502
PSNR
28.83
$2.64
52.55
53.02
54.96
56.78
$7.37
57.65
57.65
$7.68
$7.67
greybox
SSIM
0.928
0.998
0.998
0.998
0.998
0.998
0.998
0.998
0.998
0.998
0.998
LPIPS
0.028
0.001
0.001
0.001
0.001
0.001
0.001
0.001
0.001
0.001
0.001
SIFID
0.017
0.001
0.001
0.001
0.001
0.001
0.001
0.001
0.001
0.001
0,001
Table 5: Performance of DwtDetSvd 12
under steganalysis-based removal and the corresponding
image quality degradations. NRmv stands for no removal.
Images Averaged
NRmv
10
20
50
100
200
500
1000
2000
5000
Bit Acc
1.000
0.485
0.494
0.534
0.526
0.548
0.340
0.482
0.478
0.428
0.572
PSNR
37.59
17.89
20.58
22.28
24.67
26.88
27.74
28.62
27.41
27.46
27.57
Blackbox
SSIM
0.975
0.521
0.650
0.753
0,847
0.900
0.923
0.942
0.941
0.943
0.945
LPIPS
0.019
0.426
0.330
0.240
0.154
0.102
0.071
0.044
0.037
0.032
0.030
SIFID
0.017
1.401
0.704
0.296
0.113
0.057
0.048
0.030
0.033
0.031
0.030
Bit Ace
1.000
0.639
0.562
0.541
0.585
0.580
0.531
0.480
0.470
0.401
0.317
PSNR
37.59
37.74
38,21
38.38
38.47
38.61
38.67
38.70
38.71
38.69
38.67
greybox
SSIM
0.975
0.967
0.972
0.973
0.974
0.976
0.976
0.976
0.976
0.977
0.977
LPIPS
0.019
0.018
0.015
0.014
0.013
0.013
0.013
0.013
0.013
0.013
0.013
SIFID
0.017
0.012
0.010
0.010
0.009
0.009
0.009
0.009
0.009
0.009
0.009
Table 6: Performance (bit accuracy) of RoSteALS 13 under steganalysis based removal and the
corresponding image quality degradations. NRmv stands for no removal.
Images Averaged
NRmv
10
20
50
100
200
500
1000
2000)
5000
Bit Ace
0.994
0.402
0.394
0.390
0.360
0.322
0.273
0.245
0.241
0.243
0.244
PSNR
28.00
17.51
19.81
21.19
22.94
24.38
24.87
25.37
24.68
24.71
24.77
Blackbox
SSIM
0.858
0.457
0.566
0.661
0.747
0.794
0.816
0.833
0.834
0.836
0.838
LPIPS
0.039
0.395
0.305
0.224
0.149
0.109
0.087
0.068
0.065
0.061
0.059
SIFID
0.048
1.606
0.795
0.347
0.145
0.095
0.090
0.074
0.077
0.075
0.074
Bit Acc
0.994
0.225
0.254
0.274
0.272
0.262
0,240
0.229
0.230
0.236
0.238
PSNR
28.00
26.88
27.68
28.08
28.29
28.39
28.40
28.42
28.43
28.44
28.44
greybox
SSIM
0.858
0.750
0.805
0.831
0.846
0.852
0.855
0.856
0.856
0.857
0.857
LPIPS
0.039
0.153
0.099
0.071
0.054
0.047
0.045
0.045
0.045
0.044
0.045
SIFID
0.048
0.140
0.072
0.056
0,050
0.049
0,049
0.049
0.049
0.049
0.049
22

---

Table 7: Performance (bit accuracy) of Gaussian Shading 19 the corresponding image quality degradations. NRmv stands for no removal.
under steganalysis-based removal and
Images Averaged
NRmv
10
20
50
100
200
500
1000
2000)
5000
Bit Acc
0,999
0.490
0.469
0.537
0.488
0.486
0.479
0.461
0.463
0.465
0.462
PSNR
9.726
9.485
9.754
9.844
9.956
10.09
10.11
10.13
10.12
10.12
10.12
Blackbox
SSIM
0.322
0.164
0.197
0.242
0.268
0.286
0.294
0.301
0.305
0.306
0.307
LPIPS
0.613
0.686
0.672
0.648
0.636
0.631
0.632
0.632
0.632
0.631
0.632
SIFID
0.471
0.831
0.604
0.478
0.449
0.443
0.443
0,444
0,444
0.444
0,443
Bit Acc
0.999
0.507
0.501
0.540
0.490
0.490
0.479
0.462
0.461
0.462
0.462
PSNR
9.726
9,552
9.791
9.961
10.03
10.11
10.14
10.16
10.16
10.17
10.17
greybox
SSIM
0.322
0.161
0.196
0.248
0.273
0.290
0.298
0.302
0.305
0.306
0.306
LPIPS
0.613
0.669
0.660
0.639
0.630
0.626
0.626
0.627
0.627
0.627
0.627
SIFID
0.471
0.802
0.597
0.474
0.449
0.444
0.439
0.437
0.438
0.438
0.438
Table 8: Performance (bit accuracy) of Stable Signature [22 the corresponding image quality degradations. NRmv stands for no removal.
under steganalysis-based removal and
Images Averaged
NRmv
20
50
100
200
500
1000
2000
5000
Bit Acc
0.998
0.929
0.970
0.994
0.998
0.998
0.998
0.998
0.998
0.998
0.998
PSNR
30.28
15.75
19.46
22.23
23.99
25.15
26.31
28.88
29.62
29.87
29.85
Blackbox
SSIM
0.879
0.526
0.613
0.698
0.768
0.805
0.831
0.861
0.869
0.873
0.874
LPIPS
0.049
0.437
0.357
0.254
0,168
0.120
0.091
0.065
0.057
0.054
0.052
SIFID
0.068
1.015
0.575
0.246
0.143
0.115
0.106
0.086
0.077
0.074
0.072
Bit Acc
0.998
0.997
0,998
0.998
0.998
0.998
0.998
0.998
0.998
0.998
0.998
PSNR
30.28
29.05
29.80
30.28
30.45
30.51
30.52
30.53
30.54
30.54
30.55
greybox
SSIM
0.879
0.790
0.833
0.860
0.871
0.875
0.876
0.877
0.878
0.878
0.878
LPIPS
0.049
0.167
0.115
0.074
0.056
0.052
0.050
0.050
0.049
0.049
0.049
SIFID
0.068
0.152
0.095
0.074
0.069
0.068
0,067
0.067
0.067
0.067
0.067
Table 9: Performance (bit accuracy) of WmAdapter 27
under steganalysis-based removal and the
corresponding image quality degradations. NRmv stands for no removal.
Images Averaged
NRmv
10
20
50
100
200
500
1000
2000
5000
Bit Acc
0.902
0.825
0.873
0.888
0.897
0.898
0.903
0.902
0.903
0.903
0.903
PSNR
35.18
15.74
19.67
22.75
24.80
26.17
27.74
31.78
33.13
33.60
33.60
Blackbox
SSIM
0.960
0.571
0.667
0.761
0.836
0.876
0.905
0.939
0.948
0.952
0.953
LPIPS
0.010
0.338
0.269
0.171
0.093
0.049
0.029
0.016
0.013
0.012
0.011
SIFID
0.004
0.318
0.155
0.050
0.023
0.016
0.015
0.010
0.007
0.006
0.006
Bit Ace
0.902
0.904
0.904
0.902
0,902
0.903
0.903
0.903
0,903
0.904
0.903
PSNR
35.18
34.12
34,70
35.04
35.18
35.22
35.23
35.25
35.26
35,26
35.26
greybox
SSIM
0.960
0.924
0.942
0.952
0.956
0.957
0.958
0.958
0.959
0.959
0.959
LPIPS
0.010
0.033
0.018
0.012
0.011
0.010
0.010
0.010
0.010
0.010
0.010
SIFID
0.004
0.007
0.005
0.005
0.004
0.004
0,004
0.004
0.004
0.004
0.004
Table 10: Performance (bit accuracy) of RivaGAN [1 under steganalysis- bassed removal and the corresponding image quality degradations. NRmv stands for no removal.
Images Averaged
NRmv
10
20
50
100
200
500
1000
2000)
5000
Bit Ace
0.973
0.738
0.800
0.864
0,902
0.930
0.946
0.959
0.961
0.963
0.967
PSNR
39.84
17.91
20.63
22.35
24.79
27.07
27.96
28.91
27.63
27.69
27.80
Blackbox
SSIM
0.979
0.525
0.654
0.757
0.851
0.904
0.925
0.945
0.944
0.946
0.948
LPIPS
0.036
0.426
0.331
0.243
0.159
0.110
0.083
0.059
0.054
0.050
0.049
SIFID
0.067
1.465
0.773
0.362
0.176
0.118
108
0.088
0.089
0.087
0.085
Bit Acc
0.973
0.972
0.973
0.972
0.972
0.972
0.972
0.972
0.972
0.973
0.973
PSNR
39.84
39.61
40.01
40.17
40.26
40.30
40.29
40.28
40.28
40,27
40.27
greybox
SSIM
0.979
0.974
0.977
0.978
0.979
0.980
0.980
0.980
0.980
0.980
0.980
LPIPS
0.036
0.045
0.040
0.038
0.037
0.036
0.036
0.036
0.035
0.035
0.035
SIFID
0.067
0.085
0.074
0.070
0.067
0.066
0,065
0.065
0.065
0.065
0.065
23

---

Table 11: Performance (bit accuracy) of SSL 25
under steganalysis- removal and the corre-
sponding image quality degradations. NRmv stands for no removal.
Images Averaged
NRmv
10
20
50
100
200
500
1000
2000
5000
Bit Acc
0.928
0.531
0.587
0.651
0.719
0.778
0.826
0.880
0.905
0.919
0.917
PSNR
35.05
15.78
19.77
22.85
24.88
26.22
27.81
31.80
33.07
33.52
33.52
Blackbox
SSIM
0.937
0.585
0.675
0.752
0.820
0.858
0.887
0.918
0.927
0.931
0.931
LPIPS
0.040
0,302
0.234
0.155
0.098
0.069
0.055
0.046
0.043
0.042
0.041
SIFID
0.037
0.415
0.250
0.121
0.071
0.057
0.051
0.043
0.040
0.039
0.038
Bit Ace
0.928
0.895
0.912
0.921
0,924
0.923
0.926
0.927
0.929
0.929
0.929
PSNR
35.05
34.26
34.61
34.79
34.90
34.94
34.96
34.98
35.00
35.02
35.04
greybox
SSIM
0.937
0.920
0.928
0.932
0.935
0.936
0.936
0.936
0.936
0.937
0.937
LPIPS
0.040
0.048
0.044
0.042
0.041
0.041
0.041
0.041
0.041
0.041
0.041
SIFID
0.037
0.048
0.042
0.039
0.038
0.037
0.037
0.037
0.037
0.037
0.037
Table 12: Performance (bit accuracy) of HiDDeN 100
under steganalysis- removal and the
corresponding image quality degradations. NRmv stands for no removal.
Images Averaged
NRmv
10
20
50
100
200
500
1000
2000
5000
Bit Acc
0.976
0.804
0.835
0.888
0.928
0.942
0.946
0.954
0.959
0.960
0.961
PSNR
36.67
17.88
20.44
21.78
24.27
26.42
28.00
28.68
27.61
27.27
27.80
Blackbox
SSIM
0.956
0.521
0.639
0.738
0.839
0.886
0.913
0.931
0.931
0.932
0.936
LPIPS
0.012
0,322
0.244
0.158
0.086
0.053
0.037
0.025
0.021
0.019
0.018
SIFID
0.018
0,643
0.348
0.156
0,058
0.037
0.030
0.026
0.025
0.024
0.023
Bit Acc
0.976
0.958
0.960
0,962
0.962
0.961
0.960
0.960
0.960
PSNR
36.67
36.00
36.45
36.69
36.85
36.90
36.93
36.97
36.99
37.01
37.04
greybox
SSIM
0.956
0.941
0.949
0.953
0,955
0.956
0.956
0.956
0.956
0.956
0.956
LPIPS
0.012
0.017
0.014
0.013
0.012
0.012
0.012
0.012
0.012
0.012
0.012
SIFID
0.018
0.022
0.019
0.018
0.018
0.018
0.017
0.017
0.017
0.017
0.017
their own discretion, understanding that is not a comprehensive solution the inherent security challenges of content-agnosti watermarking algorithms.
Table 13: Performance (bit accuracy) of DwtDet 11
under steganalysis-basee based removal and the corresponding image quality degradations. NRmv stands for no removal.
Images Averaged
NRmv
50
100
200
5000
Bit Acc
1.000
0.989
0.994
0.996
0.997
0.998
0.999
0.998
0.998
0.998
0.998
PSNR
37.61
17.88
20.57
22.27
24.64
26.82
27.67
28.54
27.35
27.40
27.51
Blackbox
SSIM
0.962
0.518
0.645
0.745
0.836
0.887
0.909
0.927
0.926
0.928
0.930
LPIPS
0.036
0.427
0.331
0.241
0.157
0.107
0,080
0.058
0.055
0.053
0.052
SIFID
0.063
1.454
0.766
0.352
0.168
0.110
0.096
0.079
0.080
0.078
0.076
Bit Ace
1.000
1,000
1.000
1.000
1,000
1.000
1,000
1.000
1.000
1.000
1.000
PSNR
37.61
36.79
37.17
37.31
37.39
37.46
37.49
37.49
37.49
37.47
37.46
greybox
SSIM
0.962
0,959
0.960
0.960
0.961
0.961
LPIPS
0.036
0.042
0.039
0.038
0.037
0.037
0.036
0.036
0.036
0.036
0.036
SIFID
0.063
0.075
0.069
0.066
0,064
0.063
0.063
0.063
0.063
0.063
0.063
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims are accurately reflected in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Yes. The limitations are discussed in Section Iisionnounaae uuder the paragraph "limita-
tions".
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
25

---

Justification:
The paper does not include any theoretical results proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes)
Justification: Yes. The experimental setups are provided in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer:
[No]
Justification:
The code and data are not provided in this manuscript submission.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeeee //nips.com htte://alllllgiiiee///oeeeuaaass ssiniiiiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
Yes. The experimental setups are provided in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: No. Error bars not presented because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
27

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes. These are included in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Yes. The research conforms to the NeurIPS Code of Ethics, as discussed in Section
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes. The broader impacts, including potential positive and negative societal impacts, are discussed in Section 6
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
28

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer:[NA]
Justification:
The paper does not release any data or models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes. The use of existing datasets and evaluation methods are all credited with citations, requested by the property owners or licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
29

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce any assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "UinAdIlF8g", "reviewer": "Reviewer_aL14", "summary": "This paper introduces a steganalysis-based attack aimed at image watermarking methods. The attack is effective in both gray-box and black-box scenarios and focuses on identifying repeating patterns present in watermarked images. These patterns can be exploited to either remove watermarks or add them to non-watermarked images without needing access to the original watermarking algorithm. Experimental results demonstrate that this attack can successfully compromise watermarks that use content-agnostic watermark patterns.", "questions": "- I suggest that the authors compare their attack to existing watermark attacks (e.g., from this watermark benchmark [1]). This comparison is necessary to illustrate the advantages of their attacks.\n\n- The results for the forgery attack are provided only for the Tree-Ring watermark. I recommend including results for other watermarks as well to provide a more comprehensive evaluation.\n\n\n[1] An, B., Ding, M., Rabbani, T., Agrawal, A., Xu, Y., Deng, C., Zhu, S., Mohamed, A., Wen, Y., Goldstein, T., et al.: Benchmarking the robustness of image watermarks, 2024.", "limitations": "- The attack only works on content-agnostic watermarks, which the authors adequately mentioned in several parts of the paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "- The authors illustrate the vulnerability of content-agnostic watermarks to simple attacks, serving as a good reminder for researchers to ensure their watermarks are content-aware.\n\n- The proposed attack is simple and compared to some existing watermark attacks, has a lower computational cost. However, it's worth noting that the computational cost of other attacks is usually not significant either, so the value of this simplicity and speed-up remains debatable.", "weaknesses": "- Similar ideas (i.e., considering watermarked and non-watermarked samples to extract watermark patterns for removal and forgery) have been explored in existing works [1][2]. For instance, [1] discusses a spoofing attack using a noise image $X$, watermarking it to obtain $X_{wm}$, and then adding $(X_{wm}-X)$ to clean images to create watermarked versions. Both [1] and [2] also describe a surrogate model adversarial attack, which involves training a CNN to distinguish between watermarked and non-watermarked images. This CNN is then used to adversarially alter samples to change their watermark status. This adversarial attack is a more advanced version of the method suggested in this paper and can potentially target content-aware watermarks as well.\n\n- The authors claim, \"Notably, our steganalysis-based approach is the first to effectively remove Tree-Ring watermarks without access to the algorithm,\" which is inaccurate. The adversarial attack proposed in [1], though ML-based, requires only a set of watermarked and non-watermarked samples (similar to the black-box attack described in this paper) to break Tree-Ring.\n\n- The method introduced in this paper lacks novelty, and the vulnerability of content-agnostic watermarks to this type of attack is trivial and not a significant discovery.\n\n[1] Saberi, M., Sadasivan, V. S., Rezaei, K., Kumar, A., Chegini, A., Wang, W., and Feizi, S. Robustness of AI-image detectors: Fundamental limits and practical attacks, 2023.\n\n[2] An, B., Ding, M., Rabbani, T., Agrawal, A., Xu, Y., Deng, C., Zhu, S., Mohamed, A., Wen, Y., Goldstein, T., et al.: Benchmarking the robustness of image watermarks, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Similar ideas have been explored in existing works [1, 2].\n\nWe argue that our work fundamentally differs from [1, 2] with distinctive advantages.\n\n**Differences with the spoofing attack in [1]**\n\nThe spoofing attack described in [1] requires access to the watermark encoder (white-box) and does not generalize to removal, while our steganalysis-based approach unifies removal and forgery without accessing the decoder (black-box). In terms of explainability, our method extends beyond [1] by using steganalysis to extract content-agnostic watermarks. This step enables visualizing to show the existence of the extracted and added patterns during forgery, whereas [1], lacking this step, cannot explain why and which component of the weak overlayed watermark signals enables spoofing. We thus believe the idea explored by [1] is fundamentally different from our work.\n\n**Differences with adversarial attacks in [1, 2]**\n\nSteganalysis, adversarial, and distortion-based attacks are parallel methods, each fundamentally different from the others. Steganalysis attacks involve extracting hidden information to facilitate removal, adversarial attacks focus on solving optimization problems to harness adversarial examples, and distortion-based attacks typically employ image-processing techniques to create distortions. They are not variations of one another; therefore, we don't think one is a more advanced version of another.\n\nSpecifically, steganalysis has advantages over the adversarial attacks described in [1, 2]: our approach can effectively remove watermarks from just 500 images without leaving noticeable artifacts, seamlessly unifying both removal and forgery under whitebox and blackbox scenarios. In contrast, the adversarial attacks in [1, 2] require at least 3000 images for training. The attack in [2] also does not support forgery, as perturbations disrupt the entire latent space. The claim about the potential applicability of AdvCls to content-aware watermarks remains debatable [1, 2], as both work only demonstrated effectiveness in removing Tree-Ring watermarks unless with large perturbations ($\\epsilon>10$) [1]. [1] fails on StegaStamp. [2] fails on Stable Signature and StegaStamp.\n\n**In conclusion**, the uniqueness of steganalysis seamlessly unifies removal and forgery with visualizable explainability under both whitebox and blackbox settings. It requires significantly fewer images than adversarial attacks, further showcasing its advantage over existing approaches.\n\n　\n\n> Steganalysis is not the first blackbox method, as [1] also works under blackbox.\n\nThanks for pointing out. We will rephrase to clarify that our method is the first that is training-free.\n\n　\n\n> The method lacks novelty, and the vulnerability is trivial.\n\nWe argue that our method is novel, and the vulnerability is severe. As outlined in our common rebuttal:\n- **Novelty**\n  - We are the first to introduce a blackbox training-free method that effectively removes and forges Tree-Ring watermarks.\n  - We are the first to explain what modern content-agnostic watermark patterns look like (such as the Tree-Ring ripples), and why they can be easily removed.\n  - We highlight a future direction for robust watermarking: to address the content-agnostic issue by evaluating against steganalysis attacks, similar to how robustness is evaluated against distortions.\n- **Severity and Impact**: Our experiments show that a sufficiently simple and straightforward method easily fools several modern watermark detectors. The vulnerability is therefore severe with wide impact.\n\n　\n\n> Comparison with the watermark benchmark [2].\n\nWe compared with attacks in [2] following its Detection setting, reporting TPR@0.1%FPR (`low_1000` in its code). The comparison is on Tree-Ring watermarks, which is the watermark that could be successfully removed by both [2] and our work.\n\n|Method|TPR@0.1%FPR|\n|---|---|\n|Dist-Rotation / Rcrop / Erase / Bright / Contrast / Blur / Noise / JPEG|0.009 / 0.013 / 1.000 / 0.992 / 0.995 / 0.230 / 0.945 / 0.688|\n|DistCom-Geo / Photo / Deg|0.016 / 0.994 / 0.726|\n|Regen-Diff / DiffP / VAE / KLVAE / 2xDiff / 4xDiff|0.407 / 0.397 / 0.464 / 0.964 / 0.222 / 0.171|\n|AdvEmbG-KLVAE8 / RN18 / CLIP / KLVAE16 / SdxlVAE|0.152 / 0.898 / 0.853 / 0.751 / 0.835|\n|AdvCls-UnWM&WM / Real&WM / WM1&WM2|0.291 / 1.000 / 0.253|\n|Ours n=5 / 10 / 20 / 50 / 100 / 200 / 500 / 1000 / 2000 / 5000|**0.000** / **0.000** / **0.000** / **0.000** / **0.000** / **0.000** / **0.000** / **0.000** / **0.000** / **0.000**|\n\nAs can be seen, our removal lowers TPR@0.1%FPR to near 0, more effective than any attacks in [2].\n\n　\n\n> Include forgery results for other watermarks to provide a more comprehensive evaluation.\n\n|Method/Metric|Setting|NoForgery|5|10|20|50|100|200|500|1000|2000|5000|\n|---|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|ROS (Bit Acc)|Graybox|0.506|0.990|0.988|0.987|0.987|0.987|0.988|0.989|0.988|0.988|0.988|\n|RAW (AUC)|Graybox|0.500|0.732|0.732|0.730|0.727|0.723|0.722|0.721|0.721|0.721|0.721|\n|SVD (Bit Acc)|Graybox|0.520|0.568|0.558|0.554|0.562|0.563|0.560|0.561|0.559|0.562|0.557|\n|ROS (Bit Acc)|Blackbox|0.506|0.900|0.926|0.953|0.968|0.978|0.982|0.984|0.983|0.983|0.982|\n|RAW (AUC)|Blackbox|0.500|0.010|0.018|0.038|0.051|0.069|0.085|0.127|0.184|0.222|0.243|\n|SVD (Bit Acc)|Blackbox|0.520|0.530|0.530|0.510|0.528|0.535|0.480|0.517|0.488|0.493|0.517|\n\n- **RoSteALS (ROS)**: highly effective (0.9+ bit acc), suggesting RoSteALS adds linearly content-agnostic watermarks\n- **RAWatermark (RAW)**: moderately effective under graybox (0.72 AUC) but behave like removal under blackbox (0.25 AUC)\n- **DwtDctSvd (SVD)**: ineffective forgery, suggesting the effective removal could be due to $\\delta_w$ disrupting the decoding space (similar to AdvCls-WM1&WM2 in [2])\n\n　\n\n**References**\n\n[1] M. Saberi et al., Robustness of AI-image detectors: Fundamental limits and practical attacks, 2023.\n\n[2] B. An et al., Benchmarking the robustness of image watermarks, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ceo1YRsJsd", "reviewer": "Reviewer_Jv2m", "summary": "The paper addresses vulnerabilities in digital watermarking methods, especially those that are content-agnostic. These methods, which embed fixed watermark patterns regardless of image content, are susceptible to steganalysis attacks that can extract and manipulate these patterns, potentially removing or forging watermarks with minimal impact on perceptual quality.", "questions": "Please see above.", "limitations": "Please see above.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The findings of this paper are interesting, and it is reasonably well-written and easy to follow.\n\n1. This paper introduces a novel method for steganalysis that effectively exposes vulnerabilities in content-agnostic digital watermarking techniques. Using simple averaging to extract watermark patterns, the study demonstrates a practical approach that can be applied in gray-box and black-box settings. \n\n2. This paper provides a thorough evaluation of various watermarking methods under different settings. It includes quantitative and qualitative analyses across eight different watermarking techniques, highlighting the particular susceptibility of content-agnostic methods to steganalysis attacks.\n\n3. Recognizing the limitations of current watermarking methods, the paper proposes actionable security guidelines and strategies for enhancing digital watermark security.", "weaknesses": "1. The steganalysis method proposed in this paper is fundamentally general and not restricted to images. However, the experiments are confined to analyzing image watermarks. Providing evidence of successful steganalysis in other modalities, such as audio, would further validate and broaden the method's impact.\n\n2. The steganalysis method proposed in this paper can extract a unified low-level watermark pattern from tree-ring watermarked images. So why can the tree-ring change the layout of the generated image?\n\n3. The quality of the figures could be further improved. For example, the font in Figure 2.3.4 is too small. Please enlarge it for easier reading.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The method is fundamentally general and not restricted to images. Providing its effectiveness on other modalities, such as audio, would further validate and broaden the impact.\n\n**You are correct that the issue extends beyond images.** After conducting steganalysis on Audioseal [1] invisible audio watermarks, we found that it also adds content-agnostic watermark that could be removed (<0.76 detection rate). Your suggestion even promotes the significance of our work serving as a reminder of the re-emerging content-agnostic issue.\n\n| Graybox        |   NRmv  |    5    |    10   |    20   |    50   |   100   |   200   |   500   |   1000  |   2000  |   5000  |\n|----------------|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|\n| Detection rate | 1.000   | 0.376   | 0.463   | 0.540   | 0.636   | 0.675   | 0.694   | 0.737   | 0.750   | 0.750   | 0.755   |\n| SNR            | 30.432  | 26.647  | 27.435  | 28.690  | 29.516  | 29.828  | 30.107  | 30.102  | 30.305  | 30.408  | 30.343  |\n\n| Blackbox       |   NRmv  |    5    |   10   |   20   |    50   |   100   |   200   |   500   |   1000  |   2000  |   5000  |\n|----------------|:-------:|:-------:|:------:|:------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|\n| Detection rate | 1.000   | 0.352   | 0.244  | 0.269  | 0.399   | 0.517   | 0.561   | 0.637   | 0.622   | 0.669   | 0.729   |\n| SNR            | 30.432  | -0.146  | 3.290  | 7.865  | 13.468  | 16.908  | 19.534  | 22.235  | 24.502  | 26.831  | 28.450  |\n\n　\n\n> Why can Tree-Ring change the layout of the generated image?\n\n**Adding a Tree-Ring watermark inevitably changes the generation seed.** Replacing the low-frequency part of the initial noise with a Ring pattern will cause global changes of the noise pattern, and hence the layout of the generated image will change. However, we have visualized that Tree-Ring also contains a content-agnostic component, and removing this part of the watermark is enough to fool the tree-ring watermark detector, which makes Tree-Ring vulnerable.\n\n　\n\n> The quality of the figures could be further improved.\n\nWe will increase the font size in Figure 2-4 to enhance readability, and improve the overall quality of the figures.\n\n　\n\n**References**\n\n[1] Robin San Roman et al., Proactive Detection of Voice Cloning with Localized Watermarking, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zr5uMqb2SY", "reviewer": "Reviewer_85eL", "summary": "The paper introduces steganalysis techniques targeting watermark removal and forgery. The authors demonstrate through experiments that existing content-agnostic watermarking methods are unable to resist steganalysis attacks, advocating for the adoption of content-adaptive strategies.", "questions": "See Weaknesses", "limitations": "1. The premise of steganalysis for watermarks requires a large number of images embedded with watermarks, unlike attacks such as JPEG compression, which can directly affect the watermark itself. This limitation makes steganalysis attacks somewhat impractical in real-world applications because attackers may not always have access to a sufficient number of watermarked images.\n2. This method is only effective against content-agnostic watermarking.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper is interesting and well-written. The author introduces steganalysis attack can remove and forge watermarks from watermark images using simple linear addition and subtraction operations in both gray-box and black-box modes.", "weaknesses": "1. Although the examples provided by the authors in Figure 7 show that the images after watermark removal exhibit good image quality compared to the watermarked images. However, the PSNR data shown in Figure 2 are not ideal. For instance, in the first example \"Tree-Ring,\" the average PSNR of the removed watermark image is only 17. According to [1],  distortion of modified images with PSNRs lower than 36 dB is noticeable to human visual system. If the quality of the image is severely degraded after watermark removal, then the act of removing the watermark becomes meaningless.\n2. The author said “In contrast, traditional distortion-based removal techniques, such as noise perturbations or blurring, typically result in substantial perceptual degradation (as visualized in  Figure 9 in Appendix A.2)”. But I think comparing perceived degradation like this isn't fair because it's difficult to objectively evaluate the severity of different attacks.\n3. While the authors present an interesting steganalysis method for removing or forging content-agnostic watermarks, the contribution of this approach may be somewhat limited. Additionally, the method's requirement for a large number of pre-existing watermarked images poses practical challenges for real-world applications.\n\n[1] S. Sarreshtedari and M. A. Akhaee, “A source-channel coding approach to digital image protection and self-recovery,” IEEE Trans. Image Process., vol. 24, no. 7, pp. 2266–2277, Jul. 2015.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Why the reported PSNR for Tree-Ring is low (17dB)?\n\n**Short Answer: Our method achieves 29.79dB/34.58dB (blackbox/graybox, n=5000) PSNR when removing Tree-Ring watermarks, indicating minimal visible alteration. The 17dB PSNR value mentioned in the paper measures a different aspect.**\n\n$$\n\\text{Clean image } A \\overset{\\text{Add watermark}}{\\longrightarrow} B \\overset{\\text{Remove watermark}}{\\longrightarrow} C\n$$\n\nThe 17dB PSNR reported in the paper compares $A$ and $C$, reflecting the cumulative impact of watermarking and removal (Sec 4.1). However, this doesn't represent the degradation caused by watermark removal alone.\n\nTo assess removal-induced degradation, we calculate PSNR between $B$ and $C$, yielding 29.79dB/34.58dB (blackbox/graybox, n=5000) for Tree-Ring watermarks.\n\nTree-Ring, a diffusion-based semantic watermark method, alters image layout significantly. This results in a low PSNR (~16dB) between $A$ and $B$ (Figure 2 NRmv). Consequently, $C$'s PSNR relative to $A$ is also low due to layout differences, despite $C$ closely resembling $B$. This low $A$-$C$ PSNR doesn't accurately reflect the extent of modification during watermark removal.\n\n　\n\n> Comparing perceived degradation like this isn't fair because it's difficult to objectively evaluate the severity of different attacks.\n\n**Short Answer: We evaluated both subjectively and objectively. The objective quality evaluation is in Figure 3, which is supplimented by the visualizations (subjective quality evaluation) in your mentioned Figure 9.**\n\nFigure 9 supplements Figure 3 in the main text, which already includes objective quality evaluations. We evaluated four metrics: PSNR, SSIM, LPIPS, and SIFID, and plotted the trade-off curves between watermark removal effectiveness (vertical axis) and image quality degradation (horizontal axis) for different distortion methods. Methods closer to the lower-left corner of the graph are better because they achieve significant watermark removal with minimal distortion. Figure 9 visualizes the data points and their parameters used to plot these curves, thus serving as a complement to Figure 3.\n\n　\n\n> The method's requirement for a large number of pre-existing watermarked images poses practical challenges for real-world applications. Attacks such as JPEG compression can directly affect the watermark itself.\n\n - **The setting is practical:** For example, popular products like Stable Diffusion defaults to adding a fixed watermark to all generated images. Online platforms like Midjourney can add a unique watermark to all images generated by the same user. What's more, these products produce a large number of images everyday. For these popular products, it is possible to obtain tens or hundreds of pre-existing watermarked images.\n\n - **Steganalysis requires fewer images than recent works:** Many watermark detectors like Tree-Ring have demonstrated robustness against JPEG compression. Recent works have explored adversarial attack-based methods to improve watermark removal. While these methods have successfully removed Tree-Ring watermarks, they require training a surrogate model with 3000 to 7500 images [1, 2]. In comparison, steganalysis requires as few as 500 images for effective removal without significant artifacts, as shown in our experiments.\n\n　\n\n> The contribution of the approach may be somewhat limited.\n\nWe argue that our work has made significant contributions. As outlined in our common rebuttal:\n- **We are the first to introduce a black-box, training-free method** that effectively removes and forges Tree-Ring watermarks, which is a SOTA diffusion-based watermarking method.\n- **We offer a new perspective on understanding watermarks** by classifying them and visualizing the patterns of content-agnostic watermarks, providing explainability on why these watermarks are vulnerable.\n- **We highlight that addressing the content-agnostic issue is a future direction** for robust watermarking, as we demonstrated the wide impact of simple steganalysis on several modern detectors, such as Tree-Ring, RAWatermark, and RoSteALS.\n\n　\n\n**References**\n\n[1] M. Saberi et al., Robustness of AI-image detectors: Fundamental limits and practical attacks, 2023.\n\n[2] B. An et al., Benchmarking the robustness of image watermarks, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "65uyx3iOr5", "reviewer": "Reviewer_VkHo", "summary": "This paper introduces a study on the vulnerabilities of digital watermarking techniques to steganalysis attacks. Extensive experiments are conducted to demonstrate the effectiveness of steganalysis in detecting and removing watermarks from images, especially when targeting content-agnostic watermarking methods.", "questions": "Please see Weaknesses.", "limitations": "The authors have addressed the limitations in section 5.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and well-organized. The watermark detection and removal/forgery methods are simple and clear.\n2. The method of estimating the watermark disturbance by subtracting the average of watermarked images from the average of original images is interesting.\n3. The proposed watermark attacks are effective for content-agnostic watermarking methods.", "weaknesses": "1. This paper adopts a simple linear assumption that the watermarked image is obtained by adding the content-agnostic perturbation to the original image. The generality of this assumption needs to be clarified, as it forms the theoretical foundation for the watermark attacks discussed in this paper. For instance, the Tree-Ring method embeds information in the frequency domain of initial noise, but why does it conform to this model?\n\n2. The definition of steganalysis needs to be clearly stated, as it is an important concept in this paper.\n\n3. In section 3.1, the paper mentions that ''Instead of applying strong distortions (e.g., noise), the adversary could take a steganalysis approach, first **approximating w** and then crafting minimal distortions to fool D''. However, in the following method, the watermark is never approximated. Instead, the additive perturbation is directly predicted to fool D.\n\n4. In Figure 2, why are the performance metrics inconsistent? Although the caption specifies the use of AUC, the figure mixes AUC with Bit Acc as metrics for different methods.\n\n5. It is better to explain why there is a sudden drop when n = 5 for content-adaptive methods like Stable Signature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Why Tree-Ring conforms to the additive Simple Linear Assumption model?\n\n1. Addition in spatial domain equates to addition in frequency domain: $x(t)+y(t) \\xleftrightarrow{\\mathcal{F}} X(j\\omega)+Y(j\\omega)$.\n2. Tree-Ring's pattern added to the initial noise propagates through the generation pipeline to the final image (Figure 6).\n3. Thus, Tree-Ring conforms to our model, despite appearing to be a Ring in the frequency domain.\n\n　\n\n> The definition of steganalysis needs to be clearly stated, as it is an important concept in this paper.\n\nWe appreciate the suggestion. Steganalysis is the detection of hidden data within cover media. In our context, it's applied as a technique for watermark removal. We'll clarify this definition in the revised paper.\n\n　\n\n> In section 3.1, the paper mentions that ''Instead of applying strong distortions (e.g., noise), the adversary could take a steganalysis approach, first approximating w and then crafting minimal distortions to fool D''. However, in the following method, the watermark is never approximated. Instead, the additive perturbation is directly predicted to fool D.\n\nYour understanding is correct. It was $\\delta_w$ we approximated (denoted as $\\hat{\\delta}_w$), not $w$. We appreciate this observation and we will revise Section 3.1 to accurately reflect our method.\n\n　\n\n> In Figure 2, why are the performance metrics inconsistent? Although the caption specifies the use of AUC, the figure mixes AUC with Bit Acc as metrics for different methods.\n\nThanks for catching. The caption should be \"Performance (AUC or bit accuracy)\", not just AUC. We will revise.\n\n　\n\n> It is better to explain why there is a sudden drop when n = 5 for content-adaptive methods like Stable Signature.\n\n - **The point at n=5 isn't a drop, but a discontinuity** between no removal (leftmost point) and watermark removal (other points). \n - The lowest performance at n=5 occurs because small n values lead to imperfect $\\delta_w$ approximations, causing larger image modifications and reduced watermark detection. We're happy to provide further clarification if needed.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### Common Response

We appreciate the reviewers' comments and the opportunity for rebuttal. Here we would like to clarify the significance and contributions of our work.

&nbsp;

**Our contributions**
- **We introduce the first blackbox, training-free method** that successfully removed and forged Tree-Ring watermarks. This method integrates watermark removal and forgery into a single operation, enhancing diversity in watermark analysis.
- **We offer a new perspective on understanding watermarks**: By classifying watermarks into content-agnostic and content-adaptive, we provide explainability. We visualize the extracted content-agnostic patterns (such as the ripple-like pattern in Tree-Ring or the vertical bars in DwtDctSvd's $C_B$ chroma). This perspective helps explain why some watermarks are vulnerable to steganalysis.
- **We highlight a future direction for robust watermarking: addressing the content-agnostic issue.** We identify the recurring problem where recent works, despite their methodological complexity, still add fixed patterns without considering the image content. For example, we visualized that a Tree-Ring pattern could propagete through a DDIM diffusion sampling process still being content-agnostic, and watermark decoding relies on this pattern. This allows a sufficiently simple steganalysis to easily deceive several modern watermark detectors, including DNN-involved methods like Tree-Ring, RAWatermark, and RoSteALS. We provide security guidelines, recommending that future researchers include additional evaluations against simple steganalysis attacks, contributing to the security and advancement of watermarking community.

Meta-review: The real average rating of this paper should be 5.75 since reviewer 85eL said that the score should be bumped up to 5 but still remained as 4 (no response after emailing).

The paper introduces a steganalysis-based attack targeting content-agnostic image watermarking methods. Reviewers acknowledge that the paper effectively demonstrates vulnerabilities in these watermarking techniques (aL14, Jv2m, VkHo). However, concerns are raised about the novelty of the method, as similar ideas have been explored in prior works (aL14). Questions arise regarding the generality of the assumptions made, particularly for methods like Tree-Ring that embed watermarks in the frequency domain (Jv2m, VkHo). Practical impact is questioned due to potential image quality degradation after watermark removal (85eL) and the requirement for a large number of pre-existing watermarked images (85eL, aL14). While the paper is well-written and organized (Jv2m, 85eL, VkHo), the contribution is considered limited due to existing similar research and practical challenges in real-world applications.

In the rebuttal the authors address concerns about the novelty of their steganalysis-based attack on content-agnostic watermarks. They emphasize that their method is the first black-box, training-free approach that unifies both watermark removal and forgery without needing access to the watermark encoder or decoder, differing fundamentally from prior works. Additional experiments were provided comparing their method with existing attacks, demonstrating effectiveness with fewer images and minimal perceptual degradation. They also clarify that the vulnerability is severe and not trivial, extending their findings to other modalities like audio. The AC recommends accept as most of the reviewers acknowledged their concerns were addressed and bumped up their ratings.
Decision: Accept (poster)