SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Table 1, we present the main experimental results for Stable Diffusion and the ImageNet model. In the clean setting, all baselines except DwtDct and all Tree-Ring Watermarking variants are strongly detectable. Tree-RingRand and Tree-RingRings show negligible impact on the FID and no impact on the CLIP score.
4.3 Benchmarking Watermark Robustness
To benchmark the robustness of our watermark, we focus on documenting its performance under 6 prevalent data augmentations utilized as attacks. These include  $75^{\circ}$  rotation,  $25\%$  JPEG compression,  $75\%$  random cropping and scaling, Gaussian blur with an  $8 \times 8$  filter size, Gaussian noise with  $\sigma = 0.1$ , and color jitter with a brightness factor of 6. Additionally, we conduct ablation studies to
<sup>3</sup>https://github.com/openai/guided-diffusion
github.com/CompVis/stable-diffusion/blob/main/scripts/txt2img.py#L69
Table 1: Main Results. T@1%F represents TPR@1%FPR. We evaluate watermark accuracy in both benign and adversarial settings. Adversarial here refers to average performance over a battery of image manipulations. An extended version with additional details and standard error estimates can be found in Appendix Table 3.
Model	Method	AUC/T@1%F(Clean)	AUC/T@1%F(Adversarial)	FID ↓	CLIP Score ↑
Stable Diff.	DwtDct	0.974 / 0.624	0.574 / 0.092	25.10 .09	0.362 .000
DwtDctSvd	1.000 / 1.000	0.702 / 0.262	25.01 .09	0.359 .000
FID = 25.29	RivaGAN	0.999 / 0.999	0.854 / 0.448	24.51 .17	0.361 .000
CLIP Score= 0.363	Tree-RingZeros	0.999 / 0.999	0.963 / 0.715	26.56 .07	0.356 .000
Tree-RingRand	1.000 / 1.000	0.918 / 0.702	25.47 .05	0.363 .001
Tree-RingRings	1.000 / 1.000	0.975 / 0.694	25.93 .13	0.364 .000
ImageNet	DwtDct	0.899 / 0.244	0.536 / 0.037	17.77 .01	-
DwtDctSvd	1.000 / 1.000	0.713 / 0.187	18.55 .02	-
RivaGAN	1.000 / 1.000	0.882 / 0.509	18.70 .02	-
FID = 17.73	Tree-RingZeros	0.999 / 1.000	0.921 / 0.476	18.78 .00	-
Tree-RingRand	0.999 / 1.000	0.940 / 0.585	18.68 .09	-
Tree-RingRings	0.999 / 0.999	0.966 / 0.603	17.68 .16	-
Table 2: AUC under each Attack for Stable Diffusion, showing the effectiveness of Tree-RingRings over a number of augmentations. Cr. & Sc. refers to random cropping and rescaling. Additional results for the ImageNet model can be found in Appendix Table 4.
Method	Clean	Rotation	JPEG	Cr. & Sc.	Blurring	Noise	Color Jitter	Avg
DwtDct	0.974	0.596	0.492	0.640	0.503	0.293	0.519	0.574
DwtDctSvd	1.00	0.431	0.753	0.511	0.979	0.706	0.517	0.702
RivaGan	0.999	0.173	0.981	0.999	0.974	0.888	0.963	0.854
Stable Sig.	1.00	0.658	0.989	1.00	0.565	0.731	0.976	0.880
T-RZeros	0.999	0.994	0.984	0.999	0.977	0.877	0.907	0.963
T-RRand	1.00	0.486	0.999	0.971	0.999	0.972	0.994	0.918
T-RRings	1.00	0.935	0.999	0.961	0.999	0.944	0.983	0.975
investigate the impact of varying intensities of these attacks. We report both AUC and TPR@1%FPR in the average case where we average the metrics over the clean setting and all attacks. In all ablation studies, we report the average case.
In Table 1, the baseline methods fail in the presence of adversaries. On the contrary, our methods demonstrate higher reliability in adversarial settings. Among them, Tree-RingRings performs the best under adversarial conditions, a result of our careful watermark pattern design.
Further, we show the AUC for each attack setting in Table 2. Notably, Tree-RingZeros demonstrates high robustness against most perturbations, except for Gaussian noise and color jitter. Similarly, Tree-RingRand is robust in most scenarios but performs poorly when faced with rotation, as expected. Overall, Tree-RingRings delivers the best average performance while offering the model owner the flexibility of multiple different random keys. It is worth noting that the baseline method RivaGan also demonstrates strong robustness in most scenarios, but it is important to highlight that our method is training-free and really "invisible".
4.4 Ablation Experiments
In this section, we undertake exhaustive ablation studies with the Ring pattern on several key hyperparameters to demonstrate the efficacy of Tree-Ring Watermarking. Except for the ablation on attacks, the reported numbers represent averages over all attack scenarios and clean images.
Number of Steps Used for Generation and Detection. A key unknown variable for the model owner at the detection time is the actual number of inference steps used during the generation time.
(a) Ablation on Watermark Radii
Figure 4: Ablation on Watermark Radii and Guidance Scales.
(b) Ablation on Guidance Scales
This factor could potentially impact the precision of the DDIM inversion approximation of the initial noise vector. To scrutinize this, we systematically vary the number of steps for both the generation and detection time. Due to the computational demands of sampling with a high number of inference steps, we employ a total of 400 images for each run.
In Figure 3, we compare AUC across all step combinations. Surprisingly, even with a significant difference between the generation-time and detection-time #steps, the decrease in AUC is minimal when the model owner uses a reasonable number of inference steps for detection without knowledge of the true generation-time steps. This indicates that the DDIM inversion maintains its robustness in approximating the initial noise vector, and is effective for watermark detection irrespective of the exact number of steps employed. Interestingly, we notice a trend where the detection power appears to be slightly stronger with fewer inference steps at detection
Figure 3: Ablation on Number of Generation Steps versus Detection Steps. Detection succeeds independent of the number of DDIM used to generate data.
time or a larger number of inference steps at generation time. This is an advantageous scenario as the model owner now does not actually need to carry out a large number of steps for DDIM inversion, while concurrently, the model owner (or the user) is free to choose the number of generation steps that achieve the best quality [Rombach et al., 2022].
Watermark radii. The radius of injected watermarking patterns is another critical hyperparameter affecting robustness and generation quality. The corresponding results are shown in Figure 4(a). As the watermarking radius increases, the watermark's robustness improves. Nevertheless, there is a trade-off with generation quality. We overall confirm a radius of 16 to provide reasonably low FID while maintaining strong detection power.
Guidance scales. Guidance scale is a hyperparameter that controls the significance of the text condition. Higher guidance scales mean the generation more strictly adheres to the text guidance, whereas lower guidance scales provide the model with greater creative freedom. Optimal guidance scales typically range between 5 and 15 for the Stable Diffusion model we employ. We explore this factor from 2 to 18 in Figure 4(b) and highlight that the strength of the guidance is always unknown during detection time. Although a higher guidance scale does increase the error for DDIM inversion due to the lack of this ground-truth guidance during detection, the watermark remains robust and reliable even at a guidance scale of 18. This is again beneficial for practical purposes, allowing the model owner to keep guidance scale a tunable setting for their users.
(a) Rotation
(b) JPEG Compression
(c) Cropping  $^+$  Scaling
(d) Gaussian Blurring
(e) Gaussian Noise
(f) Color Jitter
Figure 5: Ablation on Different Perturbation Strengths.
Attack strengths. Further, we test out the robustness of Tree-Ring Watermarking under each attack with various attack strengths. As shown in Figure 5, even with extreme perturbations like Gaussian blurring with kernel size 40, Tree-Ring Watermarking can still be reliably detected.
Tree-Ring Watermarking requires the model owner to use DDIM during inference. Today, DDIM is still likely the most popular sampling method due to its economical use of GPU resources and high quality. However, the proposed watermark will need to be adapted to other sampling schemes should DDIM fall out of favor. Further, the proposed watermark is by design only verifiable by the model owner because model parameters are needed to perform the inversion process. This has advantages against adversaries, who cannot perform a white-box attack on the watermark or even verify whether an ensemble of manipulations broke the watermark. However, it also restricts third parties from detecting the watermark without relying on an API. Finally, it is currently not yet clear how large the capacity for multiple keys  $k^{*}$  would be. Would it be possible to assign a unique key to every user of the API?
The effectiveness of the proposed watermark is directly related to the accuracy of the inverse DDIM process. Future work that improves the accuracy of this inversion [Zhang et al., 2023], or utilizes invertible diffusion models as described in Wallace et al. [2022], would also improve watermarking power further.
We propose a new approach to watermarking generative diffusion models using minimal shifts of their output distribution. This leads to watermarks that are truly invisible on a per-sample basis. We describe how to optimally shift, so that the watermark remains detectable even under strong image manipulations that might be encountered in daily usage and handling of generated images.
(a) W/o Watermark
(b) DwtDct
(c) RivaGAN
Figure 6: Various watermarked generations with the same random seed are presented, showcasing the "invisible" nature of our proposed watermark. A zoomed-in view with high contrast is provided in the bottom right corner. For more high-resolution watermarked images, please refer to Appendix Figure 8.
(d) Tree-Ring (Ours)
Figure 7: Histogram of the array  $x_{T}^{\prime}$  obtained for a natural image, which is Gaussian.
Table 3: Main Results with Error Bars. T@1%F represents TPR@1%FPR. We evaluate watermark accuracy in both benign and adversarial settings. Adversarial here refers to average performance over a battery of image manipulations.
Model	Method	AUC/T@1%F(Clean)	AUC/T@1%F(Adversarial)	FID ↓	CLIP Score ↑
Stable Diff.	DwtDct	0.974.001 / 0.624.013	0.574.005 / 0.092.004	25.10.09	0.362.000
DwtDctSvd	1.000.000 / 1.000.000	0.702.000 / 0.262.011	25.01.09	0.359.000
FID = 25.29	RivaGAN	0.999.000 / 0.999.000	0.854.002 / 0.448.006	24.51.17	0.361.000
CLIP Score= 0.363	T-RZeros	0.999.000 / 0.999.000	0.963.001 / 0.715.021	26.56.07	0.356.000
T-RRand	1.000.000 / 1.000.000	0.918.005 / 0.702.017	25.47.05	0.363.001
T-RRings	1.000.000 / 1.000.000	0.975.001 / 0.694.018	25.93.13	0.364.000
ImageNet	DwtDct	0.899.040 / 0.244.203	0.536.016 / 0.037.029	17.77.01	-
DwtDctSvd	1.000.000 / 1.000.000	0.713.019 / 0.187.008	18.55.02	-
RivaGAN	1.000.000 / 1.000.000	0.882.010 / 0.509.009	18.70.02	-
FID = 17.73	T-RZeros	0.999.000 / 1.000.000	0.921.000 / 0.476.000	18.78.00	-
T-RRand	0.999.000 / 1.000.000	0.940.004 / 0.585.006	18.68.09	-
T-RRings	0.999.000 / 0.999.000	0.966.005 / 0.603.006	17.68.16	-
Table 4: AUC under each Attack for the ImageNet model, showing the effectiveness of Tree-RingRings over a number of augmentations. Cr. & Sc. refers to random cropping and rescaling.
Method	Clean	Rotation	JPEG	Cr. & Sc.	Blurring	Noise	Color Jitter	Avg
DwtDct	0.899	0.478	0.522	0.433	0.512	0.365	0.538	0.536
DwtDctSvd	1.000	0.669	0.568	0.614	0.947	0.656	0.535	0.713
RivaGan	1.000	0.321	0.978	0.999	0.988	0.962	0.924	0.882
T-RZeros	0.999	0.953	0.806	0.997	0.999	0.938	0.775	0.921
T-RRand	0.999	0.682	0.962	0.997	0.999	0.986	0.956	0.940
T-RRings	0.999	0.975	0.940	0.994	0.999	0.979	0.861	0.966
Table 5: Additional experiments about watermark capacity for multiple keys. We employ a Bonferroni correction for multiple hypotheses and set the threshold at  $\mathrm{{FPR}} = 1{\mathrm{e}}^{-6}$  . The identification accuracy is presented.
Method	50 users/keys	100 users/keys	500 users/keys	1000 users/keys
T-RRand	1.00	1.00	1.00	1.00
T-RRings	0.99	0.99	0.98	0.98
W/o Watermark
Tree-RingZeros
Tree-RingRand
Tree-RingRings
Figure 8: More generated images with Tree-Ring Watermarking with the first 7 prompts in MS-COCO-2017 training dataset.
Figure 9: Results on  $k$  number of random attacks applied at the same time.
(a)  $75^{\circ}$  Rotation
(b)  $25\%$  JPEG Compression
(c)  $75\%$  Random Cropping + Scaling
(d) Gaussian Blurring with an  $8\times$  8 Filter
(e) Gaussian Noise with  $\sigma = 0.1$
Figure 10: Attacked images.
(f) Color Jitter with a Brightness Factor of 6