Title: FreqMark: Invisible Image Watermarking via Frequency Based Optimization in Latent Space
Venue: NeurIPS 2024
Accepted: yes
Keywords: deep watermarking

latent frequency optimization
Abstract: Invisible watermarking is essential for safeguarding digital content, enabling copyright protection and content authentication. 
However, existing watermarking methods fall short in robustness against regeneration attacks.
In this paper, we propose a novel method called FreqMark that involves unconstrained optimization of the image latent frequency space obtained after VAE encoding. Specifically, FreqMark embeds the watermark by optimizing the latent frequency space of the images and then extracts the watermark through a pre-trained image encoder. This optimization allows a flexible trade-off between image quality with watermark robustness and effectively resists regeneration attacks.
Experimental results demonstrate that FreqMark offers significant advantages in image quality and robustness, permits flexible selection of the encoding bit number, and achieves a bit accuracy exceeding 90\% when encoding a 48-bit hidden message under various attack scenarios.

FreqMark: Invisible Image Watermarking via Frequency Based Optimization in Latent Space

Abstract

Invisible watermarking is essential for safeguarding digital content, enabling copyright protection and content authentication. However, existing watermarking methods fall short in robustness against regeneration attacks. In this paper, we propose a novel method called FreqMark that involves unconstrained optimization of the image latent frequency space obtained after V AE encoding. Specifically, FreqMark embeds the watermark by optimizing the latent frequency space of the images and then extracts the watermark through a pre-trained image encoder. This optimization allows a flexible trade-off between image quality with watermark robustness and effectively resists regeneration attacks. Experimental results demonstrate that FreqMark offers significant advantages in image quality and robustness, permits flexible selection of the encoding bit number, and achieves a bit accuracy exceeding 90% when encoding a 48-bit hidden message under various attack scenarios.

1 Introduction

As the development of generative models [42, 32], distinguishing between AI-generated and real images becomes increasingly challenging, which brings new risks such as deepfakes and copyright infringement [5, 31]. By adding invisible watermarks within images, the concealed message can be tracked and used for purposes such as copyright verification, identity authentication, copy control, etc., thereby safeguarding against the misuse of image content.

Traditional methods [12, 15, 14, 36, 15] conceal hidden messages in the frequency space of images, providing resistance to Gaussian noise attacks but susceptibility to brightness, contrast, and regeneration attacks. The rapid development of deep learning has also promoted the iteration of watermarking techniques with enhanced robustness to numerous attacks. A common approach is to train a watermark embedding network through supervised learning to introduce imperceptible perturbations to images, and then to retrieve the hidden messages through a decoding network [46, 10, 55, 3]. An alternative method sets the optimization objective as the image itself, utilizing pre-trained neural networks to calculate the perturbations to be added, thereby bypassing the higher cost of training networks and providing a more flexible approach [29, 20]. However, with the continued advancement of generative models, leveraging their generalization capabilities to denoise watermarked images through regeneration attacks has proven to be an effective method for watermark removal [58, 43]. A logical step is to encode the watermark in the image latent space, allowing the watermark to

'

Watermark Message Encoding in

0

FFT

•

Regeneration

Attack

Attack

fQa#ek  •

'

Watermark Message Encoding in

Image Latent Space

!

Regeneration

Attack

Attack

Hli;o!

A?sce3 .

Watermark Message Encoding in

Image Latent Frequency Space

0

FFT

'

!

•

Regeneration

Attack

Attack

Encoder

Encoder

•

•

•

•

be incorporated into the latent semantics of images. This approach improves robustness against regeneration attacks but increases susceptibility to Gaussian noise.

In this paper, we propose FreqMark, a novel self-supervised watermarking approach that amalgamates the benefits of frequency domain space and latent space, endowing it with a dual-domain advantage. Specifically, it utilizes fixed pre-trained Variational Autoencoder (V AE) [28] to embed watermark messages into images by making subtle adjustments in the latent frequency space. These messages are then decoded using a fixed pre-trained image encoder. Figure 1 illustrates the primary motivation behind our method. Introducing perturbations to watermark images in the latent and frequency domains offers distinct advantages. By optimizing the latent frequency domain of the image, we combine both approaches to effectively leverage their strengths, achieving a synergistic effect where the whole is greater than the sum of its parts.

We evaluate the performance of FreqMark on the DiffusionDB [49] and ImageNet [16] datasets, experimental results demonstrated that FreqMark achieves strong robustness while maintaining image quality, configurable payload capacity, and flexibility. With a 48-bit encoding setting, the bit accuracy can exceed 90% under various attacks. This performance indicates significant advantages over baseline methods, particularly excelling during regeneration attacks [7, 13, 58].

Contributions: (1) We propose a novel invisible image watermarking method named FreqMark, which encodes hidden messages within the latent frequency space of images. FreqMark achieves watermark embedding through indirect optimization centered on the image itself without requiring network training. (2) FreqMark is highly flexible, allowing for a free trade-off between the bits number of the encoded message, image quality and watermark robustness to meet diverse requirements. (3) FreqMark demonstrates significant robustness advantages, particularly during regeneration attacks compared to baseline methods. Experimental results validate the superiority of our proposed method.

2 Related Work

Generative Models For a long time, Generative Adversarial Networks (GANs) [27, 1, 26, 22, 41, 33] have dominated image generation. Recently, diffusion models have emerged as a solid alternative to GANs for image generation [25, 17, 34, 35]. These models show significant improvements and applications across various domains. DDIM sampling [45] and latent diffusion [42] further accelerate the generation progress, while ControlNet [56] provides a powerful, controllable generation method. As high-quality image generation becomes more accessible, digital watermarking gains importance for protecting intellectual property rights and ensuring content authenticity.

Image Watermarking The research history of image watermarking techniques is extensive. Early methods employ hand-crafted methods to hide messages within the spatial or frequency domain of images. Frequency-domain-based techniques typically exhibit better robustness and have been widely applied even before the rise of deep learning [12, 15, 14, 36, 15, 4]. The widely used open-source

model, Stable Diffusion [42], employs DwtDctSvd [14] as its default watermarking method. However, this method has been demonstrated to be relatively vulnerable to various attacks [58].

With the development of deep learning, methods utilizing encoder-decoder architectures have gained prominence. HiDDeN [60] and RivaGAN [55] train encoders to embed watermark messages into images and decoders to extract them, thereby enhancing robustness against noise while preserving image quality. RedMark [3] and StegaStamp [46] improve robustness by integrating a series of differentiable perturbations. RoSteALS [10] enhances the robustness of the watermark by fine-tuning the secret encoder and secret decoder in the latent space. In recent years, several innovative methods have been introduced. WatermarkDM [59] trains a diffusion model on watermarked images to create detectable watermarked images. Stable Signature [19] fine-tunes the decoder of a latent diffusion model to embed specific hidden messages. Tree-Ring [50] adopts a unique approach by encoding a particular pattern shape in the initial noise frequency space during the diffusion process and utilizes DDIM inversion [45] to detect watermarks. Differently, FNNS [29] and SSL [20] achieve message encoding by optimizing the image itself. However, the aforementioned methods struggle to strike a perfect balance between flexibility and robustness. In contrast, FreqMark offers higher degrees of freedom, allowing for a better trade-off among task requirements, and demonstrates robustness against regeneration attacks.

3 Background

FreqMark operates on the following scenario: A user embeds a k -bit watermark message m e into an image I to get the watermarked image I w for which they hold the copyright. Upon discovering unauthorized usage of the image I w , the user decodes the infringed image to obtain the message m d , which serves as evidence to prove their ownership of the image's copyright.

Assuming that each bit of the decoded message from clean images is independent and has an equal probability of being -1 or 1, this method allows us to mathematically calculate the False Positive Rate (FPR) of decoding.

Let the encoded message be m e ∈ {-1 , 1 } k and the decoded message be m d ∈ {-1 , 1 } k . The function M ( m e , m d ) measures the number of matching bits between m e and m d . Given the assumption regarding the image encoder output mentioned earlier, each bit of the decoded message from clean images is independent and follows Bernoulli random variables with a probability of 0.5 [19]. By set a decoding threshold τ ∈ { 0 , . . . , k } , once M ( m e , m d ) ≥ τ , we consider that the image has encoded the message m e . Consequently, M ( m e , m d ) follows a binomial distribution B ( k, 0 . 5) . The final message will be transformed to { 0 , 1 } k by applying function f ( x ) = ( x +1) / 2 for easier processing of the binary format.

We could test the hypothesis H 1 : the image x has hidden watermark , and against the null hypothesis H 0 : the image x has no hidden watermark . From this, we can obtain a closed-form solution for the FPR under the threshold ϵ ( τ ) using the regularized incomplete beta function I x ( a ; b ) :

Based on the above formula, FPR ≈ 1 . 65 × 10 -6 when k = 48, τ = 39; FPR ≈ 5 . 04 × 10 -8 when k = 48, τ = 41. The results of watermark detection True Positive Rate (TPR) under various FPR settings are shown in Figure 8 of Appendix A.3.

4 Method

4.1 Overview

Figure 2 presents an overview of FreqMark. FreqMark employs a strategic methodology to embed invisible watermarks in images by adding perturbations in the latent frequency space. Specifically, we utilize a Variational Autoencoder (VAE) [28] to encode images into representations and then transform these image latents into the frequency domain. Only the perturbation within the latent frequency space of images is trained during watermark encoding and all the networks are fixed. A

'

I

I

I

I

I

Encoding

I

VAE Encoder

, --------------------------'

Vector 1

Vector 2

Vector 3

Vector K

Pre-defined

DotProduc~

•

~◄--1111-

Image

Encoder

I

I

I

I  1111

I

I

I

/

t

I

w

I

I

0

FFT

-

1111

VAE Decoder

I

I

I

I

I

1  o

1 1

1 1

1

o  1 1  1

o  I

o  11  1

.

..

i_ -

-

t

-

-

-

Ez

I

Decoding

/

•

I

I

I

I

0

iFFT

8.

m

Fz

'

~-----------------------;

pre-trained image encoder is used for watermark extraction, the features of the watermarked image obtained from the encoder are compared with predefined directional vectors to reveal the hidden message.

In the optimization process, the image quality is maintained with minimal degradation by utilizing PSNR and LPIPS loss [57], while the watermark message is constrained by using hinge loss. In addition, noises and augmentation are introduced for enhanced robustness. Figure 3 shows some watermark image examples from FreqMark.

4.2 Message Embed in Latent Frequency Space

Some current methods embed hidden messages by directly optimizing image pixels [20, 29], which leads to poor robustness against regeneration attacks [58]. FreqMark involves strategically adding perturbations in the frequency domain of the image latent representation to embed an invisible watermark. These perturbations in the frequency domain are more concealed than those in the pixel space, making them harder to eliminate and thus offering superior robustness against regeneration attacks while minimizing the impact on image quality.

First, we use a pre-trained V AE encoder E to encode the image into a latent image and then transform the latent image into the frequency domain using Fast Fourier Transform (FFT), represented as:

where I is the input image, E ( I ) is the latent representation encoded by the V AE encoder E , and F Z is the frequency domain latent representation of the image.

Next, we embed the hidden message into the frequency domain latent representation F Z by adding a slight perturbation, then decode the frequency domain latent representation back into the image using inverse Fast Fourier Transform (iFFT) FFT -1 and the VAE decoder D , as follows:

where I w is the watermarked image, δ m is the perturbation added in the frequency domain of the latent image for embedding the watermark.

4.3 Decoding by Pre-trained Image Encoder

During the decoding process, similar to SSL[20], we predefine a set of K N-dimensional vectors V N K = { v 1 , v 2 , ..., v K | K ≤ N } within the feature space of the pre-trained image encoder E img . For images with embedded watermarks I w , the feature vector z I w = E img ( I w ) is obtained after passing through E img . By calculating the signs of the dot product between the direction vectors V N K and z I w , we can extract the hidden message m d :

where m k d represents the k -th bit of m d and v k denotes the k -th direction vector of V N K , sign ( x ) = 1 when x > = 0 ; sign ( x ) = -1 when x < 0 .

One significant advantage of this decoding method is its flexibility in setting the number of watermark bits. The number of direction vectors determines the encoding bits. Using the technique mentioned above of optimizing images within the frequency domain, we can ensure the robustness of the hidden messages while significantly minimizing the quality impact on the image.

4.4 Training Objective

Our goal is to embed an invisible watermark by optimizing the frequency map of the image latent F Z , where the perturbation δ m serves as the trainable parameter, with all pre-trained networks remaining fixed. The optimization via perturbations ensures both watermark robustness and the preservation of image quality, offering flexibility in encoding bits and embedding strength.

Image Quality We utilize PSNR loss to reduce discrepancies between the watermarked image and the original image and also incorporate LPIPS loss [57] to make alterations less perceptible.

Watermark Message The optimization goal is to modify the image features z I w processed by the pre-trained image encoder E img , aligning them on the K direction vectors V N K to correspond with the encoded message. We define message loss as the hinge loss with margin µ ≥ 0 on the projections:

4.5 Robustness Enhancement Strategy

During the training process, we employ an augmentation strategy by adding Gaussian noise with a mean of 0 and standard deviations of s 1 and s 2 to the latent space and pixel space, represented as ϵ 1 and ϵ 2 , respectively. The result is the acquisition of the perturbed images I p 1 and I p 2 :

The final loss is defined as:

where λ p and λ i are the respective weights for each loss function.

By incorporating corresponding attacks during training, we optimize the solution space for hidden message embedding, enabling the image to withstand a broader range of attacks. Experimental results reveal that this straightforward approach can effectively enhance the model's robustness against regeneration attacks.

中

5 Experiments

In this section, we evaluate our method based on the image quality and robustness metrics under various attacks, comparing it with the baseline methods. Additionally, ablation studies and analyses are carried out to explore the process deeply. In our experiments, KL auto-encoder [28] from Stable Diffusion [42] is used as the pre-trained V AE, and DINO v2 small [37] is used as the pre-trained image encoder. More implementation details are in the Appendix A.1.

5.1 Experimental Setup

Datasets Atest dataset is compiled, consisting of 500 images randomly selected from the ImageNet validation set [16], in conjunction with 500 images generated using Stable Diffusion [42] based on prompts from the DiffusionDB [49] dataset. This diverse data collection enables a thorough assessment of the performance across various scenarios and perspectives.

Comparison Methods Three methodologies are selected for comparison. DwtDctSvd [14] is a classic frequency-domain-based approach and the default watermarking method in Stable Diffusion [42]. Stable Signature [19] embeds specific hidden messages by fine-tuning the VAE decoder of Stable Diffusion, enabling the watermarking process to be integrated with the generation process. SSL [20] is the latest method that employs image pixel optimization to embed watermarks. We employ their default configurations, wherein the PSNR threshold of SSL is set to 31 dB for a convenient comparison of bit accuracy.

Evaluation Metrics Following previous work [19, 20, 58], PSNR and SSIM [48] are used as the image quality benchmark. To evaluate robustness, we utilize bit accuracy as a metric to measure the degradation of hidden messages under diverse attacks. Various attack methods have been implemented, including a brightness change of 0.5, a contrast change of 0.5, 50% JPEG compression, Gaussian blur with a kernel size of 5, and Gaussian noise with σ = 0.05. Moreover, the experiments incorporate two types of V AE regeneration attacks [7, 13] from the CompressAI library [9] with a compression factor of 3, and a diffusion regeneration attack is carried out with 60 denoising steps [58].

5.2 Benchmarking Watermark Accuracy and Image Quality

Table 1 displays the image quality and bit accuracy following watermark embedding on ImageNet [16] and DiffusionDB [49] datasets. The classic method DwtDctSvd [14] shows robustness against Gaussian attacks but poor performance against others, while the others show better robustness against

brightness, contrast, and JPEG attacks but still poor in regeneration attacks. FreqMark demonstrates exceptional robustness against regeneration attacks with acceptable image quality.

We also calculate the PSNR and SSIM of the images from two datasets after VAE reconstruction. The data in Table 2 demonstrates that the impact of FreqMark on image quality is limited. Moreover, the standard deviation of image quality after FreqMark processing is also reduced to some extent.

In order to verify the hypothesis proposed in Section 3, we randomly select 2,000 clean images to perform message decoding in 128-bit and calculate the correlation coefficients between each bit, comparing them to random messages. Figure 4 shows that different bits with near-zero correlation coefficients can be considered as independent random variables.

Notably, FreqMark allows users to adjust image quality and encoding bits, showing remarkable robustness while maintaining acceptable image quality, particularly against VAE and diffusion regeneration attacks [9, 7, 13, 58]. We will further evaluate the robustness of FreqMark on the DiffusionDB dataset [49] in Section 5.4. The ablation studies in Section 5.5 will explore the impact of parameter adjustments on robustness.

5.3 Why the Frequency Domain of Image Latent Space

To further illustrate the advantages of optimizing in the frequency domain of the image latent space, We conducted experiments using the same settings to optimize the pixel, the image latent space, and the frequency domain of image pixels in the DiffusionDB dataset [49]. Simultaneously, enhancements in pixel and latent domains are retained (if the method involves the latent space). A consistent average image quality range is maintained for comparison purposes.

As illustrated in Figure 5 and Table 3, optimizing the frequency domain of image pixels provides the watermarked image with robustness against all attacks except the regeneration attack. In contrast,

Origin

Pixel Frequency

Latent

Latent Frequency

images obtained by optimizing the image latent space appear more natural and smooth, with a strong correlation to the semantic information of the image. This approach demonstrates robust potential against diffusion attacks, indicating that both optimization methods have unique features. Combining the advantages of both approaches, FreqMark optimizes the frequency space of the image latent. The watermark is closely related to local patterns while retaining the characteristics of frequency domain optimization. Combining these two aspects produces a synergistic effect, making FreqMark robust against regeneration attacks.

5.4 Further Robustness Results

Diffusion Attack Steps We evaluate the impact of diffusion attacks of varying intensities on bit accuracy and also calculate the PSNR between the attacked image and the original watermarked image. Table 4 indicates that FreqMark maintains commendable robustness even under higher intensity attacks.

Vae Attack Strength We employ the same VAE used in the watermarking process of FreqMark to conduct perturbation attacks. Table 5 demonstrates that FreqMark exhibits exceptional robustness under such attacks. This is because the perturbation on the latent FFT changes the overall distribution of the image latent, making the watermark message affect the entire image globally. Therefore, it

is difficult for perturbations on the latent of image to damage the watermark message. A similar phenomenon is observed in the pixel FFT when facing the Gaussian Noise attack.

Adversarial Attack Following the settings in [5], we apply adversarial attacks targeting the latent representations of the watermarked images.

Table 6 demonstrates that FreqMark exhibits strong robustness when facing adversarial attacks targeting latent representations. We believe that this can be attributed to the limited impact of attacks targeting latent representations on the latent FFT domain.

5.5 Ablation Studies

Image Quality As watermarking always requires modifications to the image pixels, a trade-off between image quality and embedded message robustness is inevitable. FreqMark allows users to find a satisfactory balance. We control watermarked image quality within a specific range by adjusting the weight of the PSNR loss function. As shown in Figure 6a, FreqMark can achieve a PSNR that is nearly 2dB higher compared to the reconstruction obtained using VAE with robustness against varying attacks, maintaining strong performance with over 80% accuracy.

Encoding Bits The number of bits in the encoding message significantly impacts robustness, with more bits embedded at a given image quality potentially reducing robustness. FreqMark allows users to adjust the watermark bit number based on their needs. Figure 6b shows bit accuracy for encoding bit numbers. We test robustness from 32 to 128 bits of watermark message in 16-bit increments, setting the PSNR of the watermarked image to about 31 dB. FreqMark also demonstrates strong robustness, with the lowest bit accuracy remaining above 0.75 even under regeneration attacks on images watermarked with 128-bit messages.

Noising Scale We also study the impact of latent noise ϵ 1 and pixel noise ϵ 2 on robustness. Incorporating noise attacks during training can effectively improve robustness in Gaussian noise and regeneration attacks, but excessive noise may disrupt training and reduce performance. We analyze how noises in latent and pixel space affect robustness by fixing either ϵ 1 or ϵ 2 and changing the other standard deviation. As shown in Figure 6c and Figure 6d, the results confirm that introducing both types of noise positively influences robustness, and adding an appropriate amount of noise can significantly enhance resistance against regeneration attacks. We finally set ϵ 1 to 0.25 and ϵ 2 to 0.06 based on the experimental results. The complete experimental data can be found in Table 11 of the Appendix A.3.

Spacial Perturbations The bit accuracy of FreqMark significantly declines under specific spatial transformation attacks, such as rotation and cropping. By incorporating relevant augmentations like

(a) Image Quality

(b) Encoding Bits

random rotation and random cropping during training, we effectively bolster the robustness against these transformations, as evidenced in Table 7.

6 Conclusion

In this paper, we introduce a technique for embedding hidden messages in images by optimizing the frequency domain in the latent space, named FreqMark, providing remarkable robustness due to deeper perturbations. FreqMark allows user-defined encoding bits and watermark strength, striking an optimal balance between image quality and robustness.

Limitations and Broader Impacts FreqMark exhibits notable flexibility and robustness, yet there is room for optimization, particularly in the degree of naturalness in the fusion of watermarks and images. Future research can focus on improving image quality and human perceptibility constrained by the limitations of V AE, as a superior V AE would likely boost processing time and overall effectiveness. FreqMark presents an innovative image watermarking approach for image provenance, copyright protection, etc. However, like other image watermarking methods, it also necessitates the prevention of unauthorized misuse such as copyright abuse [5].

References

ABDAL, R., QIN, Y., AND WONKA, P. Image2stylegan: How to embed images into the stylegan latent space? In 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (2019), IEEE Computer Society, pp. 4431-4440.

ABDELNABI, S., AND FRITZ, M. Adversarial watermarking transformer: Towards tracing text provenance with data hiding. In 2021 IEEE Symposium on Security and Privacy (SP) (2021), IEEE, pp. 121-140.

AHMADI, M., NOROUZI, A., KARIMI, N., SAMAVI, S., AND EMAMI, A. Redmark: Framework for residual diffusion watermarking based on deep networks. Expert Systems with Applications 146 (2020), 113157.

AL-HAJ, A. Combined dwt-dct digital image watermarking. Journal of computer science 3 , 9 (2007), 740-746.

AN, B., DING, M., RABBANI, T., AGRAWAL, A., XU, Y., DENG, C., ZHU, S., MOHAMED, A., WEN, Y., GOLDSTEIN, T., ET AL. Benchmarking the robustness of image watermarks. arXiv preprint arXiv:2401.08573 (2024).

ATHALYE, A., ENGSTROM, L., ILYAS, A., AND KWOK, K. Synthesizing robust adversarial examples. In International conference on machine learning (2018), PMLR, pp. 284-293.

BALLÉ, J., MINNEN, D., SINGH, S., HWANG, S. J., AND JOHNSTON, N. Variational image compression with a scale hyperprior. arXiv preprint arXiv:1802.01436 (2018).

BARDES, A., PONCE, J., AND LECUN, Y. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906 (2021).

BÉGAINT, J., RACAPÉ, F., FELTMAN, S., AND PUSHPARAJA, A. Compressai: a pytorch library and evaluation platform for end-to-end compression research. arXiv preprint arXiv:2011.03029 (2020).

BUI, T., AGARWAL, S., YU, N., AND COLLOMOSSE, J. Rosteals: Robust steganography using autoencoder latent space. In Proc. CVPR WMF (2023).

CHAI, L., BAU, D., LIM, S.-N., AND ISOLA, P. What makes fake images detectable? understanding properties that generalize. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVI 16 (2020), Springer, pp. 103-120.

CHANG, C.-C., TSAI, P., AND LIN, C.-C. Svd-based digital image watermarking scheme. Pattern Recognition Letters 26 , 10 (2005), 1577-1586.

CHENG, Z., SUN, H., TAKEUCHI, M., AND KATTO, J. Learned image compression with discretized gaussian mixture likelihoods and attention modules. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2020), pp. 7939-7948.

COX, I., MILLER, M., BLOOM, J., FRIDRICH, J., AND KALKER, T. Digital watermarking and steganography . Morgan kaufmann, 2007.

COX, I. J., KILIAN, J., LEIGHTON, T., AND SHAMOON, T. Secure spread spectrum watermarking for images, audio and video. In Proceedings of 3rd IEEE international conference on image processing (1996), vol. 3, IEEE, pp. 243-246.

DENG, J., DONG, W., SOCHER, R., LI, L.-J., LI, K., AND FEI-FEI, L. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition (2009), Ieee, pp. 248-255.

DHARIWAL, P., AND NICHOL, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems 34 (2021), 8780-8794.

FEI, J., XIA, Z., TONDI, B., AND BARNI, M. Supervised gan watermarking for intellectual property protection. In 2022 IEEE International Workshop on Information Forensics and Security (WIFS) (2022), IEEE, pp. 1-6.

FERNANDEZ, P., COUAIRON, G., JÉGOU, H., DOUZE, M., AND FURON, T. The stable signature: Rooting watermarks in latent diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (2023), pp. 22466-22477.

FERNANDEZ, P., SABLAYROLLES, A., FURON, T., JÉGOU, H., AND DOUZE, M. Watermarking images in self-supervised latent spaces. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2022), IEEE, pp. 3054-3058.

FRANK, J., EISENHOFER, T., SCHÖNHERR, L., FISCHER, A., KOLOSSA, D., AND HOLZ, T. Leveraging frequency analysis for deep fake image recognition. In International conference on machine learning (2020), PMLR, pp. 3247-3258.

GOODFELLOW, I., POUGET-ABADIE, J., MIRZA, M., XU, B., WARDE-FARLEY, D., OZAIR, S., COURVILLE, A., AND BENGIO, Y. Generative adversarial nets. Advances in neural information processing systems 27 (2014).

GRAGNANIELLO, D., COZZOLINO, D., MARRA, F., POGGI, G., AND VERDOLIVA, L. Are gan generated images easy to detect? a critical analysis of the state-of-the-art. In 2021 IEEE international conference on multimedia and expo (ICME) (2021), IEEE, pp. 1-6.

GUARNERA, L., GIUDICE, O., AND BATTIATO, S. Deepfake detection by analyzing convolutional traces. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops (2020), pp. 666-667.

HO, J., JAIN, A., AND ABBEEL, P. Denoising diffusion probabilistic models. Advances in neural information processing systems 33 (2020), 6840-6851.

KARRAS, T., AILA, T., LAINE, S., AND LEHTINEN, J. Progressive growing of gans for improved quality, stability, and variation. In International Conference on Learning Representations (2018).

KARRAS, T., LAINE, S., AITTALA, M., HELLSTEN, J., LEHTINEN, J., AND AILA, T. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2020), pp. 8110-8119.

KINGMA, D. P., AND WELLING, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013).

KISHORE, V., CHEN, X., WANG, Y., LI, B., AND WEINBERGER, K. Q. Fixed neural network steganography: Train the images, not the network. In International Conference on Learning Representations (2021).

LEE, J.-E., SEO, Y.-H., AND KIM, D.-W. Convolutional neural network-based digital image watermarking adaptive to the resolution of image and watermark. Applied Sciences 10 , 19 (2020), 6854.

LUKAS, N., DIAA, A., FENAUX, L., AND KERSCHBAUM, F. Leveraging optimization for adaptive attacks on image watermarks. In The Twelfth International Conference on Learning Representations (2023).

MIDJOURNEY.COM. Midjourney. Accessed 3 April 2024. https://www.midjourney.com, 2022.

MIRZA, M., AND OSINDERO, S. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014).

NICHOL, A., DHARIWAL, P., RAMESH, A., SHYAM, P., MISHKIN, P., MCGREW, B., SUTSKEVER, I., AND CHEN, M. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 (2021).

NICHOL, A. Q., AND DHARIWAL, P. Improved denoising diffusion probabilistic models. In International conference on machine learning (2021), PMLR, pp. 8162-8171.

Ó RUANAIDH, J., DOWLING, W., AND BOLAND, F. Watermarking digital images for copyright protection. IEE PROCEEDINGS VISION IMAGE AND SIGNAL PROCESSING 143 (1996), 250-256.

OQUAB, M., DARCET, T., MOUTAKANNI, T., VO, H., SZAFRANIEC, M., KHALIDOV, V., FERNANDEZ, P., HAZIZA, D., MASSA, F., EL-NOUBY, A., ET AL. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023).

O'RUANAIDH, J. J., AND PUN, T. Rotation, scale and translation invariant digital image watermarking. In Proceedings of International Conference on Image Processing (1997), vol. 1, IEEE, pp. 536-539.

PARMAR, G., ZHANG, R., AND ZHU, J.-Y. On aliased resizing and surprising subtleties in gan evaluation. In CVPR (2022).

QIAO, T., MA, Y., ZHENG, N., WU, H., CHEN, Y., XU, M., AND LUO, X. A novel model watermarking for protecting generative adversarial network. Computers & Security 127 (2023), 103102.

RADFORD, A., METZ, L., AND CHINTALA, S. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015).

ROMBACH, R., BLATTMANN, A., LORENZ, D., ESSER, P., AND OMMER, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2022), pp. 10684-10695.

SABERI, M., SADASIVAN, V. S., REZAEI, K., KUMAR, A., CHEGINI, A., WANG, W., AND FEIZI, S. Robustness of ai-image detectors: Fundamental limits and practical attacks. In The Twelfth International Conference on Learning Representations (2023).

SHI, C., HOLTZ, C., AND MISHNE, G. Online adversarial purification based on selfsupervision. arXiv preprint arXiv:2101.09387 (2021).

SONG, J., MENG, C., AND ERMON, S. Denoising diffusion implicit models. In International Conference on Learning Representations (2020).

TANCIK, M., MILDENHALL, B., AND NG, R. Stegastamp: Invisible hyperlinks in physical photographs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2020), pp. 2117-2126.

VUKOTI ´ C, V., CHAPPELIER, V., AND FURON, T. Are deep neural networks good for blind image watermarking? In 2018 IEEE International Workshop on Information Forensics and Security (WIFS) (2018), IEEE, pp. 1-7.

WANG, Z., BOVIK, A. C., SHEIKH, H. R., AND SIMONCELLI, E. P. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13 , 4 (2004), 600-612.

WANG, Z. J., MONTOYA, E., MUNECHIKA, D., YANG, H., HOOVER, B., AND CHAU, D. H. Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models. arXiv preprint arXiv:2210.14896 (2022).

WEN, Y., KIRCHENBAUER, J., GEIPING, J., AND GOLDSTEIN, T. Tree-rings watermarks: Invisible fingerprints for diffusion images. In Advances in Neural Information Processing Systems (2023), A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36, Curran Associates, Inc., pp. 58047-58063.

WU, H., LIU, G., YAO, Y., AND ZHANG, X. Watermarking neural networks with watermarked images. IEEE Transactions on Circuits and Systems for Video Technology 31 , 7 (2020), 25912601.

YOON, J., HWANG, S. J., AND LEE, J. Adversarial purification with score-based generative models. In International Conference on Machine Learning (2021), PMLR, pp. 12062-12072.

YU, N., SKRIPNIUK, V., CHEN, D., DAVIS, L., AND FRITZ, M. Responsible disclosure of generative models using scalable fingerprinting. arXiv preprint arXiv:2012.08726 (2020).

ZHANG, C., BENZ, P., KARJAUV, A., SUN, G., AND KWEON, I. S. Udh: Universal deep hiding for steganography, watermarking, and light field messaging. Advances in Neural Information Processing Systems 33 (2020), 10223-10234.

ZHANG, K. A., XU, L., CUESTA-INFANTE, A., AND VEERAMACHANENI, K. Robust invisible video watermarking with attention. arXiv preprint arXiv:1909.01285 (2019).

ZHANG, L., RAO, A., AND AGRAWALA, M. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (2023), pp. 3836-3847.

ZHANG, R., ISOLA, P., EFROS, A. A., SHECHTMAN, E., AND WANG, O. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition (2018), pp. 586-595.

ZHAO, X., ZHANG, K., SU, Z., VASAN, S., GRISHCHENKO, I., KRUEGEL, C., VIGNA, G., WANG, Y., AND LI, L. Invisible image watermarks are provably removable using generative ai. Saastha Vasan, Ilya Grishchenko, Christopher Kruegel, Giovanni Vigna, Yu-Xiang Wang, and Lei Li,'Invisible image watermarks are provably removable using generative ai, ' Aug (2023).

ZHAO, Y., PANG, T., DU, C., YANG, X., CHEUNG, N.-M., AND LIN, M. A recipe for watermarking diffusion models. arXiv preprint arXiv:2303.10137 (2023).

ZHU, J., KAPLAN, R., JOHNSON, J., AND FEI-FEI, L. Hidden: Hiding data with deep networks. In Proceedings of the European conference on computer vision (ECCV) (2018), pp. 657-672.

A Appendix

A.1 Implement details

Hyperparameters The KL auto-encoder from Stable Diffusion 2-1 [42] is utilized. Due to the significant reconstruction loss associated with low-resolution images, the images are upscaled to 512 × 512 for processing. In the watermark addition stage, the Adam optimizer is used with a learning rate of 2.0 and training for 400 steps. We set the PSNR loss weight λ p to 0.05 and the LPIPS loss weight λ i to 0.25. To encode the watermark, the first 128 dimensions of the output feature generated by the Dino v2 small image encoder [37] are utilized. In the experiments, we set the directional vectors as a set of 48 vectors, where the i -th vector has a value of 1 in its i -th dimension and 0 for the remaining dimensions. In addition, during the training phase, two types of spatial transformations and pixel noise are selected with equal probability. For rotation augmentation, the rotation angle is randomly chosen in 90-degree increments. The crop augmentation is set with a crop scale range of [0 . 2 , 1 . 0] and a crop ratio range of [3 / 4 , 4 / 3] .

Compute Resources All experiments could be conducted on a single A-100 GPU with 40GB memory. Processing two images ( 512 × 512 ) in parallel takes about 5 to 6 minutes (400 steps, fp32). Increasing the batch size will improve overall efficiency. Figure 7 illustrates the relationship between the number of training steps and performance. Experimental results indicate that FreqMark still exhibits satisfactory performance with fewer training steps.

License The assets and models used in this paper are all publicly available, including Stable Diffusion 2-1 (Open Rail++-M License) [42], DINO v2 (Apache-2.0 License) [37], DiffusionDB (CC0 1.0 License) [49], and ImageNet (Custom License, as viewed on https://www.image-net. org/download ) [16].

A.2 Additional Comparison Results

We additionally compare the performance of our method with two watermarking approaches different from FreqMark: StegaStamp [46], a classical encoder-decoder-based watermarking method, and TreeRing [50], a watermarking technique that combines watermark embedding with image generation without requiring any training. Specifically, we evaluate the results under similar image quality conditions (PSNR of 28) and the same encoded bit number (100 bits) for StegaStamp. Due to TreeRing is a 1-bit encoding method, the true positive rate at a 1% false positive rate (TPR@1%FPR) is specifically compared against TreeRing [50].

Compared to StegaStamp [46], FreqMark offers strong robustness against diffusion attacks and greater flexibility due to its network independence. Users can adjust image quality, encoding bits, and customize the decoding vectors to enhance the watermark security. As opposed to Tree-Ring [50], FreqMark is capable of encoding significantly more information (48 bits vs. 1 bit) at a similar TPR@1%FPR.

A.3 Additional Experimental Results

Training Steps Figure 7 shows that FreqMark achieves considerable robustness even with fewer training steps, and there is no significant change in image quality throughout the process. Users can adjust the number of training steps according to their requirements as a trade-off.

Robustness

Quality

DiffusionDB [49]

ImageNet [16]

True Positive Rate vs. False Positive Rate We utilize 5,000 watermarked images to plot the FPRTPR curve. Figure 8 shows that FreqMark exhibits remarkably high watermark detection accuracy in the range of FPR= 10 -6 to 10 -7 . It is observed that the results for the DiffusionDB dataset [49] exhibit a significantly lower TPR at extremely low FPR compared to the ImageNet dataset [16]. This could be attributed to the higher standard deviation of bit accuracy in the ImageNet dataset [16], which consequently leads to a superior TPR under extreme conditions.

Additional Quality Metrics We include CLIP-FID [39] and L2 as supplementary image quality metrics to further evaluate FreqMark's image quality on the DiffusionDB dataset [49]. Table 9 demonstrates that FreqMark exhibits outstanding robustness performance while maintaining acceptable image quality.

TPR/FPR Results on Diffusion Attacks We present the average TPR/FPR results across varying diffusion steps on the two datasets. Table 10 indicates that FreqMark maintains excellent performance in TPR@0.1% FPR, particularly at higher diffusion steps.

Noising Scale Table 11 displays the average bit accuracy under Gaussian noise and three regeneration attacks for different combinations of Latent Noise and Pixel Noise. The results substantiate that judiciously introducing moderate noise in both the latent and pixel dimensions can effectively enhance the robustness of our method.

A.4 Additional Qualitative Results

Origin

32 bits

48 bits

96 bits

128 bits

Origin

psnr 28

psnr 30

psnr 31

psnr 32

psnr 33

Origin

Watermarked

Difference(

×

10

)

Original

Watermarked

Difference(

×

10

)

Original

Watermarked

Difference(

×

10

)

MUNCHY

MUNCHY

BEGIN

BEGIN

OBAMAI

OBAMAI

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We reflect the paper's contributions and scope in the abstract and introduction. Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitations in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have disclosed all the information needed to reproduce the main experimental results in Appendix A.1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open Access to Data and Code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We will declutter and release the code in the future. We have provided sufficient details for the replication of the paper in Appendix A.1.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have specified all the training and test details in section 5 and Appendix A.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We have included the mean and standard deviation of quality metrics (PSNR, SSIM) and robustness metrics (Bit Acc) across all test images in Section 5.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have shown the computer resources information in Appendix A.1

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in this paper complies with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed the broader impacts of our work in Section 6.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We have not released any new models or datasets.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for Existing Assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The assets utilized in the paper are publicly available, with detailed license information provided in Appendix A.1.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper do not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of watermarking of images. In particular, it introduces a method called FreqMark where watermark is embedded in the latent frequency space obtained after variation auto encoder (VAE) encoding. Numerical results are carried out on test datasets containing 500 randomly selected images from the ImageNet and 500 images from the DiffusionDB dataset.  Comparative results against DwtDctSvd, Stable Signature and SSL indicate that FreqMark exhibits superior quality-robustness tradeoff compared to the benchmark methods.\n\nSTRENGTHS:\nThe main strengths of this paper are the novelty of the proposed FreqMark algorithm and the superior performance of FreqMark compared to the baseline image watermarking methods.\n\nThe main idea behind FreqMark is hiding the message bits in the FFT outputs of latent vectors coming from the VAE. In numerical experiments on two image datasets (ImageNet and Diffusion DB), authors show that FreqMark provides a bit accuracy of better than 90% for a 48-bit encoding setting. The Ablation Studies investigating image quality, number of encoding bits, noise levels and spatial perturbations are reasonably convincing of the benefits offered by FreqMark.\n\nThe Appendices provided contain significant additional information regarding FreqMark and is one of the strengths of this paper.\n\nWEAKNESSES:\nOne of the main weaknesses of the work is that there is no convincing proof or theoretical justification of why hiding watermark bits in the FFT of latent space should be beneficial. The usual argument of using frequency domain for hiding watermark bits makes sense in that the FFT is applied to images (i.e., image pixels) and captures the spatial relationships between the image pixels. What kind of \"spatial\" relationship is captured by the FFT in the latent space and why is that beneficial for hiding watermark?\n\nFFT produces complex values, yet there is no discussion of how the watermark bits are embedded on these complex values. In Fig. 1, the outputs of FFT are displayed as magnitude images and if this is what is intended, what happened to the phase of the FFT outputs.\n\nQUESTIONS:\n1. Are the FFT outputs complex and if so, how are the watermark bits being embedded?\n\n2. What do FFT outputs represent when applied to the latent vectors? When applied to 1D time signals or 2D images, the FFT output describes the frequency content of the signals and images. What does the FFT of latent space represent and why is it useful?\n\nLIMITATIONS:\nAuthors mention that there is room for optimization in the fusion of images and watermarks. It would have been useful to have a more detailed discussion of this aspect of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-25"}

comments: SUMMARY:
This paper introduces FreqMark, a novel invisible watermarking method that enhances digital content protection through optimization in the image's latent frequency space. Experiments have been conducted to demonstrate the robustness against regeneration attacks like VAE and diffusion model.

STRENGTHS:
1. The paper is well-written and well-organized.

2. Extensive experiments have been conducted to demonstrate the robustness against various attacks including VAE and diffusion models.

WEAKNESSES:
1. Lacks some theoretical analysis. In the experiments, do you need to use the same VAE model for the attack as you have used during watermarking training? If not, then an analysis may be needed as to why it is still robust against attacks with different VAE models.

2. Since the proposed method requires case-by-case optimization, what is the watermarking time for each case, and how does it compare to other competing methods?

3. Why do we need a set of pre-trained N-dimensional direction vectors, but not directly produce the message? 

4. In Equation (4), why $k \in N$? $N$ is a number, not a set, so $\in N$ is not appropriate. Besides, using $v^N$ to denote a vector set and $v^k$ to represent a vector can lead to confusion. In Figure 2, there are $K+1$ vectors in the set, I think $K+1$ is the bit length.

4. Minors:
- $\epsilon_1$ and $\epsilon_2$ (or $\varepsilon_1$ and $\varepsilon_2$) appear in Figure 2, so it is better to mention them in the caption of Figure 2.
- In line 213, page 7, the word 'We' should be 'we'.

QUESTIONS:
Please see the Weaknesses.

LIMITATIONS:
The authors have addressed the limitations in their paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

comments: SUMMARY:
This paper proposes a method called FreqMark that is able to prevent the invisible watermarks from the regeneration attack. By using the unconstrained optimization of the image latent frequency space obtained after VAE encoding, the proposed FreqMark achieves better robustness against the regeneration attacks and traditional attacks.

STRENGTHS:
The proposed method achieves better robustness against regeneration attacks. By using a newly proposed optimization strategy, the proposed method achieves a balance between the image quality and the envisioned robustness.

WEAKNESSES:
1. The major contribution of this paper is to introduce a kind of image watermark robust to regeneration attacks. However, this part is not highlighted in its title. The authors are suggested to consider this in the title. 
2.  The proposed methodology follows an established manner with incremental innovations. The whole approach does not demonstrate enough differences when compared with the previous approaches. 
3. For the second contribution, the authors claim that their proposed framework is flexible and then say such flexibility guarantees a trade-off between the bits number, image quality, and robustness. However, why can such properties be considered as flexibility?
4. The whole pipeline presented in Figure 2 lacks novelty. This is just a very common encoder and decoder framework. The authors need to better consider their contributions for this part.

QUESTIONS:
I have shown my concerns in the weakness part. The authors are suggested to consider them during the rebuttal.

LIMITATIONS:
Please address my concerns within the weakness part.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new post-processing watermark for imagery, FreqMark. The FreqMark embeds a binary message into the frequency domain of a VAE-encoded image via a small perturbation, and then following IFFT + decompression, utilizes a pre-trained image encoder to extract the message. A PSNR + LPIPS metric is used to protect mage quality against the message embedding. The dual approach of VAE encoding + Gaussian noise makes the mark resilient against regeneration and Gaussian noise attacks. The method demonstrates high bit accuracy and good image quality when compared against DwtDct, SSL, Stable Signature on watermarked DiffusionDB & Imagenet images.\n\nSTRENGTHS:\n-Easy-to-follow narrative and nice motivation in the age of generative imagery. \n\nThe intuition behind the FreqMark algorithm is sound, easy to follow, and presumably simple to implement. \n\n-Method demonstrates robustness against regeneration, which is a very potent attack.\n\nWEAKNESSES:\nThis work is missing two critical competitor baselines which must be addressed. \n\n-In my opinion, the FreqMark is an incremental variant of the StegaStamp [1]. Surprisingly, the authors did not compare against this method even though it is well-known in existing literature that the StegaStamp is robust against many attacks, including regeneration [2]. Like FreqMark, the StegaStamp increases resilience against attacks by incorporating them into the training pipeline and uses a critic loss to preserve image quality (LPIPS + L2, versus LPIPS + PSNR for FreqMark). The resemblance of equation (10) in this manuscript to the loss function equation (2) in [1] begs the question of novelty. It is also incorrectly stated in line 71 that the StegaStamp only relies on differential image perturbations for the training pipelines -- in fact, any attack can be added, as the decoder is trained after image manipulation. \n\n-Again, if the spirit of the paper is to increase resilience against regenerations, the authors also needed to compare against the state-of-the-art Tree-Ring watermark [4], which was noted to be incredibly resilient against regenerations by the authors of the regeneration attack [2]. As the Tree-Ring is an in-processing technique that embeds a message within a diffusion process, one way to set up a comparison is to post-process a collection of Tree-Ring watermarked images via FreqMark, and then independently extract both watermarks. \n\n-As noted in [3], there is no single perceptual metric that is an objective measure of image quality, thus low PSNR or LPIPS distance does not necessarily indicate the method is not introducing artifacts. The authors need to add 1-2 more metrics (maybe L2 and FID, for example) for a more convincing argument. \n\n-500 images is too small a sample size for the tested FPR thresholds. Modern literature in this field such as [3,4,5] are using several thousand images. \n\n-VAE regenerations are far weaker compared to diffusion regenerations. Readers will want to see how the FreqMark holds up against longer, deeper regenerations (>= 100 steps) to see how the decoding accuracy is affected.\n\n-As observed in [3], the use of publicly available VAEs to encode/decode watermarks is easily defeated if the attacker uses a regeneration leveraging encoders/decoders with the same architecture. \n\n\n[1] Tancik, M., Mildenhall, B., & Ng, R. (2020). Stegastamp: Invisible hyperlinks in physical photographs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 2117-2126).\n\n[2] Zhao, X., Zhang, K., Su, Z., Vasan, S., Grishchenko, I., Kruegel, C., ... & Li, L. (2023). Invisible image watermarks are provably removable using generative ai. arXiv preprint arXiv:2306.01953.\n\n[3] An, B., Ding, M., Rabbani, T., Agrawal, A., Xu, Y., Deng, C., ... & Huang, F. (2024). Benchmarking the robustness of image watermarks. arXiv preprint arXiv:2401.08573.\n\n[4] Wen, Y., Kirchenbauer, J., Geiping, J., & Goldstein, T. (2023). Tree-ring watermarks: Fingerprints for diffusion images that are invisible and robust. arXiv preprint arXiv:2305.20030.\n\n[5] Saberi, M., Sadasivan, V. S., Rezaei, K., Kumar, A., Chegini, A., Wang, W., & Feizi, S. (2023). Robustness of ai-image detectors: Fundamental limits and practical attacks. arXiv preprint arXiv:2310.00076.\n\nQUESTIONS:\n1. See weaknesses.\n\n2. Which version of Stable Diffusion was used for the regeneration attack?\n\nLIMITATIONS:\nAdequately discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}