Title: RAW: A Robust and Agile Plug-and-Play Watermark Framework for AI-Generated Images with Provable Guarantees
Venue: NeurIPS 2024
Accepted: yes
Keywords: Copyright Protection

Privacy
Abstract: Safeguarding intellectual property and preventing potential misuse of AI-generated images are of paramount importance. This paper introduces a robust and agile plug-and-play watermark detection framework, referred to as RAW.
As a departure from existing encoder-decoder methods, which incorporate fixed binary codes as watermarks within latent representations, our approach introduces learnable watermarks directly into the original image data. Subsequently, we employ a classifier that is jointly trained with the watermark to detect the presence of the watermark.
The proposed framework is compatible with various generative architectures and supports on-the-fly watermark injection after training. By incorporating state-of-the-art smoothing techniques, we show that the framework also provides provable guarantees regarding the false positive rate for misclassifying a watermarked image, even in the presence of adversarial attacks targeting watermark removal. 
Experiments on a diverse range of images generated by state-of-the-art diffusion models demonstrate substantially improved watermark encoding speed and watermark detection performance, under adversarial attacks, while maintaining image quality. Our code is publicly available [here](https://github.com/jeremyxianx/RAWatermark).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel image watermarking technique that injects signals into both the frequency and pixel domains of images. The watermark is specifically designed for the detection of AI-generated images and incorporates smoothing techniques to provide provable error guarantees against attacks under certain conditions. Additionally, the method's simplicity leads to faster watermarking speeds compared to existing techniques.\n\nSTRENGTHS:\n- Unlike most watermarking techniques in the literature which train an encoder decoder model without theoretical guarantees, this paper provides certified error guarantees against a certain range of attacks. This is a novel and valuable addition to the watermarking literature, since reliability is of great importance in this domain.\n\n- While having a fast inference time, the watermark achieves high robustness against some common attacks.\n\nWEAKNESSES:\n- A comparison against more recent watermarking techniques is needed. In Table 4, the paper does not provide performance results (i.e., AUROC) against attacks compared to recent methods such as TreeRing [1], Trustmark [2], and RoSteLAS [3]. The included watermarking techniques (i.e., DwtDCT, RivaGAN) are known to be weaker against attacks compared to state-of-the-art watermarks.\n\n- The paper lacks a comprehensive analysis of performance against image manipulations. While Table 4 illustrates performance on various attacks, it is unclear whether the hyper-parameters for these attacks have been set to their full strength. For instance, what hyper-parameters are being used for the VAE and Diff attacks? It is recommended that the authors include image quality metrics for the attacks (e.g., PSNR, SSIM) to demonstrate that their method maintains acceptable performance even under strong attacks that preserve image quality. Additionally, including a wider variety of diffusion attacks (e.g., pixel space and latent space) and potentially adversarial attacks (e.g., model substitution attacks), as discussed in the literature [4][5], would provide a more comprehensive evaluation.\n\n\n\n[1] Yuxin Wen, John Kirchenbauer, Jonas Geiping, and Tom Goldstein. Tree-ring watermarks: Fingerprints for diffusion images that are invisible and robust, 2023.\n\n[2] Tu Bui, Shruti Agarwal, and John Collomosse. Trustmark: Universal watermarking for arbitrary resolution images, 2023.\n\n[3] Tu Bui, Shruti Agarwal, Ning Yu, and John Collomosse. Rosteals: Robust steganography using autoencoder latent space, 2023.\n\n[4] Saberi, M., Sadasivan, V. S., Rezaei, K., Kumar, A., Chegini, A., Wang, W., and Feizi, S. Robustness of ai-image detectors: Fundamental limits and practical attacks, 2023.\n\n[5] An, B., Ding, M., Rabbani, T., Agrawal, A., Xu, Y., Deng, C., Zhu, S., Mohamed, A., Wen, Y., Goldstein, T., et al.: Benchmarking the robustness of image watermarks, 2024.\n\nQUESTIONS:\n- Based on Table 3, RAW achieves a comparably low FID and high CLIP score, suggesting it applies an imperceptible and low-budget watermark signal to the images in the current evaluations. However, a more detailed evaluation of RAW's robustness against provable diffusion attacks [1][2] that claim to break imperceptible watermarks would be valuable. I am curious to know the authors' opinion on how their method might perform against such attacks.\n\n[1] Saberi, M., Sadasivan, V. S., Rezaei, K., Kumar, A., Chegini, A., Wang, W., and Feizi, S. Robustness of ai-image detectors: Fundamental limits and practical attacks, 2023.\n\n[2] Xuandong Zhao, Kexun Zhang, Zihao Su, Saastha Vasan, Ilya Grishchenko, Christopher Kruegel, Giovanni Vigna, Yu-Xiang Wang, and Lei Li. Invisible image watermarks are provably removable using generative ai, 2023.\n\nLIMITATIONS:\n- As noted in the limitations section, this technique is designed for use with a single watermark key (w) rather than multiple values, which is a disadvantage compared to existing methods. However, given the importance of the task of detecting AI-generated images, this focus on a single key value is justified and does not limit the value of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper proposes a Robust, Agile plug-and-play Watermarking (RAW) framework, which adds learnable watermarks directly on the original image and employs a classifier to detect the presence of the watermark. This design enhances both adaptability and computation efficiency, providing an model-agnostic for real-time applications. This paper show that RAW achieves provable guarantees on the false positive rate (FPR) for detection, even under adversarial attacks.

STRENGTHS:
1. The proposed RAW provides provable guarantees on FPR for watermarking under signal processing and adversarial attacks.
2. RAW shows improved watermarking speed and robustness performance, while maintaining image quality.

WEAKNESSES:
1. In Equation (5), the watermark is also optimized, which indicates that this method can only add the same watermark. This limits the application.
2. The proposed method still embeds the watermark into the carrier image and is not specifically designed for AI-generated images.
3. In Table 2, some columns lack bolded data, and some bolded entries are not the best results.
4. The compared methods are outdated, failing to highlight the robustness and quality of this approach.
5. Since the carrier is involved in the method, PSNR should be evaluated.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

comments: SUMMARY:
This paper proposes a post-processing watermarking strategy that embeds watermarks into images after generation. The strategy is designed to be computationally efficient and model-agnostic. It involves training a learnable watermark and embedding it into both the frequency and spatial domains of the original image. The embedded watermark is later verified by a classifier, which is trained jointly with the watermark.

STRENGTHS:
* Addresses an important problem.
* The problem formulation is clear and easy to understand.
* The paper is generally well-presented.
* Experimental results show improvements in adversarial robustness and speed.

WEAKNESSES:
* The survey of existing works in the "Related Works" section is incomplete. Numerous works on watermark frameworks for diffusion models are not included.
* It would be helpful if the authors provided a figure depicting an overview of the proposed method.
* The authors should clarify why they did not use signSGD to optimize the verification model if it is better. What are the data-level optimization problems?
* The reasoning for optimizing the watermark only based on $L_0$ instead of $L_{raw}$ needs to be explained.
* The method is not plug-and-play; it requires training the watermark and the verifier for each target generative model.
* Experimental evaluation:
  * For the SDXL model, no comparison with baselines is made.
  * Metrics for image quality comparison are limited. The authors should also include metrics like SSIM.

QUESTIONS:
see above.

LIMITATIONS:
see above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-17