VideoShield: Regulating Diffusion-based Video Generation Models via Watermarking

Authors: Runyi Hu, Jie Zhang, Yiming Li, Jiwei Li, Qing Guo, Han Qiu, Tianwei Zhang
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: video, watermarking, tamper localization

Abstract:
Artificial Intelligence Generated Content (AIGC) has advanced significantly, particularly with the development of video generation models such as text-to-video (T2V) models and image-to-video (I2V) models. However, like other AIGC types, video generation requires robust content control. A common approach is to embed watermarks, but most research has focused on images, with limited attention given to videos. Traditional methods, which embed watermarks frame-by-frame in a post-processing manner, often degrade video quality. In this paper, we propose VideoShield, a novel watermarking framework specifically designed for popular diffusion-based video generation models. Unlike post-processing methods, VideoShield embeds watermarks directly during video generation, eliminating the need for additional training. To ensure video integrity, we introduce a tamper localization feature that can detect changes both temporally (across frames) and spatially (within individual frames). Our method maps watermark bits to template bits, which are then used to generate watermarked noise during the denoising process. Using DDIM Inversion, we can reverse the video to its original watermarked noise, enabling straightforward watermark extraction. Additionally, template bits allow precise detection for potential spatial and temporal modification. Extensive experiments across various video models (both T2V and I2V models) demonstrate that our method effectively extracts watermarks and detects tamper without compromising video quality. Furthermore, we show that this approach is applicable to image generation models, enabling tamper detection in generated images as well. Codes and models are available at https://github.com/hurunyi/VideoShield.

Decision: accept (poster)
Reviewer scores: 6, 6, 3, 8, 5, 8

Reviews and discussion:
<thread>
creation date:
1730210770120

thread_type:
official_review

<thread_object>
creator: reviewer_jecq

<content>
summary:
This article proposes a zero-shot video generation model with active watermarking for video copyright protection and tampering localization.  Based on the DDIM inversion process, the watermark information hidden in the initial Gaussian noise can be detected again from the generation results, and this active watermark is robust against video tampering,  image degradation, and other attacks. Furthermore, for the task of video tampering localization, the authors have designed a detection method that localizes tampering in both temporal (frame position) and spatial  (image information) aspects. Experiments have proven the performance advantages of VideoShield.

strengths:
- VideoShield leverages the unique properties of active watermarking, innovatively combining the model watermarking task with the tampering localization task, demonstrating significant performance advantages.
- This method requires no training and follows a zero-shot paradigm, making it easy to reproduce and validate its performance.
- This method is based on the properties of diffusion and Gaussian distribution, allowing it to be generalized to other diffusion-based models, such as text-to-image models, exhibiting good generalization capabilities.

weaknesses:
- The threshold and hyperparameter settings in the method section are mostly the result of manual searches. The authors may consider exploring how these thresholds could be adaptively searched in the future.

questions:
- In Table 1 and other tables, the metric "Video quality" is mentioned. How is it specifically calculated? The authors mentioned it is the average of a series of metrics; if possible, could you provide the specific values for each metric?
- Currently, the VideoShield method can hide 512 bits. What is the upper limit for watermark capacity?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730337075615

thread_type:
official_review

<thread_object>
creator: reviewer_wfue

<content>
summary:
The watermark can actively identify generated content and is currently the primary method used for detection and attribution in diffusion models. However, there has been no watermarking method specifically addressing video generation models based on diffusion models. This paper proposes VIDEOSHIELD to fill this research gap. VIDEOSHIELD introduces a Gaussian Shading watermark embedding method that does not require additional training. In addition to its basic watermarking functionality, VIDEOSHIELD also acts as a fragile watermark, enabling both temporal (across frames) and spatial (within individual frames) tampering localization. Furthermore, it introduces Hierarchical Spatial-Temporal Refinement to enhance accuracy. Experimental results demonstrate that VIDEOSHIELD performs effectively on both T2V and I2V diffusion models.

strengths:
1. This paper introduces a Gaussian Shading watermark embedding method that does not require training, thereby avoiding additional computational overhead.
2. Unlike previous methods that focus solely on spatial tampering localization, VIDEOSHIELD takes into account both temporal and spatial tampering localization during the tampering detection process. The paper introduces Hierarchical Spatial-Temporal Refinement, which enables the generation of more detailed masks for tampering localization, significantly improving the accuracy of the detection.

weaknesses:
1. The innovation presented in this paper is moderate. It extends Gaussian Shading from images to videos and further explores its potential as a fragile watermark. 
2. The presentation of the text has significant issues, particularly in Section 3.4, "TAMPER LOCALIZATION." In this section, the extensive use of symbols due to the inclusion of formulas creates obstacles to understanding the paper's content. The frequent reuse of subscripts p and q makes reading quite difficult; the matrix subscript notation is inconsistent, such as in C_(p(q=M(p))), which is recommended to be changed to C_(p,M(p)). Additionally, the comparison bits matrix Cmp could be written as CMP to standardize with other matrix symbols and to avoid confusion with the subscript p. In line 334, the letter m is used both for the variable and for watermark information, which may lead to confusion. After reviewing the main text, I found that in Supplementary Material A3.1, the authors provide a simple example of the process. I strongly recommend including Figure 7 in the main text to aid reader comprehension. Furthermore, there is some overlap between Table 1 and the Datasets section.
3. In my view, the authors have not addressed an important issue regarding the sequence of watermark attribution and tampering localization. The paper does not demonstrate the accuracy of the watermark after spatial tampering. If we assume that a video undergoes spatial tampering and the tampered areas are detected through localization, but the watermark attribution fails, we cannot conclude that a third party maliciously altered the generated video. This raises the question of whether the watermark has become ineffective in such a scenario.

questions:
Address the weakness, especially the novelty issue.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730553626988

thread_type:
official_review

<thread_object>
creator: reviewer_g7rd

<content>
summary:
This paper presents a VideoShield, which embeds watermarks directly during video generation. It maps watermark bits to template bits, which are then used to generate watermarked noise during the denoising process. Template bits allow precise detection for potential spatial and temporal modification. Extensive experiments demonstrate its effectiveness.

strengths:
The topic is interesting. Using DDIM Inversion, the video can be reversed to its original watermarked noise. The paper is written and organized well. Experiments are plenty and convincing. The ablation study verified the contribution of each design towards the whole framework.

weaknesses:
No obvious drawbacks are found. However, typesetting needs to be improved, such as keeping the same or similar text size of the table as the text, no overlapping between table and text.

questions:
The framework looks heavy, how about the computational overload? 
The relationship with "Gaussian Shading" needs to be further clarified.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730557514690

thread_type:
official_review

<thread_object>
creator: reviewer_rsgi

<content>
summary:
The paper introduces VIDEOSHIELD, a novel watermarking framework for diffusion-based video generation models. By embedding the watermark directly during the generation process, VIDEOSHIELD eliminates the need for additional training, providing a cost-effective solution for safeguarding generated videos. Additionally, the model includes a proactive tamper detection mechanism. Using template bits derived from the watermark, the authors enable both temporal and spatial localization of tampering within the video.

strengths:
- This paper is well-written.
- This paper proposes a novel scheme for proactive video forensics.

weaknesses:
1.	The proposed model may not entirely align with watermarking application scenarios. In its context, verification requires the recipient to compare the watermarked video against the original video's bit template. This requires the verifier to either have access to the original video to generate the bit template using the same encryption method, or to obtain the bit template directly from the video creator. Such requirements differ from standard watermark extraction practices, where verifiers typically can extract watermark information without needing the original video.

2.	The model uses the CHACHA20 stream cipher to generate template bits, requiring the algorithm to use the same key with different nonces for each encryption. Does this mean that a unique random number needs to be set for each video? If so, how does the method handle the storage of a large number of video-to-key mappings? Additionally, it’s unclear whether the primary goal of this approach is to protect the model itself or the generated videos.

3.	The proposed method uses watermarking for proactive tamper detection; however, the baseline experiments in the paper compare it primarily with passive detection methods (e.g., MVSS-Net, HiFi-Net), which may not be entirely appropriate. It would be more suitable to compare the approach with other active tamper detection methods. Below are references to such methods for consideration:
[1] Zhang X, Li R, Yu J, et al. Editguard: Versatile image watermarking for tamper localization and copyright protection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 11964-11974.
[2] Zhou Y, Ying Q, Wang Y, et al. Robust watermarking for video forgery detection with improved imperceptibility and robustness[C]//2022 IEEE 24th International Workshop on Multimedia Signal Processing (MMSP). IEEE, 2022: 1-6.

4.	There are formatting issues: the bottom line of Table 1 overlaps with the text below, and the font size in Table 3 is too small, making it difficult for readers to follow.

questions:
NA
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716923511

thread_type:
official_review

<thread_object>
creator: reviewer_jpgb

<content>
summary:
The paper "VideoShield: Regulating Diffusion-Based Video Generation Models via Watermarking" presents VideoShield, a novel framework for embedding watermarks in diffusion-based video generation models, including both text-to-video (T2V) and image-to-video (I2V) models. The paper addresses the need for content control and authenticity verification in AI-generated video content, focusing on integrating watermarks during the video generation process to prevent quality degradation. Key contributions include In-Generation Watermark Embedding, Tamper Localization for Videos, and Watermark Extraction and Robustness.

strengths:
1. The paper introduces an innovative approach to watermarking in diffusion-based video generation models by embedding watermarks during the generation process, which diverges from traditional post-processing methods. Compared to image watermarking, this is a less explored field. The originality of embedding watermarks during the generation process, as well as the novel dual tampering localization, is a meaningful supplement to this field.
2. The technical methods are rigorous and fully described. This paper uses DDIM inversion to provide a solid foundation for watermark embedding and extraction without affecting video quality. The extensive experimental evaluation of multiple T2V and I2V models strongly demonstrates this method's robustness, flexibility, and effectiveness. The author's detailed analysis of different watermark extraction and localization scenarios further strengthens the contribution of this article to this field.
3. This paper is well-organized and provides a clear understanding of motivation, methods, and experimental setup.
4. VideoShield's training-free and high-fidelity watermarking method provides a reliable and efficient solution for generating watermarks in video models. This paper addresses a highly relevant issue - ensuring the integrity of content in videos generated by artificial intelligence.

weaknesses:
1. Although this article introduces relevant watermarking and tampering localization methods, there is no comparative analysis with other video generation or post-processing watermarking methods. For example, including baselines for time tampering localization or using alternative methods to evaluate the robustness of specific types of distortions.
2. VideoShield is positioned as a non-training and efficient framework, but this paper lacks specific comparisons with baselines regarding time/computational complexity, such as runtime.
3. Although these experiments covered various distortions, they did not explore broader real-world attack scenarios, including testing with denser video compression techniques, color distortion, or frame rate changes, which could further validate the robustness of VideoShield in various real-world environments. In addition, analyzing the performance of VideoShield under adversarial conditions where attackers actively attempt to bypass watermarks could be an interesting solution.
4. The author can further discuss the limitations by listing the performance of VideoShield under different video quality and generation settings. This method seems to perform better on higher-quality video output, but further discussion on factors that may affect watermark integrity, such as video resolution or content complexity, will further demonstrate the effectiveness boundary of VideoShield.
5. This article briefly introduces the adaptability of image watermarking, but can VideoShield adapt to image tampering localization scenarios?

questions:
1. Can the author provide more comparative details on how VideoShield compares to other watermarking frameworks or post-processing methods in terms of tamper localization accuracy and robustness?
2. Can the author share the computational performance of VideoShield, especially regarding the running time of different video resolutions or models?
3. Will the author consider testing the robustness of VideoShield under other types of distortion, such as extreme video compression, frame rate variations, or color adjustments? These additional distortions are common in real-world applications and can enhance confidence in VideoShield's resilience.
4. The results indicate that VideoShield has a certain dependence on video quality. Does the performance of VideoShield still vary due to resolution or model complexity? If so, can the author provide more insights or data on these factors?
5. Can VideoShield adapt to image tampering localization?
6. Is the watermark capacity of 6 512 bits suitable for all video resolutions, or is the number of bits that can be embedded flexible based on the characteristics of the video?
7. Placing visual content in the main text seems better.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730718739572

thread_type:
official_review

<thread_object>
creator: reviewer_yumj

<content>
summary:
The paper addresses the need for the control of integrity and misuse of AI Generated Content. The usual approach is to use watermarks for video domain, but they are underdeveloped and have a post-processing manner, which results in video quality degradation. The authors propose a novel way of watermarking images while generating it (VideoShield). Their method is training-free and works with diffusion-based video models. Authors extract the watermark from the images using DDIM Inversion.

strengths:
1) The proposed method does not require additional training, which could simplify integration with existing diffusion models.
2) The framework’s ability to detect tampering both within frames and across the sequence

weaknesses:
1) Notation T_temp for a threshold in tampering is a bit confusing, considering T_p is a tensor.
2) No introduction for chacha20
3) No formula for calculating video quality is given 
4) Figure 2 provides only abbreviations in the legend
5) Over all paper, formulas and tables seem to have decreased font. Moreover, some tables overlap the text (e.g. Table1). This may be a reason for a possible desk rejection.

questions:
1) Results from Table 1 shows that the proposed method is only marginally better than RivaGAN w/o real images conditioning. Did you check RivaGAN with the same setup as the last row? 
2) Why do you compare only to SVD in Table 2? 
3) The authors appeal to decreased visual quality of videos generated with the existing watermarking methods, but did not organise subjective study to show that their method outperforms the others. Video quality metrics are known to have limited capabilities to estimate AIGC. Subjective comparison is required in this work, can you provide one?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732433567239

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
# General Response (All Reviewers)


Dear reviewers,

We would like to express our sincere gratitude for your thorough and constructive feedback. Your valuable insights significantly contribute to the improvement of our work. We deeply appreciate the time and effort you dedicate to reviewing our paper.

Based on your suggestions, we make the following revisions:

- We move the "Background" section to Appendix A to reduce the overall length and improve the global layout.
- We adjust the fonts for all formulas and tables, as well as the space between tables and text. (YUmj, RSGi, g7Rd, WfUe)
- We move two important figures from the appendix to the main text: Figure 2 and Figure 5. (Jpgb, WfUe)
- We expand the legend of Figure 3, replacing abbreviations with full terms. (YUmj)
- In Section 2.3, we introduce ChaCha20 (highlighted in blue). (YUmj)
- In Section 2, we make some adjustments to the notations, highlighting their first appearances in red, to avoid confusion and enhance the readability of the paper. (YUmj, WfUe)
- In Section 3.2, we place the SVD results conditioned on real images in a separate Table 4, to avoid any confusion. (YUmj)
- In Appendix B, we further elaborate on the relationship between VideoShield and Gaussian Shading. (g7Rd, WfUe)
- In Appendix D.1, we provide the detailed calculation methods for the video quality metrics. (YUmj, JeCQ)
- In Appendix E.1, we include the baseline comparison with specific video quality evaluation metrics, along with subjective evaluation results for video quality comparison. (YUmj, JeCQ)
- In Appendix E.2, we provide the baseline comparison results for watermark robustness evaluation. (Jpgb)
- In Appendix E.4, we assess the robustness of watermark extraction against various types of spatial tamper. (WfUe)
- In Appendix E.5, we present and analyze the robustness of VideoShield against distortions in more real-world scenarios. (Jpgb)
- In Appendix E.6, we compare and analyze VideoShield with the recently released open-source active image tamper localization method, EditGuard. (RSGi)
- In Appendix E.7, we present and analyze the computational overhead of VideoShield. (Jpgb, g7Rd)
- In Appendix F, we include the results and analysis of VideoShield's performance on videos generated at different resolutions (highlighted in blue). (Jpgb)

We hope these revisions address your concerns and improve the clarity and quality of the paper. Once again, thank you for your valuable feedback and support.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732508306408

thread_type:
official_comment

<thread_object>
creator: area_chair_yjce

<content>
comment:
Hi Reviewers,

We are approaching the deadline for author-reviewer discussion phase. Authors has already provided their rebuttal. In case you haven't checked them, please look at them ASAP. Thanks a million for your help!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper works on watermarking for diffusion-based video generation models. Authors proposed VideoShield for generative video watermarking. VideoShield embeds watermarks during video generation avoiding the need for additional training. Authors proposed a tamper localization feature to detect changes both temporally and spatially. The extraction is done with DDIM inversion. Experimental results show the effectiveness of the proposal methods. 

This paper was reviewed by 6 reviewers and got mixed scores as two 8, two 6, one 5, one 3.

Strength and weaknesses given by reviewers before rebuttals are as follows (notes that different reviewers has different perspectives of the paper, so conflicts in strength and weaknesses might happen):

Strength: 1) proposed method doesn't require additional training which is novel; 2) proposed methods could detect tampering both within frames and across the sequence; 3) paper is well written; 4) experimental results are convincing. 

Weaknesses: 1) lack of comparative analysis with other video generation or post-processing watermarking methods; 2) lacks comparisons with baselines on time/computational complexity; 3) did not explore broader real-world attack; 4) proposed method may not align with watermarking application scenarios; 5) comparison with other active tamper detection methods; 6) formatting issues; 7) innovation is moderate; 8)  does not demonstrate the accuracy of the watermark after spatial tampering. 9) The threshold and hyperparameter settings in the method section are mostly the result of manual searches. 

During rebuttal:
Reviewer YUmj (rating 6) thought their concerns are addressed by authors and increased rating.

Reviewer Jpgb (rating 6), Reviewer g7Rd (rating 8) and Reviewer JeCQ (rating 8) suggested authors addressed their concerns and maintained rating. 

Reviewer RSGi (rating 3) didn't provide feedback in the rebuttal.

Reviewer WfUe (rating 5) suggested authors have addressed some concerns, but thought the novelty is still not enough for ICLR. 

Several reviewers endorsed the novelty of this paper and were positive about the acceptance of this paper. Also AC checked authors addressed Reviewer RSGi (rating 3)'s concerns, so AC decided to accept this paper.