V2A-Mark: Versatile Deep Visual-Audio Watermarking for Manipulation Localization and Copyright Protection

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-02 18:46:52

Abstract: AI-generated video has revolutionized short video production, filmmaking, and personalized media, making video local editing an essential tool. However, this progress also blurs the line between reality and fiction, posing challenges in multimedia forensics. To solve this urgent issue, V2A-Mark is proposed to address the limitations of current video tampering forensics, such as poor generalizability, singular function, and single modality focus. Combining the fragility of video-into-video steganography with deep robust watermarking, our method can embed invisible visual-audio localization watermarks and copyright watermarks into the original video frames and audio, enabling precise manipulation localization and copyright protection. We also design a temporal alignment and fusion module and degradation prompt learning to enhance the localization accuracy and decoding robustness. Meanwhile, we introduce a sample-level audio localization method and a cross-modal copyright extraction mechanism to couple the information of audio and video frames. The effectiveness of V2A-Mark has been verified on a visual-audio tampering dataset, emphasizing its superiority in localization precision and copyright accuracy, crucial for the sustainable development of video editing in the AIGC video era.

Main paper: 
 introduction: 
 2024 is regarded as a boom year of AI-generated video. Benefited from diffusion models and the influx of extensive video data, a large amount of video generation models and editing methods [3, 7, 10, 12, 41, 55] have emerged, offering convenience in the production of short videos, film-making, advertising, and customized media. Specifically, local editing [40, 56, 59 ] has become a vital feature of AI video generation tools. For instance, AI dubbing software, capable of altering the facial expressions, lip movements, and voices of characters in a video, is extensively used in simultaneous interpretation and movie dubbing. However, this powerful editing capability is a double-edged sword. It not only facilitates video editors and creators but also blurs the boundaries between reality and forgery, posing new challenges for tamper forensics. Therefore, it is urgent to develop a method for visual-audio tamper localization and copyright protection, which can be widely used in court evidence, rumor verification, and beyond.Most visual-audio manipulation localization methods [27, 28, 32, 36, 45, 54] are passive, which mainly rely on excavating the temporal and spatial anomalous traces from the suspect videos themselves to predict tampered regions. However, these methods often prove ineffective against AIGC-based video tampering, which exhibits fewer artifacts and more realistic texture details. Additionally, most passive black-box localization networks typically require the introduction of specific types of manipulation during training, rendering them ineffective against previously unseen editing methods. Therefore, these methods have obvious shortcomings in generalization ability and accuracy of manipulation localization.Given the inherent drawbacks of passive detection and localization, visual-audio watermarking has become a consensus technology for proactive forensics. However, existing video watermarking methods are fraught with some issues. 1) Poor Accuracy: Although traditional fragile watermarking methods [13, 25] can achieve blockwise manipulation location via hash verification, their accuracy is unsatisfactory and difficult to reach the pixel-wise localization. 2) Singular Function: Video manipulation localization and copyright protection tend to be treated as two distinct and separate tasks. Tampering forensic methods lack the capability for copyright protection, limiting the applicative value of their prediction results. Simultaneously, robust deep video watermarking methods [30, 58] can only provide copyright protection and are unable to precisely pinpoint the locations of tampering within videos. 3) Single Modality: Most current forensic methods often only focus on a single visual [60] or audio modality [39] and have not established effective cross-modal interaction mechanisms. How to effectively utilize cross-modal information for manipulation localization and cross-verification of copyrights is an urgent issue.To address the above-mentioned issues, we propose an innovative multi-functional and multi-modal watermarking method, dubbed V 2 A-Mark. In the visual section, integrating the fragility of video-into-video steganography and the robustness of bit-intovideo watermarking, we simultaneously embed both localization and copyright watermarks into the video frames, enabling the decoding network to independently extract tampered areas and copyright information. In the audio section, we insert a versatile watermark into the host audio and use it to assist in the reconstruction of visual copyright information, while identifying the tampered periods in the audio. Thus, our contributions are as follows.❑ (1) We design an innovative deep versatile, cross-modal video watermarking framework, dubbed V 2 A-Mark, for visual-audio manipulation localization and copyright protection. It can embed invisible localization and copyright watermarks into video frames and audio samples simultaneously, and then obtain visual tampered area, audio tampered period, and exact copyright information in the decoding end. ❑ (2) In the visual section, we develop a temporal alignment and fusion module (TAFM) and a degradation prompt learning (DPL) mechanism, enabling the network to fully leverage temporal information for high-fidelity concealment and robust prediction of localization and copyright results. ❑ (3) In the audio section, we embed sample-level versatile watermarks into the pristine audio to identify the tampered samples and extract the copyright information. Furthermore, a cross-modal extraction mechanism is proposed to obtain the final copyright from the information of audio and video frames.❑ (4) The effectiveness of our method has been verified on our constructed visual-audio tampering dataset. Compared to other approaches, our method has notable merits in localization accuracy, generalization abilities, and copyright precision without any labeled data or additional training required for specific tampering types. 
related works 2.1 manipulation localization: 
 Prevalent image forensic techniques have focused on localizing specific types of manipulations via exploring artifacts and anomalies in tampered images [6, 14, 20, 22, 23, 28, 42, 46, 48, [52] [53] [54] . Recently, HiFi-Net [11] used multi-branch feature extractor and localization modules for AIGC-synthesized and edited images. SAFL-Net [42] designed a feature extraction approach to learn semantic-agnostic features with specific modules and auxiliary tasks. IML-ViT [32] firstly introduced vision transformer for image manipulation localization and modified ViT components to address three unique challenges in high resolution, multi-scale, and edge supervision. MaLP [2] introduced a large number of forgery images to learn the matched template and localization network. Targeted at video tamper localization, [45] exploited the spatial and temporal traces left by inpainting and guided the extraction of inter-frame residual with optical-flow-based frame alignment. UVL [36] designed a novel hybrid multi-stage architecture that combines CNNs and ViTs to effectively capture both local and global video features. However, the above-mentioned passive localization methods are often limited in terms of generalization and localization accuracy, which usually work on known tampering types that have been trained.Video watermarking is a widely accepted forensic method, which can be broadly utilized for the verification, authenticity, and traceability of images. In terms of robustness levels for extraction, video watermarking can be divided into fragile and robust watermarking [21, 51, 60] . Although classical fragile watermarking [13, 15, 18, 25, 26, 34, 35, 43] can achieve block-wise tamper localization, their localization accuracy and flexibility are unsatisfactory. Therefore, how to realize joint pixel-level tamper localization and copyright protection has still a lot of room for research.Thanks to the development of deep learning, learning-based video watermarking has attracted increased attention. For deep robust video watermarking, an intuitive approach is to apply image watermarking methods [16, 31, 61] frame by frame. For instance, HiDDeN [61] firstly designed a deep encoder-decoder network to hide and recover bitstream. Moreover, many differentiable distortion layers such as JPEG compression, screen-shooting, and face swaping [1, 8, 29, 47] were incorporated to enhance the robustness of the encoder-decoder watermarking framework. Meanwhile, CIN [31] and FIN [9] utilized flow-based models to further improve the fidelity of container images. However, these deep watermarking methods have a singular function and cannot accurately localize the tampered areas. Moreover, there are other explorations to address video degradation and temporal correlations. For instance, DVMark [30] used an end-to-end trainable multi-scale network for robust watermark embedding and extraction across various spatial-temporal clues. REVMark [58] focused on improving the robustness against H.264/AVC compression via the temporal alignment module and DiffH264 distortion layer. LF-VSN [33] utilized invertible blocks and the redundancy prediction module to realize large-capacity and flexible video steganography.  
methods 3.1 overall framework of v 2 a-mark: 
 To achieve multimodal, versatile, and proactive manipulation localization and copyright protection, as shown in Fig. 2 , the proposed V 2 A-Mark consists of two key sections, namely the visual hiding and decoding (Sec. 3.3), and the audio hiding and decoding (Sec. 3.4).In the visual hiding section, we sequentially embed pre-defined visual localization watermarks W 𝑙𝑜𝑐 ∈R 𝐻 ×𝑊 ×𝑇 ×𝐶 and the copyright watermark w 𝑐𝑜𝑝 ∈{0, 1} 𝑘 into the original video sequences V 𝑜𝑟𝑖 ∈R 𝐻 ×𝑊 ×𝑇 ×𝐶 to get the container video V 𝑐𝑜𝑛 ∈R 𝐻 ×𝑊 ×𝑇 ×𝐶 . In the audio hiding section, we add versatile watermark w ′ 𝑐𝑜𝑝 ∈{0, 1} 𝑛 to the original audio A 𝑜𝑟𝑖 ∈R 𝐿 in a sample-level manner to obtain the A 𝑐𝑜𝑛 ∈R 𝐿 . Note that 𝑇 and 𝐿 denote the number of video frames and length of the audio, respectively. "Versatile" means that this audio watermarking and decoding module can achieve audio manipulation localization and copyright protection at the same time. Moreover, the potential impacts on container videos during network transmission can be divided into two types, namely malicious tampering and common degradation. Thus, the network transmission pipeline of video frames and audio is modeled as follows.EQUATIONEQUATIONwhere T 𝑣 (•) and T 𝑎 (•) respectively denote the video and audio manipulation function. D 𝑣 (•) and D 𝑎 (•) respectively denote the video and audio degradation operation. M∈R 𝐻 ×𝑊 ×𝑇 and m∈R 𝐿 respectively denote the tempered visual masks and audio periods. Upon receiving the video V 𝑟𝑒𝑐 and audio A 𝑟𝑒𝑐 , we attempt to recover the previously embedded watermarks on different robustness levels and conduct corresponding forensics based on the extracted watermarks. In the visual decoding section, our framework precisely extracts the tampered video masks M𝑣𝑖𝑠 and copyright information w 𝑣 𝑐𝑜𝑝 . Concurrently, as shown in Fig. 2 , the tampered periods m𝑎𝑢𝑑 and the copyright w 𝑎 𝑐𝑜𝑝 in the audio will be extracted from the audio versatile decoding module. The final restored copyright information of the video ŵ𝑐𝑜𝑝 will be obtained via cross-modal combination of w 𝑎 𝑐𝑜𝑝 and w 𝑣 𝑐𝑜𝑝 (Sec. 3.5). Finally, the visual-audio tamper forensics process of V 2 A-Mark can be categorized into several scenarios, where ∧ and ∨ respectively denote the "element-wise and" and "element-wise or".❑ (1) ŵ𝑐𝑜𝑝 w 𝑐𝑜𝑝 : Suspicious V 𝑟𝑒𝑐 was not processed via our V 2 A-Mark, and we are also unable to ascertain the authenticity of the corresponding audio A 𝑟𝑒𝑐 . They cannot be used as evidence.❑ (2) ŵ𝑐𝑜𝑝 ≈ w 𝑐𝑜𝑝 ∧ ( M𝑣𝑖𝑠 0 ∨ m𝑎𝑢𝑑 0): Suspicious V 𝑟𝑒𝑐 or A 𝑟𝑒𝑐 has undergone tampering, disqualifying it as valid evidence.❑ (3) ŵ𝑐𝑜𝑝 ≈ w 𝑐𝑜𝑝 ∧ M𝑣𝑖𝑠 ≈ 0 ∧ m𝑎𝑢𝑑 ≈ 0: Suspicious V 𝑟𝑒𝑐 andA 𝑟𝑒𝑐 are both credible and have not been tampered with. V 2 A-Mark ensures the authenticity and integrity of this video.Previous work EditGuard [57] has already validated the feasibility of using the fragility and locality of image-into-image steganography for proactive image tamper localization. Specifically, fragility means the damage to the container image results in corresponding damage to the revealed secret image. Locality indicates that damage to the container image and the revealed secret image is essentially pixel-level and directly correlated. These two properties can also be effectively applied in proactive video localization. Meanwhile, EditGuard [57] adopts a "sequential embedding and parallel decoding" structure to realize united tamper localization and copyright protection. Clearly, one direct approach is to watermark each video frame via EditGuard. However, this method overlooks the exploitation of temporal correlation, making it challenging to ensure the robustness of the reconstructed watermarks and the temporal consistency of the watermarked videos. Therefore, the key issues addressed in this paper are: 1) How to utilize the auxiliary information from supporting frames for watermark embedding and decoding in reference frames; 2) How to improve the robustness of existing frameworks to video degradation; 3) How to employ the watermarks embedded in video frames for audio tamper localization and copyright protection. To address the above issues, we design the visual hiding module (VHM), visual revealing module (VRM), bit hiding module (BHM), and bit recovery module (BRM). Meanwhile, we design an efficient cross-modal extraction mechanism and introduce the advanced audio versatile watermarking and decoding method [39] to achieve cross-modal tamper localization and copyright protection.3.3.1 Input and Output Design of Visual Section. To achieve memory-efficient hiding and decoding, our V 2 A-Mark employs a multi-frame input, single-frame output structure. As shown in Fig. 3 , the visual hiding is operated group-by-group via a sliding window, traversing each video frame from head to tail. We set the length of a sliding window to 3. Given the original video group {I fuse them with intrinsic features to further enhance the robustness of our method against common video and audio degradations. 𝑜𝑟𝑖 . As shown in Fig. 4 , we resort to bidirectional cross-attention mechanisms between the supporting frames and the reference frame. Specifically, we define the scaled dot production operation as follows.EQUATIONwhere Q∈R 𝐻 ×𝑊 ×𝐷 is the query matrix projected by the reference frameI (𝑘 )𝑜𝑟𝑖 , and K, V∈R 𝐻 ×𝑊 ×𝐷 are the key and value matrix produced from the supporting frames {I (𝑖 ) 𝑜𝑟𝑖 } 𝑖≠𝑘 . Given the reference feature F 𝑟 and the supporting feature F 𝑠 , they are firstly layer normalized into F 𝑟 =Norm(F 𝑟 ) and F 𝑠 =Norm(F 𝑠 ). Then, we use linear layers to project F 𝑟 , F 𝑠 into 𝐷-dimension embedding space and calculate the cross-attention maps between reference and supporting frames as follows.EQUATIONEQUATIONwhere W 𝑟 1 , W 𝑟 2 , W 𝑠 1 and W 𝑠 2 respectively denote the projection matrices. Finally, we perform temporal fusion between the reference frame and supporting frames via the residual connection and concatenation operation.EQUATIONwhere 𝛾 1 and 𝛾 2 respectively denote the learnable parameters. With our TAFM, V 2 A-Mark can better exploit temporal correlations, thus achieving more effective concealment and more robust decoding.To further improve the robustness of V 2 A-Mark in decoding both visual localization and copyright watermarks, we embed learnable degradation promptsP 𝑣 ∈R ℎ 1 ×𝑤 1 ×𝑐 1 ×𝑒 1 , P 𝑏 ∈R ℎ 2 ×𝑤 2 ×𝑐 2 ×𝑒 2 intofeatures of the bit recovery and video revealing modules, where 𝑐 1 , 𝑐 2 respectively denote the channels of prompt, 𝑒 1 , 𝑒 2 respectively denote the number of degradation prompt. The degradation prompt pool comprises a series of learnable embeddings, with each corresponding to a type of potential degradation. Supposing that F 𝑣 and F 𝑏 are the outputs of the RPM in the video revealing module and the U-Net in the bit recovery module in Fig. 3 respectively, we utilize a channel attention mechanism (as shown in Fig. 5 ) to better encourage the interaction between the input features F 𝑣 /F 𝑏 and the degradation prompt P 𝑣 /P 𝑏 . Specifically, the features F 𝑣 /F 𝑏 are passed to a global average pooling (GAP) layer, a 1×1 convolution, and a softmax operator to produce a set of dynamic weight coefficients w 𝑣 /w 𝑏 , which is inspired by [38] . These coefficients are used to fuse each degradation prompt, resulting in degradation-enhanced features. Then, we utilize convolution and concatenation operations to fuse the degradation-enhanced features with the features extracted from RPM or the U-Net in BRM. Note that we learned two distinct sets of degradation prompts for visual and bit decoding, since we aim for the BRM to be absolutely robust against degradation, while the VRM should retain some fragility against tampering.Considering that video tampering is often accompanied by corresponding changes in the audio, we try to simultaneously identify the tampered areas of the audio, and utilize the extracted audio copyright to cross-verify the copyright in the video frame. To ensure the correspondence between video and audio, we set the audio versatile copyright watermark w ′ 𝑐𝑜𝑝 as part of the copyright w 𝑐𝑜𝑝 in the video frames. For instance, w 𝑐𝑜𝑝 is a 32-bit watermark, and w ′ 𝑐𝑜𝑝 is the first 16 bits of w 𝑐𝑜𝑝 . Inspired by the advanced proactive tamper localization tool Audioseal [39] , we introduce an audio watermark generator and detector to achieve audio versatile watermarking and decoding shown in Fig. 2 . Specifically, we utilize the watermark generator to predict an additive watermark waveform from the audio input A 𝑜𝑟𝑖 , and use a detector to output the probability m𝑎𝑢𝑑 of the presence of a watermark at each sample of the container audio A 𝑐𝑜𝑛 . The detector is trained with mask augmentation strategy to ensure its accuracy and robustness. Meanwhile, we add a message embedding layer [39] in the middle of the watermark generator to embed w ′ 𝑐𝑜𝑝 into A 𝑜𝑟𝑖 . In the decoding end, the detector will robustly decrypt w 𝑎 𝑐𝑜𝑝 , which will be used to combine with w 𝑣 𝑐𝑜𝑝 to get the final copyright ŵ𝑐𝑜𝑝 .Training:The training process of the visual section of the proposed V 2 A-Mark can be divided into two steps. Given an arbitrary original image I (𝑘 )𝑚𝑒𝑑 and watermark w 𝑐𝑜𝑝 , we first train the bit hiding and recovery module via the ℓ 2 loss.EQUATIONwhere 𝜆 is set to 10. Furthermore, we freeze the weights of BHM and BRM and jointly train the VHM and VRM. Given a video group {I (𝑖 )𝑜𝑟𝑖 } 𝑘+1 𝑘 -1 , localization watermark group {I (𝑖 )𝑙𝑜𝑐 } 𝑘+1 𝑘 -1 and copyright watermark w 𝑐𝑜𝑝 , the loss is:EQUATIONwhere 𝛼 and 𝛽 are respectively set to 100 and 1. In the audio section, we use a pre-trained audio watermarking tool [39] to realize audio hiding and decoding.Inference: As shown in Fig. 2 and 3 , we can conduct forensics via the pre-trained components. To extract tampered masks, we compare the pre-defined watermark W 𝑙𝑜𝑐 with the decoded one Ŵ𝑙𝑜𝑐 to obtain a binary mask M𝑣𝑖𝑠 ∈R 𝐻 ×𝑊 ×𝑇 :EQUATIONwhere 𝑖 ∈ [0, 𝐻 ), 𝑗 ∈ [0,𝑊 ) and 𝑡 ∈ [0,𝑇 ). 𝜃 𝜏 (𝑧) = 1 (𝑧 ≥ 𝜏). 𝜏 is set to 0. precise visual copyright, we conduct bitwise voting on the copyright extracted from each frame and select the most frequently occurring 0 or 1 as the final result w 𝑣 𝑐𝑜𝑝 . Meanwhile, we extract audio copyright w 𝑎 𝑐𝑜𝑝 ∈{0, 1} 𝑛 and use it to cross-verify with w 𝑣 𝑐𝑜𝑝 ∈{0, 1} 𝑘 , getting the final result ŵ𝑐𝑜𝑝 ∈{0, 1} 𝑘 . Considering that the audio copyright watermark can often be extracted more robustly and is not easily destroyed, we directly use it as the first 𝑛 bits in the final multimedia copyright ŵ𝑐𝑜𝑝 , which typically represents the ownership of the entire multimedia asset. The remaining 𝑘 -𝑛 bits are taken from the extracted visual watermark w 𝑣 𝑐𝑜𝑝 , which will be related to the information of video frames such as resolution, time length, and frame rate. The cross-modal extraction process is:EQUATION) 
experiments 4.1 implementation details: 
 We trained our V 2 A-Mark in the Vimeo-90K [50] without any tampered data. We test our method on 30 testing videos of Davis dataset [37] . All video frames have a resolution of 448×256 and consist of 50 to 100 frames. To synthesize audio, we manually extract the video captions and use them as prompts with the VALLE-E-X audio synthesis tool [44] . The Adam [19] is used for training 250𝐾 iterations with 𝛽 1 =0.9 and 𝛽 2 =0.5. The learning rate is initialized to 1×10 -4 and decreases by half for every 30𝐾 iterations, with the batch size set to 8. 𝑁 in Video hiding and revealing module is set to 16. The shape of two degradation prompts P 𝑣 and P 𝑏 are 36×36×72×2 and 36×36×16×6. We embed 32-bit copyright watermarks w 𝑐𝑜𝑝 and pure blue visual localization watermarks W 𝑙𝑜𝑐 to original videos. We use replication padding to process the first and last frame of the original video. Meanwhile, we also embed a versatile watermark w ′ 𝑐𝑜𝑝 into the original audio.To evaluate the visual localization and copyright recovery accuracy, we compared our method with some SOTA passive methods OSN [46] , PSCC-Net [27] , HiFi-Net [11] , IML-ViT [32] and a proactive forensics method EditGuard [57] . protection, bit accuracy (BA) is used to assess the copyright recovery performance. We use two SOTA deep video inpainting methods, ProPainter [59] and E 2 FGVI [24] , and a naive slicing approach to simulate malicious tampering. As reported in Tab. 1, our V 2 A-Mark achieves impressive localization performance with an F1-Score of approximately 0.95, an AUC of 0.99, and an IoU close to 0.9. In contrast, other passive localization methods, which rely solely on tampered video clues, perform poorly in localizing unseen types of manipulation. Furthermore, when using EditGuard to watermark each video frame, although it achieves satisfactory localization results, it falls short in effectively utilizing temporal information. Consequently, the IoU of the predicted masks in various tampering methods is generally about 0.03 lower than that achieved by our V 2 A-Mark. Additionally, our V 2 A-Mark achieves an over 99.5% bit accuracy across various tampering methods, which is also marginally higher than that of EditGuard. Furthermore, as shown in Fig. 6 , our method has very obvious advantages over SOTA passive localization method PSCC-Net [27] , which can be attributed to our proactive tamper localization mechanism. Meanwhile, since we adopted a more effective temporal alignment and fusion method, we found that in some scenes where EditGuard can only locate the rough outline of the tampering area, our V 2 A-Mark can still clearly predict the tampered traces.To evaluate the visual quality of V 𝑐𝑜𝑛 , we compared our V 2 A-Mark with other watermarking methods on 30 testing videos from DAVIS [37] . For a fair comparison, we also retrained our EditGuard on 448×256 original videos and 32 bits. Our comparison methods include the SOTA bit-hiding watermarking method [8, 16, 31, 47] , large-capacity steganography method [17, 33] , and a versatile image Our method can predict more accurate and clearer tampered masks. We also present our container and tampered videos.watermarking method [57] . As shown in Tab. 2, the PSNR and SSIM of our container videos far outperform most watermarking methods such as MBRS, PIMoG, and SepMark, but is close or slightly inferior to CIN. Note that these methods only hide 30 bits in the videos, but our V 2 A-Mark hides both an RGB image and 32 bits.Compared with high-capacity steganography methods EditGuard, LF-VSN, and HiNet, our method also has clear advantages in visual quality. Meanwhile, the perceptual quality (NIQE) of our watermarked videos surpasses all other methods. To verify the security of our method, we perform anti-steganography detection via StegExpose [4] on container videos of various steganography methods. All the methods concealed pure blue videos into the original videos. Note that the detection set is built by mixing container and original video frames with equal proportions. We vary the detection thresholds in a wide range in StegExpose [4] and draw the ROC curve in Fig. 7 . The ideal case represents that the detector has a 50% probability of detecting container videos from an equally mixed detection test, the same as a random guess. Evidently, the security of our method exhibits a significant advantage compared to all competitive methods.To evaluate the accuracy of V 2 A-Mark for audio tamper localization, we randomly insert 1s -2s tampered audio into our constructed 30 original audio. SNR and PESQ are used to evaluate the quantitative and perceptual quality of watermarked audio. Bit accuracy is used to evaluate the bit error rate of the pre-defined w ′ 𝑐𝑜𝑝 and extracted w 𝑎 𝑐𝑜𝑝 . AUC is used to calculate the localization accuracy between the predicted audio tampered period m 𝑎𝑢𝑑 and the ground truth of the tampered area m. We observed from Tab. 3 that the watermarked audio maintains high SNR/PESQ on 28.29 dB/4.50 with the original audio, indicating that our V 2 A-Mark has little impact To analyze the robustness of our V 2 A-Mark, we compare our method with EditGuard, the best comparative method in the clean case. We selected three types of video degradation, including Gaussian noise, H.264 video coding, and Poisson noise. As reported in Tab. 4, we found that our V 2 A-Mark has only slight performance degradation under various degradations compared to the clean scene, and both surpass EditGuard in localization accuracy and copyright reconstruction. Specifically, since we use a multi-frame input, singleframe output structure, which better explores temporal information, our method performs better in handling inter-frame degradation (such as H.264 video coding) than EditGuard which adds watermarks frame by frame. As reported in Tab. 4, the recovered bit accuracy of our method far surpasses EditGuard by 4.18% and 6.95% in QP=5 and QP=10. Meanwhile, our V 2 A-Mark also outperforms EditGuard by 0.028 and 0.022 in localization accuracy (IoU), which proves its superiority in decoding robustness.To evaluate the contribution of each component, we mainly conduct ablation studies on the temporal alignment and fusion module (TAFM) and degradation prompt learning (DPL). Our results are reported on Tab. 5, where "random degradation" denotes that we randomly select one degradation from Gaussian noise, H.264, and Poisson noise. Comparing case (a) and ours in the "clean" scene, it demonstrates that the proposed TAFM can enhance the localization accuracy and achieve 0.012 gains in IoU, which proves that the proposed TAFM can boost the temporal interaction and realize effective temporal alignment. Comparing case (b) and ours in the "random degradation" scene, due to the learned degradation representations, we find that our method achieves significant gains on localization accuracy and copyright precision.Our V 2 A-Mark can provide focused protection for videos based on user-defined areas. This allows our V 2 A-Mark to apply to some Table 5 : Abalation studies on the core parts of V 2 A-Mark.global tampering such as visual-audio deepfake. Specifically, we use EfficientSAM [49] to segment the facial regions that need focused protection, and add localization watermarks only to these parts, while still embedding a global copyright watermark. As shown in Fig. 8 , we manipulate the identity in the container video frames via SimSwap [5] , and alter the first 0.5s of this audio from "there are many jobs for American" to "there are few jobs for American. " Subsequently, our V 2 A-Mark is capable of effectively detecting tampered areas in the face region as well as alterations in the audio.For the audio portion, we determine whether each sample point has been tampered with by evaluating the probability of alteration.  
conclusion: 
 To address the challenges of AI-generated visual-audio forensics, an innovative deep watermarking method with strong generalizability, versatile function, and cross-modal properties dubbed V 2 A-Mark is proposed. It embeds invisible visual-audio localization and copyright watermarks into the original video frames and audio. If encountering malicious tampering on visual or audio information, we can get accurate tampered visual masks, video copyright, and tampered audio periods in the decoding end via our V 2 A-Mark. Facing the imminent explosive growth of the AIGC video industry, our V 2 A-Mark has the potential to safeguard the sustainable development of the AIGC industry, and also establish a clean and transparent information environment. Limitations: Since there is a certain contradiction between the fidelity and robustness of video watermarking, we are still committed to designing advanced modules to achieve better tradeoff. Additionally, as there are few video diffusion-based editing methods available, we have not conducted experiments on larger video editing models. However, we believe our method is robust and effective against all forms of local visual-audio manipulation.  
 
 Appendix:  


Decision: The paper proposed an generative AI method for detecting and localizing any visual or audio temperament. Three reviewers suggested "borderline accept" and one reviewer suggested "reject". The primary concern is the novelty of the work. In particular, one reviewer pointed out that the work simply combines watermarking and copyright to protect a video.

Reviews: review: paper_summary: paper_summary: This paper designs an innovative deep versatile, cross-modal video watermarking framework (V2A-Mark) for visual-audio manipulation localization and copyright protection. V2A-Mark combines the fragility of video-into-video steganography with deep robust watermarking to expose the visual tampered area, audio tampered period, and copyright information. 
The superiority of V2A-Mark in manipulation localization precision and copyright protection is verified from multiple perspectives.
strength_weakness: strength_weakness: 1. A temporal alignment and fusion module (TAFM) and a degradation prompt learning (DPL) mechanism are developed, to enhance the temporal consistency for information concealment and extraction. The degradation prompts of DPL are able to facilitate bit decoding and visual localization.
2. Sample-level versatile watermarks are embedded into the audio to localize the tampered samples and extract the copyright information. 
3. The proposed methodology shows better performance in localization and copyright protection abilities. Meanwhile, the security of V2A-Mark outperforms all competitive methods as well.
limitations: limitations: 1. It is not explained how robustness is ensured or whether a noise layer is implemented.
2. The method only considers cases where audio modifications do not affect the watermark. The paper does not explain how the copyright information from the audio is used for manipulation localization, and the interaction between modalities is not considered.
paper_summary: paper_summary: This paper proposes the V2A-Mark to address the limitations of current video tampering forensics, such as poor generalizability, singular function, and single modality focus.  Combining the fragility of video-into-video steganography with deep robust watermarking, the V2A-Mark can embed invisible visual-audio localization watermarks and copyright watermarks into the original video frames and audio, enabling precise manipulation localization and copyright protection. However, technological innovation is limited.
strength_weakness: strength_weakness: （1）design an innovative deep versatile, cross-modal video watermarking framework for visual-audio manipulation localization and copyright protection.
（2）develop a temporal alignment and fusion module (TAFM) and a degradation prompt learning (DPL) mechanism, enabling the network to fully leverage temporal information for high-fidelity concealment and robust prediction of localization and copyright results.
（3）Numerous experimental results have verified its effectiveness in localization accuracy, generalization abilities, and copyright precision without any labeled data or additional training required for specific tampering types.
limitations: limitations: (1)  The so-called cross-modal video watermarking framework is essentially a simple combination of video watermarking and audio watermarking, and the relationship between the different modalities is not exploited.

(2) The design of the proposed temporal alignment and fusion module is based on the existing bidirectional cross-attention mechanisms, and moreover, no relevant citation is given. 

(3) The audio watermarking section is not described in detail and, according to the paper, it used an existing model directly. Thus, the third contribution is not solid.

(4) There are many parts of the paper that are not clear: What are the invertible blocks? What is an MLP extractor?  Why replicate I\_rec three times? Why and how do you select only the intermediate frames as the final output? As for the use of the term “reference frame”, it will conflict with the one in video coding. 

(5) The data comparisons in Table 2 are not very informative because the simulated attacks adopted in their training process are different.
paper_summary: paper_summary: This paper proposes a new multi-modal watermarking method of manipulation localization and copyright protection for video tampering forensics. The method combines the fragility of video-into-video steganography with deep robust watermarking and embed invisible visual-audio localization watermarks and copyright watermarks into the original video frames and audio. Meanwhile, due to the development of TAFM and DPL, the method fully leverages the temporal information of the video and keeps robustness. The effectiveness of the method has been well demonstrated through a series of experiments.
strength_weakness: strength_weakness: This paper innovatively integrates video-into-video steganography with watermarking, utilizing the vulnerability of video-into-video steganography for tampering detection and the robustness of video watermarking for copyright protection, achieving a synergistic effect where the whole is greater than the sum of its parts by accomplishing tampering detection and copyright protection simultaneously. Additionally, in the strategy of watermarks embedding, the proposed TAFM module utilizes the unique temporal correlation of videos to embed information into the temporal domain, which is different from EditGuard and exhibits superior experimental performance. Furthermore, this paper creatively introduces audio watermarking and combines it with video watermarking to achieve cross-modal forensics, offering new insights for this field.
limitations: limitations: To further improve the quality of this paper, I advise the authors to incorporate the following comments.
1.	Using the structure of multi frame input and single frame output, the localization information is embedded into the video via a sliding window, and then a copyright watermark is embedded. Employing this strategy, the same watermark is embedded in all frames. In terms of efficiency, this does not differ from embedding in every single frame, does this lead to low efficiency? 
2.	The method proposed indeed demonstrates highly effective detection of tampered regions. However, the consideration of temporal information in this method involves only the preceding and subsequent two frames of the current frame. I am unsure whether this strategy utilizing a sliding window perfectly exploits the unique temporal correlation inherent in videos. Additionally, as proposed in Section 4.2, can it be theoretically proven that the method, due to the utilization of modules such as TAFM, outperforms EditGuard in specific scenarios? 
3.	Unlike images, videos are susceptible to frame switching and frame deletion attacks. However, this method does not provide tampering detection results at the frame level. Additionally, experimental results for other digital noises, such as cropping, have not been presented. Should these factors be considered?
paper_summary: paper_summary: This paper proposes V2A-Mark, an innovative watermarking framework for visual-audio manipulation localization and copyright protection. The framework can embed invisible visual-audio localization watermarks and copyright watermarks into the original video frames and audio to allow manipulation localization and copyright protection in the decoding end. Experimental results verify the effectiveness of V2A-Mark as a deep versatile and cross-modal application.
strength_weakness: strength_weakness: (1) For video localization, the framework adopts a temporal alignment and fusion module (TAFM), enabling the temporal correlations of the hidden features.
(2) In the audio section, we embed sample-level versatile watermarks into the pristine audio to identify the tampered samples and extract the copyright information. Furthermore, a cross-modal extraction mechanism is proposed to obtain the final copyright from the information of audio and video frames.
(3) Experimental results on the visual-audio tampering dataset verify the notable merits in localization accuracy generalization abilities, and copyright precision compared to other approaches.
limitations: limitations: (1) In the robustness experiments for watermarking, the QP for video compression is too small to demonstrate the robustness of this method.
(2) The manipulation localization of audio samples is not clearly explained.