SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanations

Authors: Zhaorun Chen, Francesco Pinto, Minzhou Pan, Bo Li
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Video Guardrail Model, Safe Foundation Models, Efficient LLMs Inference, LLM Safety, Multimodal Foundation Models

Abstract:
With the rise of generative AI and rapid growth of high-quality video generation, video guardrails have become more crucial than ever to ensure safety and security across platforms. Current video guardrails, however, are either overly simplistic, relying on pure classification models trained on simple policies with limited unsafe categories, which lack detailed explanations, or prompting multimodal large language models (MLLMs) with long safety guidelines, which are inefficient and impractical for guardrailing real-world content. To bridge this gap, we propose SafeWatch, an efficient MLLM-based video guardrail model designed to follow customized safety policies and provide multi-label video guardrail outputs with content-specific explanations in a zero-shot manner. In particular, unlike traditional MLLM-based guardrails that encode all safety policies autoregressively, causing inefficiency and bias, SafeWatch uniquely encodes each policy chunk in parallel and eliminates their position bias such that all policies are attended simultaneously with equal importance. In addition, to improve efficiency and accuracy, SafeWatch incorporates a policy-aware visual token pruning algorithm that adaptively selects the most relevant video tokens for each policy, discarding noisy or irrelevant information. This allows for more focused, policy-compliant guardrail with significantly reduced computational overhead. Considering the limitations of existing video guardrail benchmarks, we propose SafeWatch-Bench, a large-scale video guardrail benchmark comprising over 2M videos spanning six safety categories which covers over 30 tasks to ensure a comprehensive coverage of all potential safety scenarios. We have conducted extensive experiments, showing that SafeWatch outperforms all SOTA video guardrails on SafeWatch-Bench by 28.2%, and achieves a 13.6% improvement on existing benchmarks, all while reducing inference costs by an average of 10%. SafeWatch also demonstrates strong policy-following abilities and outperforms previous SOTAs by 5.6% and 15.6% in zero-shot generalizability to new policies and new prompting tasks. Additionally, both LLM-as-a-judge and human evaluators confirm the high quality of the explanations provided by SafeWatch. Our project is open-sourced at https://safewatch-aiguard.github.io.

Decision: accept (poster)
Reviewer scores: 8, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730137369995

thread_type:
official_review

<thread_object>
creator: reviewer_pm58

<content>
summary:
The paper proposed a safety aware video understanding benchmark, including 2M human verified videos. The unsafe scenarios are separated into 6 classes. The authors also design a pipeline for automatically data generation. For the video understanding model, the authors propose the Parallel Equivalent Policy Encoding and Policy-Aware Adaptive Pruning to encode the Safety Policy Guidelines and reduce the redundancy. The result of the trained model is good comparing to both close- and open- source models.

strengths:
1. The proposed dataset is novel and the data curation procedure is well-organized.
2. The proposed Parallel Equivalent Policy Encoding and Policy-Aware Adaptive Pruning can effectively encode the Safety Policy Guidelines and reduce the redundancy of video tokens.
3. The results are good.

weaknesses:
1. Dataset Construction: 
       (1) The release and separation of the dataset is a concern. The author can only provide links to publicly available sources and annotations. But it is common that the link may fail.I understand that this is something unavoidable, but it will undoubtedly reduce the frequency of use and impact of this dataset.
       (2) 2M videos for human verification is a huge effort, the authors don't provide any details of the procedure.

2. Model training：
      (1) Some of the training procedure is ambiguous, there should be more details about Preference Post-tuning procedure.

questions:
1. The author mentioned that Human Verification is used for curating the 2M video benchmark. Can the authors make a detailed description of the Human Verification procedure?

2. More procedure should be enclosed in the appendix.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730346723979

thread_type:
official_review

<thread_object>
creator: reviewer_ytks

<content>
summary:
The authors of the paper propose a multimodal large language model (MLLM) called SafeWatch, designed to follow customized safety policies and provide multi-label video guardrail categorical outputs with answer explanations in a zero-shot manner. They also introduce SafeWatch-Bench, a large-scale video guardrail benchmark containing over 2 million videos spanning 6 safety broad categories and covering over 30 finer-grained risk categories to ensure comprehensive coverage of potential safety scenarios.

The technical contributions include:
- Model Design: The authors introduce two key plug-and-play modules: Parallel Equivalent Policy Encoding (PEPE) and Policy-Aware Adaptive Pruning (PAP). 
  - PEPE mitigates high latency from extensive input contexts and policy positional bias by dividing lengthy safety guidelines into independent chunks encoded in parallel with equal importance. 
  - PAP, on the other hand, reduces latency by selecting the most relevant visual tokens for each policy while discarding those with low relevance.

- Data: Each instance in SafeWatch-Bench is annotated with multi-label guardrail categories and detailed explanations. The dataset includes 2 million videos—both real-world and generative from various SOTA models—comprising an instruction-tuning set and a test set of 1K hand-selected, high-quality annotated videos across subcategories.

- Training strategy: The authors fine-tune InternVL2-8B with their modeling changes on this new data via three stages, i.e., multi-task training, adaptive-pruning training, and preference post-tuning. 
  - Stage 1: Only PEPE is trained during this stage on a large corpus of unsafe videos, as well as traditional VQA and captioning tasks on normal videos. 
  - Stage 2: Both PEPE and PAP are fine-tuned on guardrail tasks. 
  - Stage 3: Preference pairs are curated to enable the preference post-tuning.

strengths:
1. Their model, SafeWatch, outperforms SOTA video guardrails on SafeWatch-Bench by 19.6% and on existing benchmarks by 15.4%, while reducing inference costs by an average of 25%. SafeWatch also demonstrates strong policy-following abilities, outperforming baselines by 20% in zero-shot adaptability to new policies. Additionally, both LLM-as-a-judge and human evaluators confirm the high quality of the explanations provided by SafeWatch.
2. The design choices are well-founded, following best practices for efficient MLLM construction.
3. This is an important area of study, with meaningful contributions (if these contributions are reproducible).

weaknesses:
1. Missing Evaluation: The evaluation of the Safety-aware Event Sampling step is absent. This step should be crucial for model performance, as the authors used TransnetV2 to segment videos into safety-aware events, sampling a single frame per event for further MLLM processing.
2. Dataset Clarity: The dataset’s specifics and its exact use in model training remain unclear. For instance, there is no information on the average video length or the typical length of an explanation in SafeWatch-Bench. Additionally, the quality of the SafeWatch-Bench test set is not fully addressed, which is particularly important for an evaluation dataset.
3. Reproducibility Concerns: Reproducibility in data collection and model training is questionable. For instance, Section 4.2 on "multi-agent consensus video annotation" provides a basic idea of the processes but lacks sufficient detail for replication (e.g., missing the prompts used, configurations such as the number of frames used for each model, etc.). Additional issues are noted in the "Questions" section.

questions:
1. Could you provide some evaluation results for Safety-aware Event Sampling? Currently, its effectiveness or limitations are unclear.
2. Is SafeWatch-Bench truly a video benchmark? Specifically, does it truly require reasoning across multiple frames for a model to achieve high performance?
3. It appears that humans do not directly provide annotations for SafeWatch-Bench; instead, annotations are model-generated, with human reviewers checking if re-annotation is necessary. In the caption for Figure 2, you mention that the 1K test set has high-quality annotations. Were these test set annotations created directly by humans, or were they produced via the multi-agent propose-discuss-consensus pipeline?
4. Could you describe how the preference pairs were curated for the preference post-tuning stage? Additionally, how were the challenging benign examples—those easily identified by humans but likely to mislead guardrail models—selected? A detailed explanation of these curation processes would be helpful.
5. What is the quality of the synthetic videos generated by the GenAI models? Are they accurately aligned with the unsafe prompts? Was any video filtering applied to filter out misaligned videos?
6. Could you clarify your model training data recipe? Are the unsafe videos used in stage-1 training and the guardrail tasks in stage-2 training drawn from the instruction-tuning dataset within SafeWatch-Bench? Additionally, how many samples are used for each stage, task, and dataset?
7. Which layers are tuned in the preference post-tuning stage?
8. In Table 3 of Appendix A.1, there is a column named "Temporal location". What does it mean?
9. What is the SFT Baseline mentioned in Figure 5 and Table 5?
10. It was claimed that prior datasets lack detailed descriptions of the videos, suggesting that SafeWatch-Bench offers a detailed description for each video. Is that correct?
11. There is unsafe content in SafeWatch-Bench. Will SafeWatch and SafeWatch-Bench be released? If so, how do you plan to ensure their proper use?

Minor comments:
Line 749 has placeholder text.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730471276482

thread_type:
official_review

<thread_object>
creator: reviewer_yjic

<content>
summary:
SAFEWATCH is a new video moderation system that efficiently identifies harmful content and explains its decisions. It features two main innovations: PEPE (for parallel policy encoding) and PAP (for selective video analysis), both designed to make the system faster and more accurate. The researchers also developed SAFEWATCH-BENCH, a large dataset containing 2 million videos across six categories of harmful content, which they used to train and test their system.

strengths:
1. The authors contribute a very large-scale benchmark for video security
2. The authors propose the PEPE algorithm, which can mitigate positional bias in the input
3. The authors propose the PAP algorithm, which maintains high recognition accuracy while reducing inference costs

weaknesses:
1. The working mechanism of the PEPE algorithm lacks detailed theoretical explanation or experimental validation. The authors conduct ablation experiments to prove the effectiveness of the PEPE algorithm, but they don’t provide sufficient proof of its underlying principles. In lines 293-297, the authors claim that the PEPE algorithm can provide independent representations for each policy, which can alleviate the position bias problem in MLLM mentioned in lines 266-269. **However, regarding this claim, there are neither experimental designs nor mathematical proofs to support it. I have doubts about whether the mechanism behind the algorithm truly aligns with the authors' claims.**
    
    
2. There is a lack of explanation regarding the effectiveness of the multi-agent propose-discuss pipeline mentioned in line 105. The authors mention in lines 105-106 that they use a novel pipeline for data annotation, but there is limited discussion about this pipeline. In the pipeline-related content, the authors do not cite any references, and based on my knowledge of related work, this pipeline has not been used in any previous work, making this the first work to employ this pipeline. Given this, **I am uncertain whether this pipeline can provide sufficiently high-quality annotation results**, and the authors have not provided any quality analysis of the annotation results.

questions:
1. Could the authors design corresponding experiments and proofs to demonstrate that the mechanism producing the algorithm's effects aligns with their claims that the PEPE algorithm can "allow each policy to be encoded independently and in parallel" and that "equivalent positional embedding ensures that different policies are treated without bias"?
2. Could the authors provide some quantitative analysis of the annotation quality? Can this pipeline approach human-level annotation quality? Compared to annotation by a single LLM/VLM, what advantages does incorporating Multi-agent Discussion bring?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730563594684

thread_type:
official_review

<thread_object>
creator: reviewer_qsgz

<content>
summary:
This paper presents an MLLM-based video guardrail model that takes into account safety policies to provide a multi-label video content output including explanation, considering both the safety policies and the video content. The proposed model comprises two plug-and-play modules to improve latency of the guard rail model and mitigate positional biases by breaking down the safety guidelines. This work also introduces a benchmark for video guardrailing using multi-agent consensus and comparison across existing MLLMs.

strengths:
- The paper addresses a critical topic by proposing guardrails for video MLLMs based on defined safety policies, which is timely and important with the rise of MLLMs.
- It introduces a baseline model built upon the InternVL2-8B backbone and leverages two plug-in modules to (1) improve latency during training and inference, and (2) reduce positional biases related to the policy order.
- The benchmark provides a comprehensive evaluation of existing MLLMs on video guardrailing tasks, demonstrating the model’s effectiveness across six safety policy categories, covering 30 subtopics.

weaknesses:
- Details about the training and testing splits within the benchmark are insufficient, leaving questions about data partitioning.
- The authors should clarify if any videos were discarded during dataset curation due to multi-agent discussion pipelines not reaching a consensus or human verification disagreements on final explanations. This clarification could shed light on the multi-agent approach's effectiveness in generating explanations that align with human perspectives, especially given video content's subjective nature.

questions:
- SFT Baseline: Could the authors provide additional context for the "SFT baseline" mentioned in Figure 5?
- Inference Cost: What accounts for the increase in inference cost with additional few-shot examples, as illustrated in Figure 5?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732274195473

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A Summary of Rebuttal and Paper Revisions

comment:
We sincerely thank all the reviewers and chairs for their valuable time and thoughtful feedback on our work. We are glad that all the reviewers agree on its technical novelty and critical contribution to addressing the risks given by the rise of multimodal foundation models. We have carefully addressed each reviewer’s concerns and questions point-by-point and made corresponding updates in the revisions. Below is a summary of the additional clarifications and evaluation results included in our rebuttal:

- Following reviewer yJiC's suggestions, we have designed two additional experiments and conducted theoretical analyses to validate the principles of PEPE. The results, provided in Table 10 and Figures 6–7 in the appendix, verify that the working mechanism of PEPE indeed aligns with our expectations in the paper.
 
- Following reviewer QsGZ, yJiC, and YtKs's suggestions, we have designed additional experiments and provided various case studies to demonstrate the effectiveness and efficiency of our proposed multi-agent data annotation pipeline in achieving human-like annotation quality. The results are shown in Table 18-19 and Figures 15–20 in the appendix.

- Following reviewer YtKs's suggestion, we have designed an additional evaluation of the safe-aware event sampling to demonstrate its effectiveness in providing accurate unsafe event grounding for the guardrail task. The results are shown in Table 9 in the appendix.
 
- To address reviewer QsGZ, YtKs, and PM58's concerns, we have provided a more comprehensive explanation of the dataset statistics and the training pipeline, including data details, prompts, configurations, and intermediate results for each training stage. These are presented in Table 9, Table 11-20, and Figures 10–14 in the appendix and elaborated on in our individual responses. All the prompts used in this work are provided in the Appendix. B8.

- To address all reviewers' ethical concerns, we have outlined an actionable plan including releasing only a subset of the videos that are not copyrighted, with all the extremely harmful categories such as child abuse removed, and with all the human identities blurred to ensure that potential biases, privacy concerns, harms, and legal compliance are thoroughly and responsibly addressed.

Thank you again for your thoughtful suggestions and efforts in helping us improve SafeWatch. We hope our response adequately addresses your concerns, and we look forward to engaging in further discussions during the follow-up period!

Best regards,

Submission #9949 Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work introduces a new video moderation method that efficiently identifies harmful content and explains its decisions. The work has two major innovations: PEPE (for parallel policy encoding) and PAP (for selective video analysis), and both are well designed to make the system fast and accurate. Reviewers pointed out that the design choices are well-founded, which follow best practices for efficient MLLM construction. More importantly, this work addresses an important problem of study, with meaningful contributions. Thus, all reviewers recommend acceptance. AC agrees with the reviewers.