Towards Understanding Unsafe Video Generation
Authors: name: Yan Pang; name: Aiping Xiong; name: Yang Zhang; name: Tianhao Wang
Published: 2024-07-17 14:07:22

Abstract: Video generation models (VGMs) have demonstrated the capability to synthesize
high-quality output. It is important to understand their potential to produce
unsafe content, such as violent or terrifying videos. In this work, we provide
a comprehensive understanding of unsafe video generation.
  First, to confirm the possibility that these models could indeed generate
unsafe videos, we choose unsafe content generation prompts collected from 4chan
and Lexica, and three open-source SOTA VGMs to generate unsafe videos. After
filtering out duplicates and poorly generated content, we created an initial
set of 2112 unsafe videos from an original pool of 5607 videos. Through
clustering and thematic coding analysis of these generated videos, we identify
5 unsafe video categories: Distorted/Weird, Terrifying, Pornographic,
Violent/Bloody, and Political. With IRB approval, we then recruit online
participants to help label the generated videos. Based on the annotations
submitted by 403 participants, we identified 937 unsafe videos from the initial
video set. With the labeled information and the corresponding prompts, we
created the first dataset of unsafe videos generated by VGMs.
  We then study possible defense mechanisms to prevent the generation of unsafe
videos. Existing defense methods in image generation focus on filtering either
input prompt or output results. We propose a new approach called Latent
Variable Defense (LVD), which works within the model's internal sampling
process. LVD can achieve 0.90 defense accuracy while reducing time and
computing resources by 10x when sampling a large number of unsafe prompts.