What Matters in Detecting AI-Generated Videos like Sora?
Authors: name: Chirui Chang; name: Zhengzhe Liu; name: Xiaoyang Lyu; name: Xiaojuan Qi
Published: 2024-06-27 23:03:58

Abstract: Recent advancements in diffusion-based video generation have showcased
remarkable results, yet the gap between synthetic and real-world videos remains
under-explored. In this study, we examine this gap from three fundamental
perspectives: appearance, motion, and geometry, comparing real-world videos
with those generated by a state-of-the-art AI model, Stable Video Diffusion. To
achieve this, we train three classifiers using 3D convolutional networks, each
targeting distinct aspects: vision foundation model features for appearance,
optical flow for motion, and monocular depth for geometry. Each classifier
exhibits strong performance in fake video detection, both qualitatively and
quantitatively. This indicates that AI-generated videos are still easily
detectable, and a significant gap between real and fake videos persists.
Furthermore, utilizing the Grad-CAM, we pinpoint systematic failures of
AI-generated videos in appearance, motion, and geometry. Finally, we propose an
Ensemble-of-Experts model that integrates appearance, optical flow, and depth
information for fake video detection, resulting in enhanced robustness and
generalization ability. Our model is capable of detecting videos generated by
Sora with high accuracy, even without exposure to any Sora videos during
training. This suggests that the gap between real and fake videos can be
generalized across various video generative models. Project page:
https://justin-crchang.github.io/3DCNNDetection.github.io/