ViBe: A Text-to-Video Benchmark for Evaluating Hallucination in Large
  Multimodal Models
Authors: name: Vipula Rawte; name: Sarthak Jain; name: Aarush Sinha; name: Garv Kaushik; name: Aman Bansal; name: Prathiksha Rumale Vishwanath; name: Samyak Rajesh Jain; name: Aishwarya Naresh Reganti; name: Vinija Jain; name: Aman Chadha; name: Amit P. Sheth; name: Amitava Das
Published: 2024-11-16 19:23:12

Abstract: Latest developments in Large Multimodal Models (LMMs) have broadened their
capabilities to include video understanding. Specifically, Text-to-video (T2V)
models have made significant progress in quality, comprehension, and duration,
excelling at creating videos from simple textual prompts. Yet, they still
frequently produce hallucinated content that clearly signals the video is
AI-generated. We introduce ViBe: a large-scale Text-to-Video Benchmark of
hallucinated videos from T2V models. We identify five major types of
hallucination: Vanishing Subject, Numeric Variability, Temporal Dysmorphia,
Omission Error, and Physical Incongruity. Using 10 open-source T2V models, we
developed the first large-scale dataset of hallucinated videos, comprising
3,782 videos annotated by humans into these five categories. ViBe offers a
unique resource for evaluating the reliability of T2V models and provides a
foundation for improving hallucination detection and mitigation in video
generation. We establish classification as a baseline and present various
ensemble classifier configurations, with the TimeSFormer + CNN combination
yielding the best performance, achieving 0.345 accuracy and 0.342 F1 score.
This benchmark aims to drive the development of robust T2V models that produce
videos more accurately aligned with input prompts.