LLM-AD: Large Language Model based Audio Description System
Authors: name: Peng Chu; name: Jiang Wang; name: Andre Abrantes
Published: 2024-05-02 03:38:58

Abstract: The development of Audio Description (AD) has been a pivotal step forward in
making video content more accessible and inclusive. Traditionally, AD
production has demanded a considerable amount of skilled labor, while existing
automated approaches still necessitate extensive training to integrate
multimodal inputs and tailor the output from a captioning style to an AD style.
In this paper, we introduce an automated AD generation pipeline that harnesses
the potent multimodal and instruction-following capacities of GPT-4V(ision).
Notably, our methodology employs readily available components, eliminating the
need for additional training. It produces ADs that not only comply with
established natural language AD production standards but also maintain
contextually consistent character information across frames, courtesy of a
tracking-based character recognition module. A thorough analysis on the MAD
dataset reveals that our approach achieves a performance on par with
learning-based methods in automated AD production, as substantiated by a CIDEr
score of 20.5.