One missing piece in Vision and Language: A Survey on Comics
  Understanding
Authors: name: Emanuele Vivoli; name: Andrey Barsky; name: Mohamed Ali Souibgui; name: Artemis LLabres; name: Marco Bertini; name: Dimosthenis Karatzas
Published: 2024-09-14 18:26:26

Abstract: Vision-language models have recently evolved into versatile systems capable
of high performance across a range of tasks, such as document understanding,
visual question answering, and grounding, often in zero-shot settings. Comics
Understanding, a complex and multifaceted field, stands to greatly benefit from
these advances. Comics, as a medium, combine rich visual and textual
narratives, challenging AI models with tasks that span image classification,
object detection, instance segmentation, and deeper narrative comprehension
through sequential panels. However, the unique structure of comics --
characterized by creative variations in style, reading order, and non-linear
storytelling -- presents a set of challenges distinct from those in other
visual-language domains. In this survey, we present a comprehensive review of
Comics Understanding from both dataset and task perspectives. Our contributions
are fivefold: (1) We analyze the structure of the comics medium, detailing its
distinctive compositional elements; (2) We survey the widely used datasets and
tasks in comics research, emphasizing their role in advancing the field; (3) We
introduce the Layer of Comics Understanding (LoCU) framework, a novel taxonomy
that redefines vision-language tasks within comics and lays the foundation for
future work; (4) We provide a detailed review and categorization of existing
methods following the LoCU framework; (5) Finally, we highlight current
research challenges and propose directions for future exploration, particularly
in the context of vision-language models applied to comics. This survey is the
first to propose a task-oriented framework for comics intelligence and aims to
guide future research by addressing critical gaps in data availability and task
definition. A project associated with this survey is available at
https://github.com/emanuelevivoli/awesome-comics-understanding.