HEMM: Holistic Evaluation of Multimodal Foundation Models
Authors: name: Paul Pu Liang, name: Akshay Goindani, name: Talha Chafekar, name: Leena Mathur, name: Haofei Yu, name: Ruslan Salakhutdinov, name: Louis-Philippe Morency

Multimodal foundation models that can holistically process text alongside
images, video, audio, and other sensory modalities are increasingly used in a
variety of real-world applications. However, it is challenging to characterize
and study progress in multimodal foundation models, given the range of possible
modeling decisions, tasks, and domains. In this paper, we introduce Holistic
Evaluation of Multimodal Models (HEMM) to systematically evaluate the
capabilities of multimodal foundation models across a set of 3 dimensions:
basic skills, information flow, and real-world use cases. Basic multimodal
skills are internal abilities required to solve problems, such as learning
interactions across modalities, fine-grained alignment, multi-step reasoning,
and the ability to handle external knowledge. Information flow studies how
multimodal content changes during a task through querying, translation,
editing, and fusion. Use cases span domain-specific challenges introduced in
real-world multimedia, affective computing, natural sciences, healthcare, and
human-computer interaction applications. Through comprehensive experiments
across the 30 tasks in HEMM, we (1) identify key dataset dimensions (e.g.,
basic skills, information flows, and use cases) that pose challenges to today's
models, and (2) distill performance trends regarding how different modeling
dimensions (e.g., scale, pre-training data, multimodal alignment, pre-training,
and instruction tuning objectives) influence performance. Our conclusions
regarding challenging multimodal interactions, use cases, and tasks requiring
reasoning and external knowledge, the benefits of data and model scale, and the
impacts of instruction tuning yield actionable insights for future work in
multimodal foundation models.