Personalized Multimodal Large Language Models: A Survey
Authors: name: Junda Wu; name: Hanjia Lyu; name: Yu Xia; name: Zhehao Zhang; name: Joe Barrow; name: Ishita Kumar; name: Mehrnoosh Mirtaheri; name: Hongjie Chen; name: Ryan A. Rossi; name: Franck Dernoncourt; name: Tong Yu; name: Ruiyi Zhang; name: Jiuxiang Gu; name: Nesreen K. Ahmed; name: Yu Wang; name: Xiang Chen; name: Hanieh Deilamsalehy; name: Namyong Park; name: Sungchul Kim; name: Huanrui Yang; name: Subrata Mitra; name: Zhengmian Hu; name: Nedim Lipka; name: Dang Nguyen; name: Yue Zhao; name: Jiebo Luo; name: Julian McAuley
Published: 2024-12-03 03:59:03

Abstract: Multimodal Large Language Models (MLLMs) have become increasingly important
due to their state-of-the-art performance and ability to integrate multiple
data modalities, such as text, images, and audio, to perform complex tasks with
high accuracy. This paper presents a comprehensive survey on personalized
multimodal large language models, focusing on their architecture, training
methods, and applications. We propose an intuitive taxonomy for categorizing
the techniques used to personalize MLLMs to individual users, and discuss the
techniques accordingly. Furthermore, we discuss how such techniques can be
combined or adapted when appropriate, highlighting their advantages and
underlying rationale. We also provide a succinct summary of personalization
tasks investigated in existing research, along with the evaluation metrics
commonly used. Additionally, we summarize the datasets that are useful for
benchmarking personalized MLLMs. Finally, we outline critical open challenges.
This survey aims to serve as a valuable resource for researchers and
practitioners seeking to understand and advance the development of personalized
multimodal large language models.