Title: Personalized Visual Representation Alignment for Generative Multimodal Recommendation
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Sequential Recommendation, Multimodal RecommeXx

Abstract: With the development of Vision-Language Models (VLMs) for multimodal understanding, recommender systems have increasingly leveraged them to process heterogeneous sources of user-interacted items for recommendation. By fine-tuning VLMs on user interaction data, prior works have adapted these models to capture user preferences, enabling personalized multimodal recommendation. Despite these advances, however, we identify two key limitations: 1) the visual features directly extracted by vision encoders (e.g., CLIP) are insufficient for capturing personalized user preferences, as such encoders are generally trained for generic visual perception rather than capturing user-specific preferences; and 2) existing VLM-based methods often underutilize visual features of user-interacted items in later LLM layers, relying instead on textual descriptions for recommendation—an unexpected bias that diminishes the contribution of visual features. To address these two limitations, we propose PerVRA, a VLM-based recommendation model consisting of a Personalized Visual Representation Learning (PVRL) module and a Personalized Multimodal Alignment (PMA) module. Specifically, we employ dual contrastive learning, where each module is equipped with its own contrastive objective: The PVRL module learns personalized visual features from user interaction history, while the PMA module enhances the contribution of visual features to the VLMs by explicitly aligning them with text features. Extensive experiments on real-world Amazon and H\&M Fashion datasets demonstrate that PerVRA consistently outperforms strong VLM-based methods over diverse personalized tasks. Moreover, our ablation studies show that addressing these two limitations is critical for building effective VLM-based recommender systems.

TL;DR: We propose personalized multimodal representation learning for personalized recommendation.