VisualLens: Personalization through Visual History
Authors: name: Wang Bill Zhu; name: Deqing Fu; name: Kai Sun; name: Yi Lu; name: Zhaojiang Lin; name: Seungwhan Moon; name: Kanika Narang; name: Mustafa Canim; name: Yue Liu; name: Anuj Kumar; name: Xin Luna Dong
Published: 2024-11-25 01:45:42

Abstract: We hypothesize that a user's visual history with images reflecting their
daily life, offers valuable insights into their interests and preferences, and
can be leveraged for personalization. Among the many challenges to achieve this
goal, the foremost is the diversity and noises in the visual history,
containing images not necessarily related to a recommendation task, not
necessarily reflecting the user's interest, or even not necessarily
preference-relevant. Existing recommendation systems either rely on
task-specific user interaction logs, such as online shopping history for
shopping recommendations, or focus on text signals. We propose a novel
approach, VisualLens, that extracts, filters, and refines image
representations, and leverages these signals for personalization. We created
two new benchmarks with task-agnostic visual histories, and show that our
method improves over state-of-the-art recommendations by 5-10% on Hit@3, and
improves over GPT-4o by 2-5%. Our approach paves the way for personalized
recommendations in scenarios where traditional methods fail.