Title: MIP-Bench: Can LLMs Implicitly Personalize Responses Using Long-Term Memory?
Primary area: datasets and benchmarks
Keywords: large language models, personalization, long-term memory

Abstract: Implicit Personalization (IP) is the task of tailoring responses to individual users by implicitly inferring their personal context. Prior studies in IP typically infer context from a single prompt. However, as interactions accumulate, users expect Memory-driven Implicit Personalization (MIP), where models implicitly leverage contexts from users' long-term interaction histories to provide more helpful responses. MIP introduces two unique challenges: (i) identifying sparse yet relevant personal contexts within extensive historical interactions, and (ii) understanding how varying personal contexts influence preferences among plausible answers to differentiate responses between multiple users. To navigate these challenges, we introduce MIP-Bench, the first benchmark to evaluate MIP in large language models (LLMs). Our experiments reveal that recent LLMs struggle with MIP, primarily due to difficulties in identifying and retrieving relevant personal context from memory. Furthermore, our new distribution-level evaluation framework shows that even models with strong instance-level performance often fail to differentiate responses across users, defaulting to generic or overly broad outputs rather than personalized ones.

TL;DR: We introduce MIP-Bench, the first benchmark to evaluate memory-driven implicit personalization in large language models (LLMs).