Title: LaMP: When Large Language Models Meet Personalization
Benchmark: LaMP
Summary: The paper introduces LaMP, a benchmark designed to train and evaluate large language models (LLMs) for personalized output generation. It includes personalized text classification and generation tasks, along with retrieval-augmented personalization methods that significantly improve performance in both zero-shot and fine-tuned settings.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: personalization
Phenomenon defined by authors: Yes
Phenomenon definition: "Personalizing language models can be defined as conditioning the model’s output on
a user u, represented by a user profile. In LaMP, we define user profile as the user’s historical data, i.e., the past input and personalized outputs produced by or approved by the user."
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Classify or generate outputs conditioned on user-specific information (user profiles). Tasks include predicting citation preference, assigning movie tags, rating products, generating news headlines, scholarly titles, email subjects, and paraphrasing tweets. Each dataset in the LaMP benchmark has two evaluation settings: (a) user-based data split to test personalization for new users and (b) a time-based data split to test personalization for future interactions of existing users.
Task item definition: An input sequence, a classification or generation target output, and the user profile 
Task definition detail: All 7 tasks are formally defined:

3 Personalized Text Classification tasks:
- Personalized Citation Identification (if the user u writes a paper x, the language model must determine which of two candidate papers u will cite in x)
- Personalized Movie Tagging (given a movie description x and a user’s historical movie-tag pairs, a language model must predict one of 15 tags for x)
- Personalized Product Rating (given the user u's historical review and rating pairs and an input review x, the model must predict an integer rating from 1 − 5)

4 Personalized Text Generation tasks:
- Personalized News Headline Generation (generate a headline for an input news article,
given a user profile of the authors’ historical articletitle pairs)
- Personalized Scholarly Title Generation (generate titles for an input article, given a user profile of historical article-title pairs for an author)
- Personalized Email Subject Generation (generate an email subject for an input email message, given historical email-subject pairs authored by a user)
- Personalized Tweet Paraphrasing (generate a tweet in the style of a user given an input tweet, and a user profile of historical tweets by the user)
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: Test split: 15,488 (user-based split) + 13,657 (time-based split) = 29,145 total test examples
Dataset size notes: Train split: 75,008 (user-based split) + 76,475 (time-based split) = 151,483 total training examples  // Validation split: 14,970 (user-based split) + 12,158 (time-based split) = 27,128 total validation examples  
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF)
Metric definition detail: accuracy, F1, MAE, RMSE, ROUGE-1 and ROUGE-L
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes, implicitly
Author validity detail: - Face: "LaMP provides a rich environment for developing personalized NLP models"
- Predictive/Concurrent: The authors perform empirical experiments showing that personalized models outperform non-personalized ones.
- Content: The benchmark spans seven tasks across classification and generation, covering a variety of realistic personalization scenarios (e.g., movie tagging, tweet paraphrasing). Moreover, the authors also detail the task definitions, data sources, and input/output structure.
- Convergent/Discriminant: no comparisons made
Taxonomy: Alignment / Alignment
New taxonomy: General Capability / Alignment
Procedural notes: Leaderboard: https://lamp-benchmark.github.io/leaderboard