LaMP: When Large Language Models Meet Personalization (2024)

Abstract: This paper highlights the importance of personalization in large language models and introduces the LaMP benchmark — a novel benchmark for training and evaluating language models for producing personalized outputs. LaMP offers a comprehensive evaluation framework with diverse language tasks and multiple entries for each user profile. It consists of seven personalized tasks, spanning three text classification and four text generation tasks. We additionally propose two retrieval augmentation approaches that retrieve personal items from each user profile for personalizing language model outputs. To this aim, we study various retrieval models, including term matching, semantic matching, and time-aware methods. Extensive experiments on LaMP for zero-shot and fine-tuned language models demonstrate the efficacy of the proposed retrieval augmentation approach and highlight the impact of personalization in various natural language tasks.

heading: 1 Introduction
text: The recent development of large language models (LLMs) has revolutionized natural language processing (NLP) applications. As the use of LLMs, such as GPT-4 (OpenAI, 2023), in real-world applications evolves, personalization emerges as a key factor in meeting the user’s expectations for tailored experiences that align with their unique needs and preferences (Huang et al., 2022). Personalization has been widely studied by various communities, including the information retrieval (IR) and human-computer interaction (HCI) communities, often with applications to search engines and recommender systems (Fowler et al., 2015; Xue et al., 2009; Naumov et al., 2019). Recent work has also highlighted the impact and concerns associated with personalizing LLMs and tying it to ongoing work on alignment (Kirk et al., 2023). Despite this and the importance of personalization
in many real-world problems, developing and evaluating LLMs for producing personalized responses remain relatively understudied. To bridge this gap, this paper underscores the importance of personalization in shaping the future of NLP systems and takes a first step towards developing and evaluating personalization in the context of large language models by introducing the LaMP benchmark1 — a comprehensive and diverse benchmarks of personalized text classification and generation tasks.
While many existing well-known NLP benchmarks, such as GLUE (Wang et al., 2018), SuperGLUE (Wang et al., 2019), KILT (Petroni et al., 2021), and GEM (Gehrmann et al., 2021) have led to significant progress in various NLP tasks, they have often taken the dominant NLP approach of “one-size-fits-all” to modeling and evaluation, and do not allow the development of models that adapt to the specific needs of end users – limiting extensive research on personalization in NLP tasks. In contrast, LaMP offers a comprehensive evaluation framework incorporating diverse language tasks that require personalization. LaMP consists of three personalized text classification tasks: (1) Personalized Citation Identification (binary classification), (2) Personalized Movie Tagging (categorical classification with 15 tags), and (3) Personalized Product Rating (ordinal classification from 1 to 5-star rating for e-commerce products). Further, LaMP includes four text generation datasets: (4) Personalized News Headline Generation, (5) Personalized Scholarly Title Generation, (6) Personalized Email Subject Generation, and (7) Personalized Tweet Paraphrasing. For these seven tasks, we explore the two dominant settings in personalization: (a) personalization for new users with a userbased data split and (b) personalization for future interactions of existing users with a time-based data split. Therefore, LaMP provides a rich environment
1LaMP stands for Language Model Personalization.
7370
for developing personalized NLP models. To foster research in this area, we release the LaMP benchmark, the data construction, evaluation scripts, and a leaderboard.
For personalizing the language model outputs, a straightforward solution is to incorporate the user profile into a language model prompt. However, user profiles are often large and exceed the length limitations of large language models. Even as such limitations are relaxed with evolving technology, the cost of processing large input sequences is considerable. Therefore, we propose two retrieval augmentation solutions for LLM personalization, in which for each test input, we retrieve items from the user profile to be included in the LLM prompt for personalization. The first approach uses in-prompt augmentation (IPA) for personalization, and the second approach encodes each personal item separately and integrate them later in the decoder using the fusion-in-decoder model of Izacard and Grave (2021). We demonstrate that using this approach, the performance of language models improves on all datasets in the LaMP benchmark. Based on this retrieval augmentation solution, we evaluate different retrievers for personalized prompt construction and establish benchmark results for fine-tuned and zero-shot language models. The empirical findings of our research reveal that the process of fine-tuning a language model utilizing our personalized augmentation technique yields a noteworthy relative average enhancement of 23.5% across the benchmark. Even in zero-shot settings where an off-theshelf LLM without fine-tuning (e.g., FlanT5-XXL) is used, utilizing our proposed method results in a relative average improvement of 12.2% across the tasks. Finally, this paper smooths the path towards developing advanced user-centric NLP systems.

heading: 2 The LaMP Benchmark
text: Problem Formulation. Generative language models often take an input x and predict the most probable sequence tokens y that follows x. Personalizing language models can be defined as conditioning the model’s output on a user u, represented by a user profile. In LaMP, we define user profile as the user’s historical data, i.e., the past input and personalized outputs produced by or approved by the user, Pu = {(xu1, yu1), (xu2, yu2), · · · , (xumu , yumu)}. Therefore, each data entry in the LaMP benchmark consists of three components: an input sequence x
that serves as the model’s input, a target output y that the model is expected to produce, and a profile Pu that encapsulates any auxiliary information that can be used to personalize the model for the user.
Overview of LaMP. Given the above problem formulation, we develop the LaMP benchmark that aims to assess the efficacy of LLMs in producing personalized outputs y, based on inputs x and userspecific information Pu. Outputs y of different types result in seven diverse tasks spanning personalized text classification and generation:
• Personalized Text Classification (1) Personalized Citation Identification (2) Personalized Movie Tagging (3) Personalized Product Rating
• Personalized Text Generation (4) Personalized News Headline Generation (5) Personalized Scholarly Title Generation (6) Personalized Email Subject Generation (7) Personalized Tweet Paraphrasing

heading: 2.1 Tasks Definitions
text: Next, we overview of each task used in LaMP and detail data construction in Appendix A.

heading: LaMP-1: Personalized Citation Identification
text: The citation behavior of researchers is dependent on their interests and is commonly used to evaluate and develop personalized systems for recommending papers (Färber and Jatowt, 2020). This task recasts citation recommendation as a binary classification task and assesses the ability of a language model to identify user preferences for citations. Specifically, if the user u writes a paper x, a language model must determine which of two candidate papers u will cite in x (see Figure 3).
To generate data samples, we leverage the Citation Network Dataset (V14) (Tang et al., 2008), which comprises information on scientific papers, authors, and citations. For this task, the profile of each user encompasses all the papers they have authored. We retain only the title and abstract of each paper in the user’s profile.

heading: LaMP-2: Personalized Movie Tagging
text: Users tagging behavior for media such as movies and books are known to be idiosyncratic and depends on their understanding of the tag and the aspects of the item they focus on. This has motivated a large body of work on personalized tagging
(Gupta et al., 2010). We use this task to evaluate the ability of language models to make tag assignments for a movie contingent on the user’s historical tagging behavior. Specifically, given a movie description x and a user’s historical movie-tag pairs, a language model must predict one of 15 tags for x. We obtain tag assignments from the MovieLens dataset (Harper and Konstan, 2015). Additionally, we obtain movie descriptions from MovieDB.2

heading: LaMP-3: Personalized Product Rating
text: Product reviews commonly express a nuanced set of user preferences for a product and which in turn determine their rating for the product. Predicting ratings based on user reviews has been studied extensively in personalized sentiment prediction tasks (Mireshghallah et al., 2022). While this is commonly treated as a regression task, to use autoregressive language models, we frame it as a multi-class classification task. Specifically, given the user u’s historical review and rating pairs and an input review x, the model must predict an integer rating from 1− 5. We construct our dataset from a dataset of Amazon reviews (Ni et al., 2019).

heading: LaMP-4: Personalized News Headline Generation
text: Authors writing displays distinct stylistic elements influenced by both personal and social factors (Zhu and Jurgens, 2021). Journalists authoring headlines are likely to balance between faithfully representing an article, appealing to their readers, and maintaining their own identity. This offers a useful testbed for personalized text generation. Here, we evaluate the ability of a language model to capture
2https://www.themoviedb.org/
the stylistic patterns of an author by requiring it to generate a headline for an input news article, x, given a user profile of the authors’ historical articletitle pairs. To create a dataset, we use a collection of Huffington Post articles (Misra, 2022; Misra and Grover, 2021).

heading: LaMP-5: Personalized Scholarly Title Generation
text: As with LaMP-4, the generation of titles for research articles offers a test bed for personalized text generation but varies in text domain. In this task, we require language models to generate titles for an input article x, given a user profile of historical article-title pairs for an author. Here, only use article abstracts. We create our dataset from the Citation Network Dataset (V14) (Tang et al., 2008) also used for LaMP-1.

heading: LaMP-6: Personalized Email Subject Generation
text: Similar to LaMP-4 and 5, generating email subjects also provides a valuable test bed for personalized text generation. Email assistance is also known to be a task that significantly benefits from personalization (Trajanovski et al., 2021). Here, we require language models to generate an email subject for an input email message x, given historical emailsubject pairs authored by a user. For this task, we leverage a private dataset of emails the Avocado Research Email Collection (Oard, Douglas et al., 2015). Given its private nature this is unlikely to be contained in pre-training data providing a meaningful challenge for language models.
1

heading: LaMP-7: Personalized Tweet Paraphrasing
text: Social media posts adhere strongly to various personal stylistic patterns of authors (Zhu and Jurgens, 2021). Here, we construct a personalized tweet paraphrasing task and require models to generate a tweet in the style of a user given an input tweet x, and a user profile of historical tweets by the user. To construct this task we use data from the Sentiment140 dataset (Go et al., 2009). Figure 3 provides examples of tasks 1-7 in LaMP.

heading: 2.2 Data Splits
text: To enable evaluation in common personalization settings, LaMP offers two different data splitting settings: (1) user-based separation and (2) timebased separation. In user-based separation (denoted as LaMP-iU for Task i), train/validation/test splits are made by partitioning across users, ensuring that no shared users appear across splits. This strategy measures personalization for new users.
In time-based separation (denoted as LaMP-iT for Task i), train/validation/test splits are made by partitioning user items ordered by time. The most recent user items are chosen to create the inputoutput pairs, with older items serving as user profiles. Appendix A contains additional details, and Table 1 reports dataset sizes.

heading: 2.3 Evaluation
text: For evaluating classification tasks, we use Accuracy for LaMP-1 (balanced binary classification), Accuracy/F1 for LaMP-2 (multi-class classification), and MAE/RMSE for LaMP-3 (ordinal multiclass classification). Following previous works (Zhou and Bhat, 2021; Panthaplackel et al., 2022) on text generation, we use Rouge-1/Rouge-L (Lin, 2004) as evaluation metrics for the text generation generation tasks (LaMP-4 to LaMP-7).

heading: 3 Retrieval Augmentation for Personalizing LLMs
text: To personalize a language model two broad strategies may be explored: (1) fine-tuning the LM for each user and (2) prompting a shared LM with user specific input or context. The former approach necessitates substantial computational resources, especially for fine-tuning larger LLMs. Moreover, accommodating personalized LLMs for each user in industry-scale systems encompassing millions or billions of users necessitates a significant storage and serving capacity. Therefore, we focus on developing strategies for training models personalized via user-specific inputs.
Each task in LaMP, each user profile consists of a potentially large collection of data points. Given the inherent context length constraint of many LLMs and the cost of processing long sequences, we incorporate a subset of these data points as input prompts. Further, not all entries within a user profile are necessarily relevant to the specific input at hand. To do this, we propose solutions based on retrieval augmentation (See Figure 1). This framework selectively extracts pertinent information from the user profile that is relevant to the current unseen test case and generates model predictions conditioned on this information.
Specifically, for a given sample (xi, yi) for user u, we employ three primary components: (1) a query generation function ϕq that transforms the input xi into a query q for retrieving from the user u’s profile, (2) a retrieval model R(q, Pu, k) that accepts a query q, a user profile Pu and retrieves k most pertinent entries from the user profile, and (3) a prompt construction function ϕp that assembles a personalized prompt for the user u based on input xi and the retrieved entries. For retrieval augmentation, we explore two strategies: (1) In-Prompt Augmentation (IPA) and (2) Fusion-in-Decoder (FiD) (Izacard and Grave, 2021). The input to both approaches constructs inputs, x̄i, using R to select k items from the user profile Pu:
x̄i = ϕp(xi,R(ϕq(xi), Pu, k)) (1) where we use (x̄i, yi) to train or evaluate the language models. With FiD, LLMs receive multiple inputs, each of which is encoded separately within its encoder. These separate encodings are then merged together in the decoder. Here, the inputs {x̄i1, ..., x̄ik} for the encoder are derived as:
x̄ij = ϕp(xi, dij) (2)
where dij is the j’th retrieved item using retrieval model from the user profile (i.e, R(ϕq(xi), Pu, k)). Note that, IPA and FiD offer different tradeoffs. FiD necessitates training of the language model while IPA may be applied without training. Further, while FiD can only be used with encoder-decoder models, IPA can be used across architectures. However, FiD allows us to incorporate more items from the user profile into the LLM’s input.
We explore various choices for the retrieval model R. In our experiments, we study a strong term matching model, BM25 (Robertson et al., 1995), a state-of-the-art pre-trained dense retrieval model, Contriever (Izacard et al., 2022), a retrieval model that returns most recent profile entries in descending order (i.e., Recency), and a Random document selector from the user profile. The prompt construction function ϕp concatenates the instruction for each task, the input sequence, and the user profile. The specific prompts are presented in Table 5. For the ϕq function, we use the target input for each task as the query (see Figure 3).

heading: 4 Experiments
text: This section describes our experiments, results, and findings on the LaMP benchmark.

heading: 4.1 Experimental Setup
text: For training FiD and the generative model in IPA we leverage AdamW (Loshchilov and Hutter, 2019) with a learning rate of 5 × 10−5 and a batch size 64 and set 5% of the total training steps as warmup using a linear scheduler. A weight decay of 10−4 is incorporated to prevent overfitting. The maximum input and output lengths is set to 512 and 128 tokens, respectively. We train both classification and generation models for 10 and 20 epochs, respectively. A FlanT5-base3 (Chung et al., 2022) model is used for all experiments, unless explicitly stated otherwise (in experiments with LLMs, we use FlanT5-XXL). We employ beam search (Freitag and Al-Onaizan, 2017) with a beam size of 4. All models are implemented with Huggingface transformers and evaluations are conducted using the evaluate library. All the experiments are conducted on a single Nvidia RTX8000 GPU with 49GB of GPU memory and 128GB of CPU memory for maximum 3 days on each experiment. All the results reported are based on a single run.
3https://huggingface.co/google/flan-t5-base

heading: 4.2 Fine-Tuning Retrieval Augmented LMs for Personalization
text: In the first sets of experiments, we establish baseline personalization results for a fine-tuned language model. We also investigate the impact of employing various retrieval techniques and the effect of retrieving different quantities of entries from a user profile. This analysis aims to provide insights into the efficacy of diverse retrieval methods and the potential benefits of adjusting the number of retrieved entries for personalization tasks.

heading: Impact of Retrievers on Retrieval-Augmented
text: Personalization Models. Here, we study different implementations of R with a fine-tuned FlanT5base model for generating personalized output: (1) a baseline random selector from the user profile, (2) BM25 (Robertson et al., 1995), (3) Contriever4 (Izacard et al., 2022), and (4) Recency, in which we select the latest item in the user profile based on time (only for time-based separation setting). BM25 is considered as a robust and strong termmatching retrieval model and Contriever is a pretrained dense retrieval model.
The results of this experiment are shown in Table 2 for user-based separation and in Table 3 for time-based separation. The results suggest that personalization improves the performance for all tasks within the LaMP benchmark. In most cases, even a random selection of documents from the user profile and the creation of personalized prompts leads to performance improvements compared to non-personalized prompts given to the LM. Note that non-personalized prompt can be achieved with no retrieval augmentation (No-Retrieval) or with augmentation with a random item from all user profiles. Results for more non-personalized baselines are presented in Appendix E.
When retrieving one document per user for personalizing the language model’s output, Contriever demonstrates the best performance for most classification tasks (i.e., LaMP-1U, LaMP-2U, LaMP3U, LaMP-1T, and LaMP-2T). Recency only outperforms Contriever in the LaMP-3T. Note that recency is considered as a simple yet strong personalization signal in search and recommendation (Fader et al., 2005; Reinartz and Kumar, 2000, 2003). For text generation, Contriever performs best for Personalization News Headline Generation (LaMP-4U) and Personalized Tweet Paraphrasing
4https://huggingface.co/facebook/contriever
(LaMP-7U) in user-based separation setting. For Email Generation and Scholarly Title Generation tasks (LaMP-5U and LaMP-6U), BM25 demonstrates superior performance. Both BM25 and Contriever outperform a random profile selector in all LaMP datasets. For the time-based separation setting, Contriever outperforms other methods in all generation tasks except News Headline Generation (LaMP-4T), where recency performs better.
Generally, the results indicate that incorporating any information from the user profile into the input is not sufficient, but rather selecting the most relevant and/or recent information is crucial. This underscores the importance of careful considera-
tion in selecting and incorporating pertinent user profile elements in LLM prompts. There is no clear winner among the retrieval models we study and an ensemble of relevance and temporal signals for personalization should be studied in the future.

heading: Impact of the Number of Retrieved Items, k,
text: on LLM Personalization. Each sample within this benchmark consists of a substantial number of user profile entries. As such, exploring the impact of incorporating multiple entries to augment the input of the language model can provide valuable insights into addressing the unresolved challenges posed by this benchmark. For the sake of space, we focus on our In-Prompt Augmentation (IPA) ap-
proach for personalization and depict the model’s performance w.r.t different profile sizes in Figure 2. This experiment uses the best retriever from Tables 2 and 3 across various tasks, while varying the number of retrieved entries from user profiles. The results suggest that increasing the number of retrieved items leads to improved performance in downstream tasks. However, some tasks experience a decline in performance. Given the finite context size of language models, exploring approaches to generate a unified prompt from multiple user entries may represent promising future work.

heading: Impact of Tuning Retriever Hyperparameters.
text: Based on the performance on the validation set for each dataset, we tuned two parameters: (1) the retrieval model (BM25 vs. Contriever vs. Recency) for IPA and FiD, and (2) retrieved items count (k) for IPA. We consistently utilize 16 documents for FiD, as we do not observe much variance in the results. Both IPA and FiD approaches use FlanT5base. For hyperparameter tuning, we used the following metrics on the development sets: Accuracy for LaMP-1 and LaMP-2, MAE for LaMP-3, and ROUGE-1 for all text generation tasks. The results for this tuned model are presented in the last two columns of Table 2 and Table 3. As expected, the tuned model outperforms the other models on all datasets. For text classification tasks, FiD surpasses the performance of IPA in all datasets, with the exception of LaMP-1T. Conversely, IPA exhibits superior performance across all text generation datasets.

heading: 4.3 Zero-Shot Personalized Results for LLMs
text: With the widespread adoption of employing LLMs with no fine-tuning in contemporary research, we conduct an evaluation of two such models on our benchmark.5 Particularly, we leverage GPT 3.5 (alias gpt-3.5-turbo or ChatGPT6) and FlanT5XXL (Chung et al., 2022). FlanT5-XXL comprises 11B parameters, however, the size of GPT-3.5 is unknown (GPT3 consists of 175B parameters). For evaluation, we provide each model with the inputs corresponding to individual tasks and assess their performance based on the generated outputs. In classification tasks, if the produced output does not correspond to a valid class, we resort to calculating the similarity between each class label and the generated output utilizing BERTScore (Zhang* et al., 2020). Thus, we assign the most similar label to the generated output as the output for the given input. GPT-3.5 generated out-of-the-label predictions 8%, 4%, 6%, 4%, 2%, and 4% of the time for the LaMP-1U, LaMP-1T, LaMP-2U, LaMP-2T, LaMP-3U, and LaMP-3T tasks, respectively. On the other hand, FlanT5-XXL predictions are consistently among the questioned labels.
Table 4 shows the result of LLMs on this benchmark in a zero-shot scenario. The results show that, except for the Personalized Tweet Paraphrasing task, using the user’s profile with LLMs im-
5As previously stated, FiD approach cannot be utilized with untrained models. Consequently, the experiments conducted in this section pertain solely to IPA method.
6https://openai.com/blog/chatgpt
proves their performance on this benchmark in a zero-shot setting. The outcomes in Tables 2 and 3 show the results for FlanT5-base, a 250M parameter model, fine-tuned on each task. Table 4 presents the zero-shot application of LLMs. These findings indicate that fine-tuning smaller models on downstream tasks leads to enhanced performance in comparison to zero-shot performance of LLMs.
Finally, it is crucial to highlight that the observed outcomes, which indicate superior performance of FlanT5-XXL over GPT-3.5, should not be construed as an inherent deficiency of the latter model. The efficacy of LLMs is extensively contingent upon the caliber and configuration of the input prompts. It is worth noting that prompt engineering, which plays a significant role in performance of LLMs, is not the central objective of this study. Consequently, any disparities in performance must be evaluated in light of this contextual information.

heading: 5 Research Problems Enabled by LaMP
text: LaMP can facilitate research in several areas, including but not limited to:
Prompting Language Models for Personalization. The integration of user profiles into language models can be approached using hard prompts, but their limited context size makes it difficult to include lengthy user profile entries. Exploring different prompts for personalization could be interesting. An alternative solution is generating personalization prompts based on the user profile, instead of relying on retrieved entries. Furthermore,
the use of soft prompts (Lester et al., 2021) can be helpful for personalizing language models.

heading: Evaluation of Personalized Text Generation.
text: The commonly used evaluation metrics for text generation, whether syntactical (Lin, 2004; Banerjee and Lavie, 2005; Papineni et al., 2002) or semantical (Zhang* et al., 2020), do not incorporate the user into their evaluation process. Consequently, such metrics may not be entirely suitable for evaluating personalized text generation problems. Exploring new evaluation metrics that account for the user’s preferences can benefit this area of research.
Learning to Retrieve from User Profiles. Learning to rank has been widely explored in various retrieval scenarios. Optimizing ranking models that select personalized entries for the sake of personalized text classification and/or generation would be a potentially impactful research direction.

heading: 6 Related Work
text: Personalization has been well studied for information access problems, with the organization of the Netflix Challenge and its associated datasets representing an important driver of academic focus on personalization (Konstan and Terveen, 2021). It also represents an important element of large-scale industry recommender systems (Davidson et al., 2010; Das et al., 2007; Xu et al., 2022) and has also been extensively studied for search applications (Bennett et al., 2012; Dumais, 2016; Croft et al., 2001; Tabrizi et al., 2018; Zeng et al., 2023),
in contexts ranging from query auto-completion (Jaech and Ostendorf, 2018) to collaborative personalized search (Xue et al., 2009). We refer readers to Rafieian and Yoganarasimhan (2023) for an overview of this line of work. Here, we cover personalization in NLP, focusing on research datasets.
Personalization has been examined extensively for dialogue agents (Wu et al., 2021; Zhang et al., 2018; Mazaré et al., 2018). Compared to other NLP tasks. This focus likely stems from the importance of tailoring dialogue to users and conditioning generated utterances on specific personas. Given the lack of real conversational data, some work has constructed dialogue data for users by promoting crowd-workers to author dialogues based on specific personas (Zhang et al., 2018), and through extracting user attributes and utterances from Reddit (Mazaré et al., 2018; Wu et al., 2021) and Weibo (Zhong et al., 2022; Qian et al., 2021). To leverage more realistic conversational data, recent work of Vincent et al. (2023) annotate a dataset of movie dialogues with narrative character personas and posit the potential for using LLMs for dialogue generation conditioned on these personas. Other work has also leveraged publicly available reviews and recipes to explore personalization for review (Li and Tuzhilin, 2019) and recipe generation (Majumder et al., 2019). Wuebker et al. (2018) explore parameter efficient models for personalized translation models. Finally, Ao et al. (2021) presents a personalized headline generation dataset constructed from realistic user interaction data on Microsoft News. This is closely related to the LaMP-4 task, which focuses on personalization for authors rather than readers. LaMP presents resources for the tasks which have seen lesser attention than those based on dialogue – expanding the underexplored space of personalizing text classification/generation systems (Flek, 2020; Dudy et al., 2021).
While a body of work has focused on user-facing applications, others have explored personalization for more fundamental problems in language modeling. They have used openly available user data on Reddit (Welch et al., 2022), Facebook, Twitter (Soni et al., 2022), and other blogging websites (King and Cook, 2020). Besides pre-training LMs for personalization, Soni et al. (2022) explores applying a personalized LM for downstream tasks in stance classification and demographic inference. Similarly, other work has explored personalized sentiment prediction on publicly available Yelp and IMDB data (Mireshghallah et al., 2022; Zhong
et al., 2021) – this work bears a resemblance to the LaMP-3 task and ties back to rating prediction explored in recommendation tasks. Finally, Plepi et al. (2022) examines the application of personalization methods to modeling annotators in a classification task reliant on modeling social norms – making an important connection between personalization and an emerging body of work on accommodating human label variation in NLP (Rottger et al., 2022; Gordon et al., 2022; Plank, 2022).

heading: 7 Conclusion
text: This paper presented a novel benchmark named LaMP for training and evaluating language models for personalized text classification and generation. LaMP consists of seven datasets: three classification and four generation datasets. We proposed retrieval augmentation solutions for personalizing LLMs. Notably, we studied two augmentation approaches: in-prompt augmentation (IPA) and fusion-in-decoder (FiD). We performed extensive experiments using various LLMs and retrieval techniques for selecting user profile entries for producing personalized prompts. We demonstrated that our LLM personalization approaches can lead to 12.2% average performance improvements across datasets on zero-shot setting, and 23.5% with finetuning. Finally, we underscore the paramount importance of personalization in the current era dominated by large language models. We firmly believe that the future of natural language processing systems lies in a user-centric approach, tailoring solutions to individual needs for optimal effectiveness.

heading: Acknowledgments
text: This work was supported in part by the Center for Intelligent Information Retrieval, in part by NSF grant #2143434, in part by the Office of Naval Research contract number N000142212688, in part by Google, Microsoft, and Lowe’s. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsors.

heading: Limitations
text: LaMP comes with certain limitations arising from task definitions, data leakage into LLM pre-training collections, effective evaluations for personalized generations and broader privacy concerns associated with personalization.
Task definitions Although all tasks are designed to assess language models’ proficiency in personalization, certain tasks could be better grounded in realistic scenarios and real-world applications. For instance, framing the Personalized Citation Identification task as a binary classification problem might not accurately represent real-world situations, where individuals generally need to interact with a more extensive array of articles. Additionally, while Personalized Product Rating is intrinsically linked to predicting user satisfaction, the approach may not be entirely realistic, as reviews in real-world contexts are often accompanied by a numerical rating, rendering direct score prediction less relevant. That being said, LaMP creates an environment for evaluating the abilities of LMs in producing personalized outputs.
Leakage to LLM pretraining data The data used for creating the LaMP benchmark mostly consists of publicly available data on the Web, e.g., public tweets, scholarly articles, news articles, and product reviews. We should take this into consideration that some of this data may have been observed by the large language models during their pretraining process. Therefore, they may even perform poorer in unseen cases compared to what we observe from the results on most the LaMP datasets. For this reason, we included the Avocado dataset for Personalized Email Subject Generation as this is not publicly available on the Web and we expect that language models do not use this dataset for pretraining given the restrictions on the data usage agreement.
Evaluating personalized generation The majority of text generation tasks addressed in this research employ short text generation as it offers greater convenience for evaluation purposes. Welldefined metrics, such as ROUGE and BLEU scores, are readily available to assess the quality of short text generation. Conversely, evaluating long text generation poses significant challenges due to its subjective nature, absence of a definitive reference, the necessity for coherence and consistency, contextual comprehension, varied output, semantic and factual accuracy, as well as the limitations of conventional metrics. Evaluators must account for multiple factors, encompassing structural integrity, clarity, effective employment of context, creativity, and subjectivity. Attaining consistent and objective evaluations proves arduous, as it heavily relies on human judgment, which can introduce biases.
Privacy and personalization Finally, we urge the readers to be mindful of privacy implication of LLM personalization. Several studies have shown successful membership attacks against deep learning models (including LLMs) (Mattern et al., 2023; Shokri et al., 2017), thus using personal and private data in fine-tuning may put the privacy of some users at risk. Despite its importance, this paper does not study privacy issues and further analyses are required to ensure how fine-tuned personalized models can preserve the privacy of users. Note that we do not have privacy concerns for the proposed retrieval augmentation approach when a zero-shot language model is used, assuming that the zeroshot language model is hosted in-house or at a trusted third party.

heading: A Data Creation Details for Tasks in the LaMP benchmark
text: This section explains the details behind creating inputs, outputs, and profile entries for each task in the LaMP benchmark. Note that all datasets are in English language.

heading: A.1 User-based Separation Setting
text: Personalized Citation Identification. To generate data samples, we leverage the Citation Network Dataset (V14) (Tang et al., 2008), which comprises information on scientific papers, authors, and citations. The dataset does not provide demographic details of the users in the data. We select all papers from this dataset that meet the following criteria: 1) they are written in English, 2) they contain at least one reference and one author, and 3) they include an abstract. Subsequently, we group papers based on their authors and only consider authors who have written at least 50 papers. For each author, we randomly select one of their papers and one of its cited references. For negative document selection, we randomly choose one of the first author’s
co-authors and one of the papers they have cited in one of their papers, which has not been cited by the first author. If no such author exists, we randomly select an author and repeat this process. Finally, we construct the input, output, and profile of the generated samples for this task, employing the template depicted in Figure 3 in Appendix B. After creating the samples for all users, we divide users into the train, validation, and test sets for this task.
Personalized Movie Tagging. To construct our dataset for this task, we leverage the MovieLense dataset (Harper and Konstan, 2015) obtained from the MovieLense website7. This dataset includes the tags that individual users have attributed to each film. However, it does not contain the descriptions or summaries of the films. To acquire these, we retrieve the overviews of each film in this dataset from the Movie Database website8. The MovieLense dataset comprises over a thousand tags. However, for our dataset, we retain only the top 15 most frequently selected tags as labels. The dataset does not provide demographic details of the users in the data. Furthermore, for the sake of simplicity, we only include users who have assigned a single tag to a film. We only retain users with a minimum of five tagged movies. Then, we partition the users into training, validation, and test sets. For each movie tagged by a user, we use the movie description as input, the movie’s tag as the output, and the remaining movies tagged by the same user and their tags as the user profile for that sample, following the template shown in Figure 3 in Appendix B. Finally, we randomly select 50% of the generated samples for each user in training, validation, and test sets, and add them to the samples of the corresponding set.
Personalized Product Rating. In this task, we create our dataset by leveraging the Amazon Reviews Dataset (Ni et al., 2019). We filtered out users (i.e., amazon customers who have written reviews) who have written less than 100 and the 1% users with the most reviews as outliers. Since the Amazon Reviews dataset is quite extensive, we randomly sampled a subset of users from the dataset, which we then split into training, validation, and testing sets. Note that the dataset does not provide demographic details of the users in the data.
7https://movielens.org/ 8https://www.themoviedb.org/?language=en-US
To construct the input-output pairs for our task, for each user, we randomly select one of their reviews as the input to the task and use their other reviews as their profile. Specifically, we use the profile to capture the author’s writing style, preferences, and tendencies. In this setup, the user’s score for the input review serves as the ground truth output for our task. To gain a better understanding of the input, output, and profile, refer to Figure 3 in Appendix B.

heading: Personalized News Headline Generation. To
text: construct our dataset for this task, we leverage the News Categorization dataset (Misra, 2022; Misra and Grover, 2021) from the HuffPost website. The dataset provides author information for each article and is used to group articles by their respective authors. The dataset does not provide demographic details of the users in the data. We filtered out the authors with less than four articles. In cases where an article has multiple authors, we assign it only to the first author.
We then randomly split the authors into training, validation, and test sets. For each author in each set, we create input-output pairs by selecting each article as the input, the headline of the article as the output, and the remaining articles written by the same author as their profile. This setup aims to capture the author’s writing style, preferences, and tendencies, which can be leveraged to generate headlines that align with their interests. An example of this setup is presented in Figure 3 in Appendix B. Finally, to ensure a diverse and representative dataset, we randomly select 50% of the created samples for each author and add them to the user’s corresponding set.
Personalized Scholarly Title Generation. Similar to Section LaMP-1, we leverage the Citation Network Dataset (V14) (Tang et al., 2008) that includes information about scientific papers, authors, and citations to construct our dataset. The dataset does not provide demographic details of the users in the data. We only kept the papers that meet the following criteria: 1) written in English, 2) have at least one reference and one author, and 3) have an abstract. Then, we group papers by their authors and only consider authors who have published at least 50 papers. For each author, we randomly choose one of their papers and use its abstract as input, its title as output, and the remaining papers as the author’s profile. Figure 3 in Appendix B illustrates the input format for this task. After cre-
ating the samples for all users, we divide users into the train, validation, and test sets for this task.
Personalized Email Subject Generation. In this study, we adopt the Avocado Research Email Collection (Oard, Douglas et al., 2015) as the primary dataset for our task. The dataset does not provide demographic details of the users in the data. To curate the dataset, we first perform a filtering step where we exclude emails with subject lengths of fewer than five words and content lengths of fewer than 30 words. Next, we group the emails based on their sender’s email address, retaining only those from users with email frequencies ranging between 10 to 200 emails. We further divide the users into distinct training, validation, and test sets to ensure that our model generalizes well to unseen data. To generate training examples for each user, we create input-output pairs by considering each email as the input and the corresponding email subject as the output. We supplement these pairs with other emails written by the same user as their profile, as shown in Figure 3 in Appendix B. We ensure that our dataset is diverse and representative by randomly selecting 50% of the curated samples for each user and adding them to their respective sets.
Personalized Tweet Paraphrasing. In this task, we utilize the Sentiment140 dataset (Go et al., 2009) as our tweet collection set. The dataset does not provide demographic details of the users in the data. To ensure that the collected tweets are of adequate length, we only retain tweets containing at least 10 words. We then group the tweets based on the user ID and filter out users with fewer than 10 tweets. Subsequently, we randomly select one tweet from each user profile and use it as input to ChatGPT (i.e., gpt3.5-turbo) to generate a paraphrased version. The generated paraphrase is then utilized as the input to our NLP task, with the original tweet serving as the corresponding output. The remaining tweets of the user constitute the user’s profile, excluding the one selected as input. Figure 3 in Appendix B provides an overview of the input-output-profile template for our proposed task. After creating the samples for all users, we divide users into the train, validation, and test sets for this task.

heading: A.2 Time-based Separation Setting
text: Personalized Citation Identification. To generate data samples, we leverage the Citation Network Dataset (V14) (Tang et al., 2008), which comprises
information on scientific papers, authors, and citations. The dataset does not provide demographic details of the users in the data. We select all papers from this dataset that meet the following criteria: 1) they are written in English, 2) they contain at least one reference and one author, and 3) they include an abstract. Subsequently, we group papers based on their authors and only consider authors who have written at least 50 papers. We divide each author’s papers based on the publication year into three groups chronologically: 1) profile papers, 2) train papers, 3) validation papers, and 4) test papers, where the order of groups shows the flow of time. Each train, validation, and test paper set in this task consists of only one paper. Then, for each paper in the train, validation, and test sets for the user, we select each paper and one of its cited references. For negative document selection, we randomly choose one of the first author’s coauthors and one of the papers they have cited in one of their papers, which has not been cited by the first author. If no such author exists, we randomly select an author and repeat this process. Finally, we construct the input, output, and profile of the generated samples for this task, employing the template depicted in Figure 3 in Appendix B. It should be noted that after creating all the samples in the train, validation, and test sets for all the users and aggregating them, we randomly select a subset of validation and test sets to create the final sets for the task.
Personalized Movie Tagging. To construct our dataset for this task, we leverage the MovieLense dataset (Harper and Konstan, 2015) obtained from the MovieLense website9. This dataset includes the tags that individual users have attributed to each film. The dataset does not provide demographic details of the users in the data. However, it does not contain the descriptions or summaries of the films. To acquire these, we retrieve the overviews of each film in this dataset from the Movie Database website10. The MovieLense dataset comprises over a thousand tags. However, for our dataset, we retain only the top 15 most frequently selected tags as labels. Furthermore, for the sake of simplicity, we only include users who have assigned a single tag to a film. We only retain users with a minimum of five tagged movies. We divide each user’s tagged movies based on the date they tagged into three
9https://movielens.org/ 10https://www.themoviedb.org/?language=en-US
groups, chronologically: 1) profile movies, 2) train movies, 3) validation movies, and 4) test movies, where the order of groups shows the flow of time. Each train, validation, and test movies set in this task consists of 20%, 10%, and 10% of movies, respectively. Then, for each movie in the train, validation, and test sets for the user, we use the movie’s description as the input, the movie’s tag as the output, and the profile movies and their tags as the profile for that sample, following the template shown in Figure 3 in Appendix B. It should be noted that after creating all the samples in the train, validation, and test sets for all the users and aggregating them, we randomly select a subset of validation and test sets to create the final sets for the task.
Personalized Product Rating. In this task, we create our dataset by leveraging the Amazon Reviews Dataset (Ni et al., 2019). The dataset does not provide demographic details of the users in the data. We filtered out users (i.e., amazon customers who have written reviews) who have written less than 100 and the 1% users with the most reviews as outliers. Since the Amazon Reviews dataset is quite extensive, we randomly sampled a subset of users from the dataset. We divide each user’s reviews based on the review date into three groups chronologically: 1) profile reviews, 2) train reviews, 3) validation reviews, and 4) test reviews, where the order of groups shows the flow of time. Each train, validation, and test reviews set in this task consists of only one review.
To construct the input-output pairs for our task, for each user, we select their reviews in each of train, validation, and test sets as the input to the task and use the profile reviews as their profile. Specifically, we use the profile to capture the author’s writing style, preferences, and tendencies. Additionally, the user’s score for the input review serves as the ground truth output for our task. To gain a better understanding of the input, output, and profile, refer to Figure 3 in Appendix B. It should be noted that after creating all the samples in the train, validation, and test sets for all the users and aggregating them, we randomly select a subset of validation and test sets to create the final sets for the task.

heading: Personalized News Headline Generation. To
text: construct our dataset for this task, we leverage the News Categorization dataset (Misra, 2022; Misra and Grover, 2021) from the HuffPost website. The
dataset provides author information for each article and is used to group articles by their respective authors. The dataset does not provide demographic details of the users in the data. We filtered out the authors with less than ten articles. In cases where an article has multiple authors, we assign it only to the first author. We divide each author’s articles based on the publishing date into three groups chronologically: 1) profile articles, 2) train articles, 3) validation articles, and 4) test articles, where the order of groups shows the flow of time. Each train, validation, and test articles set in this task consists of 20%, 10%, and 10% articles, respectively. Then, for each article in the train, validation, and test sets for the user, we create input-output pairs by selecting each article as the input, the headline of the article as the output, and the profile articles written by the same author as their profile. This setup aims to capture the author’s writing style, preferences, and tendencies, which can be leveraged to generate headlines that align with their interests. An example of this setup is presented in Figure 3 in Appendix B. It should be noted that after creating all the samples in the train, validation, and test sets for all the users and aggregating them, we randomly select a subset of validation and test sets to create the final sets for the task.
Personalized Scholarly Title Generation. We leverage the Citation Network Dataset (V14) (Tang et al., 2008) that includes information about scientific papers, authors, and citations to construct our dataset. The dataset does not provide demographic details of the users in the data. We only kept the papers that meet the following criteria: 1) written in English, 2) have at least one reference and one author, and 3) have an abstract. Then, we group papers by their authors and only consider authors who have published at least 50 papers. We divide each author’s papers based on the publication year into three groups chronologically: 1) profile papers, 2) train papers, 3) validation papers, and 4) test papers, where the order of groups shows the flow of time. Each train, validation, and test paper set in this task consists of only one paper. Then, for each paper in the train, validation, and test sets for the user, we use its abstract as input, its title as output, and the profile papers as the author’s profile. Figure 3 in Appendix B illustrates the input format for this task. After creating the samples for all users, we divide users into the train, validation, and test sets for this task. It should be noted that
after creating all the samples in the train, validation, and test sets for all the users and aggregating them, we randomly select a subset of validation and test sets to create the final sets for the task.
Personalized Email Subject Generation. In this study, we adopt the Avocado Research Email Collection (Oard, Douglas et al., 2015) as the primary dataset for our task. The dataset does not provide demographic details of the users in the data. To curate the dataset, we first perform a filtering step where we exclude emails with subject lengths of fewer than five words and content lengths of fewer than 30 words. Next, we group the emails based on their sender’s email address, retaining only those from users with email frequencies ranging between 10 to 200 emails. We divide each user’s emails based on the publishing date into three groups chronologically: 1) profile emails, 2) train emails, 3) validation emails, and 4) test emails, where the order of groups shows the flow of time. Each train, validation, and test emails set in this task consists of 20%, 10%, and 10% articles, respectively. Then, for each article in the train, validation, and test sets for the user, we create input-output pairs by considering each email as the input and the corresponding email subject as the output. We supplement these pairs with profile emails written by the same user as their profile, as shown in Figure 3 in Appendix B. It should be noted that after creating all the samples in the train, validation, and test sets for all the users and aggregating them, we randomly select a subset of validation and test sets to create the final sets for the task.
Personalized Tweet Paraphrasing. In this task, we utilize the Sentiment140 dataset (Go et al., 2009) as our tweet collection set. The dataset does not provide demographic details of the users in the data. To ensure that the collected tweets are of adequate length, we only retain tweets containing at least 10 words. We then group the tweets based on the user ID and filter out users with fewer than 10 tweets. We divide each user’s tweets based on the publication year into three groups chronologically: 1) profile tweets, 2) train tweets, 3) validation tweets, and 4) test tweets, where the order of groups shows the flow of time. Each train, validation, and test tweet set in this task consists of only one paper. Then, for each tweet in the train, validation, and test sets for the user, we use it as input to ChatGPT (i.e., gpt3.5-turbo) to generate a paraphrased version. The generated paraphrase
is then utilized as the input to our NLP task, with the original tweet serving as the corresponding output. The profile tweets of the user constitute the user’s profile. Figure 3 in Appendix B provides an overview of the input-output-profile template for our proposed task. It should be noted that after creating all the samples in the train, validation, and test sets for all the users and aggregating them, we randomly select a subset of validation and test sets to create the final sets for the task.

heading: B Samples of the Tasks Introduced in the LaMP Benchmark
text: As mentioned earlier, LaMP proposes seven tasks to evaluate language model personalization. In order to create the data points, we use just a carefully designed template for each task. Figure 3 depicts a sample and template for each task in LaMP. Generally, each sample in each task has an input and output accompanied by a profile consisting of several entries about the user, helping the model to produce personalized results for the user. While the profile entries in the same task have a similar structure, the structure varies between tasks. For example, Figure 3 shows that the profile for Personalized Product Rating comprised of documents with text and score sections, while the profile entries in Personalized Scholarly Title Generation have abstract and title attributes.

heading: C Prompts Used for Adding User Profile to the Language Model’s Input
text: In order to use multiple entries from the user profile to personalize the language model’s input, we construct task-specific prompts using the templates and instructions in Table 5.
The prompt creation consists of two stages: 1) Per Profile Entry Prompt (PPEP) creation and 2) Aggregated Input Prompt (AIP) creation. In the first stage, following the instructions in Table 5, we create a prompt for each profile entry. In the second stage, following the instructions in Table 5, we combine the PPEP prompts into a single prompt to be fed to the language model. It should be noted that due to the limited context size of language models, we need to trim the PPEP prompts. More accurately, considering k prompts need to be merged and that the maximum capacity for the task input is L̄ and the maximum context size of the language model is L, we let each PPEP occupy L−L̄k tokens in the language model’s input. For PPEPs
2
that are longer than the calculated number, we trim the non-template parts that have less importance in
the final performance of the model – the parts that do not provide category, score, or title. We select
L̄ = 256 in this paper.

heading: D Performance of the Models on the Validation Set
text: This section reports the results of experiments on the validation set. Table 6 reports the results of finetuning the language model on the user-based separation setting on the validation set. Table 8 shows the results of fine-tuning the language model on the time-based separation setting on the validation set. Table 7 shows the results of zero-shot evaluation of large language models on the user-based separation setting on the validation set. Table 9 depicts the results of zero-shot evaluation of large language models on the time-based separation setting on the validation set.

heading: E Performance of Some Other Non-Personalized Baselines on the LaMP Benchmark
text: To explore the performance of additional baselines, we present the performance of the Support Vector Machine (SVM) (Hearst et al., 1998) as a conventional classifier, BERT (Devlin et al., 2019) as a neural transformer-based encoder, and BART (Lewis et al., 2020) as a generative model. SVM and BERT are evaluated on classification tasks, and BART is evaluated on generation tasks within the LaMP benchmark. The results for the user-based and time-based separation configurations are documented in Table 10 and Table 11, respectively.

heading: F Dataset Licenses
text: This section specifies the licences and terms of use for each task, which is the same as the original dataset’s license:
1. Personalized Citation Identification: CC BYNC-SA 4.0 2. Personalized Movie Tagging: Educational or academic research, NON COMMERCIAL USE 3. Personalized Product Rating: CC BY-NC-SA 4.0 4. Personalized News Headline Generation: CC BY-NC-SA 4.0 5. Personalized Scholarly Title Generation: CC BY-NC-SA 4.0 6. Personalized Email Subject Generation: Avocado Collection End User Agreement LDC2015T03
7. Personalized Tweet Paraphrasing: CC BYNC-SA 4.0

heading: G AI Assistance Usage
text: In this paper, ChatGPT11 has been used as a writing assistant. In more detail, an initial paragraph is given to the ChatGPT and asked to paraphrase the given text. Further edits were also applied to the generated text and then used.
11https://chat.openai.com/

heading: Future Work
text: We firmly believe that the future of natural language processing systems lies in a user-centric approach, tailoring solutions to individual needs for optimal effectiveness