Aligning Large Language Models for Controllable Recommendations (2024)

Abstract: Inspired by the exceptional general intelligence of Large Language Models (LLMs), researchers have begun to explore their application in pioneering the next generation of recommender systems — systems that are conversational, explainable, and controllable. However, existing literature primarily concentrates on integrating domain-specific knowledge into LLMs to enhance accuracy using a fixed task template, often overlooking the diversity of recommendation tasks and the ability of LLMs to follow recommendation-specific instructions. To address this gap, we first introduce a collection of supervised learning tasks, augmented with labels derived from a conventional recommender model, aimed at explicitly improving LLMs’ proficiency in adhering to recommendation-specific instructions. Next, we propose a reinforcement learningbased alignment procedure to enhance LLMs’ generalization ability. Extensive experiments on two real-world datasets demonstrate that our approach significantly improves the capability of LLMs to respond to instructions within recommender systems, reducing formatting errors while maintaining a high level of accuracy.

heading: 1 Introduction
text: Recommender systems are designed to identify and suggest the most appropriate items to users from a vast array of candidates, based on the users’ profiles, past interactions, and present intentions. Witnessing the impressive capabilities of Large Language Models (LLMs), such as knowledge retention, reasoning, and problem-solving, researchers are now exploring the integration of LLMs into the next wave of intelligent recommender systems, which aim to be conversational, explainable, and controllable. Bridging the gap between the generalpurpose LLMs and the specific requirements of recommendation tasks poses a challenge. In this context, fine-tuning LLMs with domain-specific knowl-
edge and recommendation-focused tasks emerges as a promising strategy to harness their potential for advanced recommendation purposes (Bao et al., 2023; Zhang et al., 2023; Chen, 2023).
Typical approaches in the literature involve reformatting recommendation tasks — such as item reranking or click-through rate (CTR) prediction — into natural language constructs to facilitate the fine-tuning of LLMs. However, we have observed that LLMs fine-tuned through this straightforward method, albeit enhancing accuracy in offline evaluations, frequently generate outputs with domainspecific formatting errors. These errors may manifest as repeated items in the top-k recommendations or the inclusion of items previously interacted with by the user. Additionally, these LLMs exhibit a limited ability to adhere to diverse recommendationspecific instructions. This compromises their effectiveness as interactive agents in real-world recommender systems. A vivid example can be found in Section 3.4.
In this paper, we investigate the alignment of an LLM for recommender systems. Our objective extends beyond merely improving the recommendation accuracy of an original LLM; we aim to significantly enhance controllability and reduce formatting errors. Drawing inspiration from the Reinforcement Learning from Human Feedback (RLHF) framework (Ouyang et al., 2022), our methodology is structured into two phases: the supervised learning (SL) stage and the reinforcement learning (RL) stage. To inject domain-specific knowledge and foster recommendation-relevant control capabilities within the LLM, we devise a series of fine-tuning tasks, including item recommendation, item search, category control, and category proportion control. These tasks often necessitate generating a list of items that not only meet users’ instructions but also maintain high quality, despite the typically sparse ground-truth signals found in user behavior history. To tackle this issue, we pro-
8159
pose augmenting supervised labels with predictions from a traditional recommender model, such as SASRec (Kang and McAuley, 2018). These augmented labels can help distill knowledge from the traditionally trained recommender model and meet the dynamic requirements of recommendation instructions.
After the SL stage, the LLM exhibits a significantly enhanced ability to follow recommendationrelated instructions, surpassing existing approaches that solely fine-tune the LLM on item recommendation and search tasks. Nevertheless, the SL stage’s data generation process inherently provides only positive examples for each instruction. To address scenarios where the LLM generates suboptimal responses, we introduce an RL stage with carefully crafted reward signals to further refine the LLM’s capacity to follow instructions. To the best of our knowledge, this is the first study that employs both SL and RL stages to align LLMs for controllable recommendation purposes. We conduct comprehensive experiments on two real-world datasets, Steam and Amazon Movie, with results demonstrating that our method markedly improves the LLM’s ability to follow instructions while simultaneously reducing formatting errors. Our major contributions are summarized as follows:
• We introduce a novel supervised learning stage, which encompasses a suite of tasks designed for enhancing controllability and label-augmentation by a teacher recommender model, to align an LLM into an interactive recommender agent.
• To mitigate formatting errors and improve the instruction-following generalization, We further design an alignment stage based on reinforcement learning with a variety of rewards that are tailored for the nuances of the controllable recommendation task.
• Experiment results validate that our model markedly surpasses existing LLM-based recommendation models, and exhibits a robust capacity to follow users’ instructions while maintaining a high level of recommendation precision. Source code is available at https://github.com/microsoft/ RecAI/tree/main/RecLM-gen.

heading: 2 Methodology

heading: 2.1 Intention Categories
text: This paper aims to enhance the instructionfollowing capabilities of LLMs for recommenda-
tion tasks. We categorize recommendation instructions into three distinct types: Implicit intention. This is the assumed default where the prompt describes the user’s profile (such as attributes and past favored items). The LLM is tasked with recommending items that align with the user’s preferences. Item-wise intention. In addition to the profile, users may express specific desires, such that the recommended items should either exhibit particular characteristics (“I wish to watch an action movie”) or exclude them (“Please avoid suggesting any action movies”). List-wise intention. Users may have requirements for the entire list of recommended items; hence, evaluating individual items’ attributes is insufficient. For example, if all items in a recommendation list belong to the same category A, the user may be disappointed by the lack of diversity. Consequently, the user might request the recommender system to ensure that the proportion of category A is below a certain threshold.
To effectively train LLMs as recommender agents capable of adhering to these three types of instructions, we introduce a novel two-stage finetuning approach: a supervised learning (SL) stage (Section 2.2) followed by a reinforcement learning (RL) stage (Section 2.3). The overall framework is illustrated in Figure 1.

heading: 2.2 The SL Stage
text: We represent each data sample in the recommendation task with natural language text, adopting the format “Instruction: [Prompt Content]. Output: [Response Content]”, where [Prompt Content] includes detailed instructions such as the user’s profile and intention, and [Response Content] contains the expected item recommendations that fulfill the instructions. Different from traditional recommender systems that utilize item IDs, we employ only item titles to represent items in both [Prompt Content] and [Response Content] to fully leverage LLMs’ general abilities and ensure a smooth interaction between users and our LLM-based recommender. Data samples are generated according to the following tasks:

heading: 2.2.1 Data Generation Tasks
text: Sequential Recommendation Instructions (I0) This task represents the traditional sequential recommendations: given a user’s previously interacted items, the goal is to predict future interactions.
Specifically, we use the first n−1 items to construct the user’s behavioral profile, while the nth item serves as the ground-truth label. The LLM is instructed to recommend k items; if the ground-truth item is among these k suggestions, the recommendation is considered a successful hit. The value of k for each data sample is randomly selected from 1 to 10.
Category Control Instructions (I1) This task corresponds to instructions with item-wise intention, which we implement into two distinct types: 1) Positive control (I+C1 ), where the user hopes to receive more items in the recommendation list that match the specific category Ctarget. 2) Negative control (I−C1 ), where users indicate a weariness towards a certain category Ctarget and wish to reduce the inclusion of such items in their received recommendations as much as possible.
Category Proportion Control Instructions (I2) We implement the list-wise intention into three distinct types: 1) ICP≤x2 , in this case, the user hopes to have the proportion of the item of Ctarget less than a certain percentage x. 2) ICP≈x2 , in this case, the user hopes that the proportion of items in Ctarget will be approximately a certain percentage. 3) ICP≥x2 , in this case, the user hopes that the proportion of items in Ctarget is greater than a certain percentage.
Item Search Instructions (I3) To aid the LLM in memorizing in-domain item attributes (in this paper, we use item category as the key attribute for illustration), we introduce an item search task: the objective is to retrieve k items belonging to a target category Ctarget. For this purpose, we randomly select k items from Ctarget to serve as the ground truth for the response.
ShareGPT (I4) To avoid catastrophic forgetting and preserve the general intelligence capabilities of the LLM, we follow Zeng et al. (2023) and integrate a certain proportion of ShareGPT1 training data into the SL stage. The ShareGPT data includes a diversity of real-world tasks in the user-assistant conversation format, which helps LLM revisit past knowledge during the in-domain fine-tuning stage.

heading: 2.2.2 Label Augmentation
text: Instructions I0, I1, and I2 require the inclusion of k items within the response text. However, due to the typically sparse nature of user historical behavior, it is often impractical to construct a groundtruth response based solely on this information. To overcome this limitation, we employ the sequential recommender model SASRec (Kang and McAuley, 2018) as a teacher model to generate a set of top recommendations, PSASRec, for each data sample. We then curate the top-k list by selecting items from PSASRec that align with the given instructions. The top-k list is assembled as follows: the first item is the ground-truth item (i.e., the nth item in the user’s history). The method for compiling the remaining k−1 items varies slightly: for I0, we fill the list with SASRec’s top predictions; for I1 and I2, we filter SASRec’s predictions to ensure the final recommendation list adhere to the specified instruction.

heading: 2.3 The RL Stage
text: Following previous works (Touvron et al., 2023; Ouyang et al., 2022), we employ the Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017) to further fine-tune the model after the SL stage. Different from (Ouyang et al., 2022), scores
1https://huggingface.co/datasets/anon8231489123/ ShareGPT Vicuna unfiltered
are not produced by a reward model; instead, they are derived from reward rules that are specifically tailored for I0, I1, and I2. Fundamentally, rewards consist of two components: item-level rewards and list-level rewards.

heading: 2.3.1 Item-level Reward
text: The item-level reward assigns a score to each item generated by LLM, serving as an immediate reward of reinforcement.
For each itemi in LLM’s recommendation list, let Ranki refer to the rank of itemi in the teacher model (SASRec)’s prediction list. We can then calculate the preference scores for itemi by:
Scoresi =    −1, if itemi is illegal +1, if itemi is itemtarget
1 log2(Ranki+3)
, else (1)
itemi is considered illegal if it meets any of the following conditions: itemi does not exist, it is a duplicate of any item within the preceding set item[1,...,i−1], it is identical to an item in the user’s history, or its index i exceeds k.
Beyond the preference score, Scores, a control effect score, Scoresctl, is required to gauge the extent to which an item corresponds with user intentions. Essentially, a generated item that adheres to the given instruction is awarded a high score, signifying positive reinforcement, while a non-conforming item incurs a low score, serving as negative feedback. The complete calculation is described in algorithm 1 in the Appendix.
Finally, we get item-level reward Ritem, which measures the overall merits of each item:
Ritem = 0.5 ∗ Scores+ 0.5 ∗ Scoresctl (2)

heading: 2.3.2 List-level Reward
text: The list-wise reward, which measures how well the entire list of recommendations matches the user’s preferences and control intentions, is added to the last token of output, serving as a termination reward. To encourage the LLM to rank ground-truth items in top positions, we adjust the Scores as per Equation 3 to derive Score∗:
Scores∗i = { −1, if itemi is illegal Scoresi log2(i+2) , else (3)
In addition to this, we need to measure how well the output matches the control intention. Let Countin, Countout denote the number of items
that belong/do not belong to the target category. For different control intentions, we use different calculation methods to get Scorectllist:
Scorectllist =    sum(Scores∗), if I0 1 log2((k−Countin)+2) , if I +C 1 1 log2((k−Countout)+2) , if I −C 1 1 log2(max(Countin−k∗m,0)+2) , if I CP≤m 2 1 log2(max(k∗m−Countin,0)+2) , if I CP≥m 2
1 log2(abs(Countin−k∗m)+2) , if I CP≈m 2
(4)
Finally, we get Rlist, which measures the overall merits of the recommendation list:
Rlist = 0.5 ∗ sum(Scores∗) + 0.5 ∗ Scorectllist (5)

heading: 2.3.3 RL Implementation Notes
text: We follow the Transformer Reinforcement Learning2 package to implement the reinforcement learning stage. We set up LoRA layers (Hu et al., 2022) for both the policy network and the critic network, with a LoRA rank of 4 and a LoRA alpha of 2. During the RL sampling phase, we sample 2 responses with a temperature of 0.7 for each instruction. The final reward is calculated by combining item-level reward, list-level reward, and a KL penalty:
Rfinal[y] = Ritem[y]+Rlist[y]−ηKL(πRLθ , πSFT)[y] (6) where η is set to 0.3. y represents a generated response (which is a token sequence) for an instruction sample. KL[y] represents there is a KL penalty on each token in y. Ritem[y] means there is an item-level reward at the last token of each item title. Rlist[y] represents that there is a list-level reward on the ending token of y. To ensure that the information in Rlist is not overshadowed by Ritem, we amplify Rlist by a factor of 10. Additionally, we employ reward whitening to enhance the stability of training. We use Generalized Advantage Estimation (GAE) to estimate the advantage values, with hyperparameters γ = 0.99, λ = 0.95. During the loss calculation phase, we set the clipping range for the probability ratio to [1− ϵ, 1+ ϵ], where ϵ = 0.2. The loss weight for the critic network is 0.5.

heading: 3 Experiments

heading: 3.1 Experiment Setting

heading: 3.1.1 Dataset
text: We experiment with two popular datasets in the recommender system domain: the Amazon Movies
2https://github.com/huggingface/trl
& TVs 3 dataset (Movie for short) and the Steam dataset (Kang and McAuley, 2018). Both datasets include item category information, which aids in constructing the user’s control intentions. Basic statistics of datasets are summarized in Table 1. We employ the leave-one-out approach (Kang and McAuley, 2018) to split the dataset. Therefore, the size of test set is consistent with the number of users in each dataset. To accelerate the validation process, we only include four types of instructions in the valid set: I0, I+C1 , I −C 1 , and I CP≈50% 2 , and each type of instruction has 320 instances. Upon observing multiple metrics on the validation set, we find that SL typically converged around 30 epochs. Full details of the training set on each instruction task can be found at Table 8 in the Appendix.

heading: 3.1.2 Implementation Details
text: We choose Llama-2-7b-chat as the foundational model for our research (Touvron et al., 2023). We set the model’s maximum sequence length to 1024 tokens. User behavior sequences are truncated to incorporate no more than 10 items, and excessively lengthy item titles are condensed to a maximum of 64 tokens. Furthermore, to accommodate the complete recommendation list within the output, we reserve a larger token count, setting the maximum output length of our model to 512 tokens. Instructions and responses are formatted using the official prompt template in Touvron et al. (2023).
We use LoRA (Hu et al., 2022) to fine-tune all linear layers in Llama2-7b-chat, with trainable parameters accounting for about 0.6% of the total parameters in SL and 0.3% in RL. The optimizer is Adam. In the SL stage, the learning rate is set to 0.001, the LoRA dimension is set to 16, the LoRA alpha to 8, and the batch size is 64. The ShareGPT data accounted for 50% of the total training data. In the RL stage, the learning rate is 5× 10−6. We set separate LoRA layers for actor and critic networks, with LoRA dimension at 4 and LoRA alpha at 2. To encourage model exploration, we set the temperature to 0.7 and the weight of entropy loss to 0.01. We release the source code at: https://github. com/microsoft/RecAI/tree/main/RecLM-gen.
3https://cseweb.ucsd.edu/∼jmcauley/datasets/amazon v2/

heading: 3.1.3 Metrics
text: For all models, we use Top-k hit ratio (HR@K) and Top-k NDCG (NDCG@K) to evaluate the accuracy of recommendations. Meanwhile, we use some additional indicators to evaluate the model’s ability to follow user instructions: 1). For category control, we use the Top-k target category proportion (TCP@K) metric to evaluate the proportion of target categories in the recommendation list. The calculation of TCP@K is shown in Equation 7. 2). For category proportion control, we use the target category proportion accuracy (CPA), defined in Equation 8, to measure whether the responding item distribution complies with the instruction. Note that k indicates the number of recommended items and m indicates the proportion requirement in instructions.
TCP@K = 1
K
K∑
i=1
1(itemi ∈ Ctarget) (7)
CPA =    1(Countin ≤ k ∗m), if ICP≤m2 1(Countin ≥ k ∗m), if ICP≥m2 1(abs(Countin − k ∗m) ≤ 1), if ICP≈m2 (8)

heading: 3.1.4 Baselines
text: We divide the compared methods into 3 classes: general LLM, fine-tuned LLM, and our method and its variants. For sequential recommendation, we additionally compared with SASRec (Kang and McAuley, 2018) that we used as the teacher model.
General LLM: We use a closed-source LLM GPT-3.5-turbo-0613 4 (GPT-3.5 for short) and an open-source LLM Llama2-7b-chat (Touvron et al., 2023). Due to the high cost, all GPT-3.5 results are obtained from the first 1,000 samples of the test set.
Fine-tuned LLM: InstructRec (Zhang et al., 2023) uses 39 types of recommendation-related instructions constructed from user history and review data. It uses Beam search to obtain the top-k recommendation list during the test phase. Its backbone is 3B Flan-T5-XL. PALR (Chen, 2023) also uses Llama2-7b-chat as the backbone. We fine-tune both models using our datasets, employing data sample generation techniques as described in their respective papers. For instance, a key distinction between PALR and our supervised instruction tuning task, I0, is that PALR does not utilize a teacher
4https://platform.openai.com/docs/models/gpt-3-5-turbo
recommender model for label augmentation; instead, it relies solely on user behavior history to derive label responses.
Our method and variants: Oursv1 is a model that only uses sequential recommendation instructions (I0) in SL. Oursv2 is a model uses all instructions (I{0,1,2,3,4}) in SL. Oursv3 is a model obtained by further fine-tuning Oursv2 with RL, but without the item-level reward. Oursfull is the complete version of our method. We compare different variants of our method as an ablation study to verify the effect of different components.

heading: 3.2 Overall Performance

heading: 3.2.1 Sequential Recommendation
text: We begin by assessing model performance under standard sequential recommendation scenarios, where users have no explicit intentions. It is important to clarify that our objective in this paper is to improve the ability of LLMs to follow recommendation-related instructions, not to outperform the teacher model, SASRec. Achieving recommendation accuracy on par with the teacher model is considered satisfactory for our purposes. Table 2 presents the comprehensive results. Initially, all fine-tuned models surpass the performance of general LLMs such as GPT-3.5 and Llama2-7b, affirming the necessity of finetuning for domain-specific tasks. Additionally, our approach significantly outperforms the finetuned benchmarks (InstructRec and PALR), validating the efficacy of our Supervised Learning (SL) stage. Lastly, our completed model, Oursfull, achieves results comparable to the teacher model, SASRec, reinforcing the premise that an LLM must first grasp user preferences before its instructionfollowing capabilities can be further evaluated.

heading: 3.2.2 Category Control
text: To evaluate the Category Control Instructions (I1), we use the following two settings:
Positive (I+C1 ): We utilize the target item’s category as the control signal Ctarget and incorporate a positive descriptor of Ctarget into the input instruction to represent the user’s explicit intent to receive more items within that category.
Negative (I−C1 ): We analyze the output statistics of top-k recommendations by SASRec to identify the category with the largest proportion — excluding the target item’s category — as Ctarget for simulated control. A negative descriptor of Ctarget
is then embedded in the instruction to reflect the user’s intent to minimize items from this category.
Table 3 presents the results, using the TCP@10 metric to assess conformity to control signals, while HR@10 and NDCG@10 indicate recommendation accuracy when explicit intentions are additionally included in the prompt. Values in parentheses show the outcomes of each corresponding model without control signals. Two key observations emerge: Firstly, differentiating supervised learning tasks by intention is essential. Evidence for this includes InstructRec’s improved TCP performance in the I1+C task, which aligns with its training, versus no effect in the I1−C task not covered by its training. Similarly, Oursv1, trained solely on recommendation tasks, underperforms in TCP compared to Oursv2. Secondly, Oursv2, Oursv3, and Oursfull show notable TCP enhancements, with Oursfull leading. Additionally, HR and NDCG metrics also see a significant lift compared to Table 2, confirming our method’s effectiveness in adhering to category control instructions.

heading: 3.2.3 Category Proportion Control
text: To evaluate the Category Control Instructions (I1), we use the following three settings:
I CP≤20% 2 : In this case, we select the Ctarget of I−C1 in the same way to simulate control. The purpose is to control the proportion of Ctarget to be no greater than 20%. ICP≈30%2 : In this case, we select the category of the target item as the Ctarget of control simulating. The purpose is to control the proportion of items in Ctarget to be approximately 30%.
I CP≥30% 2 : We select the category of the target item as the Ctarget. The purpose is to control the proportion of items in Ctarget to be no less than 30%.
Results are reported in Table 4. In this case, the primary metric is CPA. Overall, Oursfull achieves the best performance, outperforming all other baselines and its own variants.

heading: 3.3 Formatting and General Evaluation
text: Finally, we assess the formatting ability and overall linguistic capabilities of LLMs in generating structured recommendations. We measure formatting ability for the recommendation domain in four dimensions: 1) CorrectCount: The accuracy of the recommended item count against the given k in the instruction. 2) RepeatItem: The frequency of repeated items in the recommendation list. 3) NonExist: The occurrence of non-existent (hallucinated) items in the list. 4) InHistory: The rate of recommended items already present in the user’s history. For a more challenging test, LLMs are tasked with recommending k items where k is a random number between 11 and 15, despite training only on ranges from 1 to 10. To evaluate the general language ability of LLMs, we utilize two standard tasks, MMLU with five examples (5- shot) and GSM8K with eight examples (8-shot), leveraging the unified framework provided by https: //github.com/EleutherAI/lm-evaluation-harness.
Table 5 presents the comprehensive outcomes. Notably, nearly all models, with the exception of PALR on the Movie dataset, excel in the Correct-
Count metric. For other formatting metrics such as RepeatItem, NonExist, and InHistory, a progressive enhancement is evident from Oursv1 through Oursv3, culminating in Oursfull. In Table 5, the primary focus is on Formatting Quality, while Precision metrics serve as supplementary references. The Generalization metrics indicate the extent of catastrophic forgetting. When compared to the finetuned baseline PALR, Oursfull shows the least performance deterioration from its underlying Llama27b architecture. Owing to the 512-token input limitation of InstructRec, its performance could not be assessed on the Generalization tasks.

heading: 3.4 Case Study
text: For clarity, we present two illustrative cases in Table 6. The first is about the ICP≈50%2 instruction on the Movie dataset. A user requests that roughly 50% of the top-5 recommendations feature “Art House & International, By Original Language, Chinese” characteristics. Our method successfully generates a list where 3 out of 5 items possess these attributes, while three baseline methods fail to meet this criterion. Llama-2-7b simply repeats items mentioned in the user history.
In the second case, concerning the I−C1 instruction, the user specifies to exclude “Shooter” games from recommendations. Our method effectively adheres to this restriction by omitting any items from the blacklisted category and successfully includes the ground-truth item in the recommendation list. Conversely, Llama-2-7b incorrectly suggests “Far Cry 5”, a Shooter game, while InstructRec not only
makes aching historical mistakes but also recommends a lot of games from the shooter genre.

heading: 3.5 Extend to combinatorial Controls
text: Considering LLMs’ strong generalization capabilities, we further examine the performance of LLMs on complex, combinatorial instructions not encountered during training, as shown in Table 7. Here, TC1¬2P@10(%) means the percentage of items belong to C1 but does not belong to C2. The results confirm our expectations that while our methods surpass baseline models, they achieve lower compliance with user instructions, as measured by TCP and CPA, compared to simpler single-control instructions. This motivates us to include some complex instructions in the alignment process in future work.

heading: 4 Related Work
text: In recent years, the remarkable natural language processing capabilities of LLMs have inspired researchers to leverage them for recommendation tasks (Fan et al., 2023). Early implementations utilized language models as feature extractors, creating knowledge-aware recommendation embeddings exemplified by models such as U-BERT (Qiu et al., 2021) and UserBERT (Wu et al., 2022). With the advent of generative models like GPT, the focus has shifted towards generative recommendation models that frame recommendations as natural language generation tasks (Wu et al., 2023).
Initially, adaptation of LLMs to recommendation scenarios relied heavily on techniques like prompt engineering (Gao et al., 2023; Sun et al., 2023) and contextual learning (Dai et al., 2023; Liu et al., 2023). However, these LLMs often underperform compared to traditional recommendation models trained on task-specific data, prompting the neces-
Instruction ICP≈50%2 : You are an expert recommender engine. You need to generate a recommendation list simultaneously considering user‘s preference and user’s intention. Here are the user’s historical interactions: ’The Blind Swordsman: Zatoichi Takeshi Kitano; Tadanobu Asano japan’, ’Buffy the Vampire Slayer - The Complete Seventh Season’, ’Number Ones’, ’24: Season 3’, ’Madonna - The Immaculate Collection’, ’Spider-Man 2’, ’Heavy Metal 2000 VHS’, ’MI-5, Vol. 1’, ’MI-5, Vol. 2’, ’Beyond Re Animator VHS’, and user’s intention: In the recommendation list, the proportion of ’Art House & International, By Original Language, Chinese’ items should be approximately 50%. You need to generate a recommendation list with 5 different items. GPT− 3.5 1. Oldboy 2. Chungking Express 3. Battle Royale 4. In the Mood for Love 5. Memories of Murder Llama2− 7b 1. The Blind Swordsman: Zatoichi 2. Buffy the Vampire Slayer 3. Number Ones 4. 24: Season 3 5. Madonna - The Immaculate Collection InstructRec 1. The Magnificent Seven VHS 2. Shanghai Noon 3. X-Men: First Class 4. X-Men VHS 5. Akira VHS
Oursfull 1. Five Deadly Venoms VHS 2. Ip Man 2 3. The Village: Vista Series 4. Dawn of the Dead 5. Fong Sai Yuk VHS Instruction I−C1 : You are an expert recommender engine. You need to generate a recommendation list simultaneously considering user‘s preference and user’s intention. Here are the user’s historical interactions: ’Yet Another Zombie Defense HD’, ’Battlefield™ V’, ’Tom Clancy’s Ghost Recon® Wildlands’, ’Agents of Mayhem’, ’Beholder’, ’HELLDIVERS™ Dive Harder Edition’, ’SUPERHOT’, ’Watch Dogs® 2’, ’Deus Ex: Mankind Divided’, ’POSTAL 2’, and user’s intention: Don’t recommend me any ’Shooter’ products. You need to generate a recommendation list with 5 different items. GPT− 3.5 1. Stardew Valley 2. Hollow Knight 3. Cities: Skylines 4. Divinity: Original Sin 2 5. The Witness Llama2− 7b 1. Assassin’s Creed Odyssey 2. The Witcher 3: Wild Hunt 3. Horizon Zero Dawn 4. Far Cry 5 5. Life is Strange 2
InstructRec 1. POSTAL 2 2. POSTAL Redux 3. Half-Life 2 4. Resident Evil 0 5. POSTAL Oursfull 1. The Witcher® 3: Wild Hunt 2. Human: Fall Flat 3. The Forest 4. Outlast 5. Middle-earth™: Shadow of War™
Table 6: Two real examples for case study. We highlight some texts for Repeat with history item ,
In target category item and target item.
sity to fine-tune LLMs for better alignment with recommendation tasks. P5 (Geng et al., 2022) introduced a unified framework that integrates 5 recommendation tasks through fine-tuning on the FLANT5 model (Raffel et al., 2020). Subsequently, InstructRec (Zhang et al., 2023) tailored FLAN-T5 for various downstream recommendation tasks using instruction tuning. TALLRec (Bao et al., 2023) fine-tunes LLaMA for recommendations with very few training samples, but it focuses on the binary classification task. PALR (Chen, 2023) employs two types of instructions for instruction tuning to facilitate list-level recommendation generation.
Despite these advancements, current studies have not fully explored the potential of LLMs to enhance the interactivity of recommender systems. We aim to harness the instruction-following
prowess of LLMs for recommendation tasks through fine-tuning, to create a conversational, controllable, and interactive recommender agent.

heading: 5 Conclusion
text: In conclusion, our paper presents a new approach to tailor LLMs for interactive recommender systems, combining a supervised learning phase with innovative control tasks and a reinforcement learning stage with specialized reward signals. Our method successfully meets the detailed demands of recommendation contexts, enhancing LLMs’ performance. Experimental results show our approach exceeds current LLM-based systems in precision, controllability, and presentation, offering a significant step towards refined and reliable recommendation services.

heading: 6 Limitation
text: This paper’s primary constraints are as follows: (1) The emphasis is placed on improving the LLM’s ability to follow recommendation-related instructions. This focus may inadvertently compromise the LLM’s broader intellectual capabilities, as revealed in Table 5. How to further reduce catastrophic forgetting remains a big challenge. (2) In real-world scenarios, users often have diverse control intentions, including intricate blends of various instructions (as discussed in Section 3.5) and new instructions beyond category control. As foundational research, this paper addresses only the most critical elements, specifically category control and formatting control. More diverse and complicated instructions are yet to be explored.

heading: 7 Acknowledgments
text: This work is supported by the National Natural Science Foundation of China (Grant Nos. 62276171 and 62072311), Guangdong Basic and Applied Basic Research Foundation (Grant Nos. 2024A1515011938, 2020B1515120028), Guangdong Provincial Key Laboratory of Popular High Performance Computers (Grant No. 2017B030314073), Guangdong Province Engineering Center of China-made High Performance Data Computing System, Shenzhen Fundamental Research-General Project (Grant Nos. 20220811155803001, 20210324094402008). Hao Liao is the corresponding author.

heading: A Appendix
text: A.1 Dataset detail We present detailed information about the instruction set we construct in Table 8, including the quantity of each type of instruction.
Instruct type Movie Steam
I0 13218 12658 (I+C1 , I −C 1 ) (6609, 6609) (6329, 6329) (ICP≤∗2 , I CP≥∗ 2 , I CP≈∗ 2 ) (4406, 4406, 4406) (4219, 4219, 4220) I3 18744 8572 I4 29199 23273
ALL 87597 69819
A.2 Computational Cost Finetuning LLMs for recommendations is a little bit more expensive than training traditional recom-
mender models like SASRec, but the resource and cost are not a big barrier even for small organizations or researchers. Here we take the Movie dataset for illustration. The dataset statistics can be found in Table 1 and Table 7.
For the SL stage, we use 4 A100 GPUs (40GB GPU memory) for training. The batch size on each GPU is 1 and the gradient accumulation step of 16. The maximum sequence length is set to 1024. Under this setting, each epoch costs about 70 minutes. Usually, models can fully converge in 30 epochs. So, the total training time for SFT is about 35 hours.
For the RL stage, we use 2 A100 GPUs (40GB GPU memory) for training. The batch size on each GPU is 1 and the gradient accumulation step of 2. For each batch, instruction sampling time + 2 candidate response generation time + model training time counts about 40 seconds, and the maximum training steps are set to 3k. Thus, the total RL training time is 3000 ∗ 40/60/60 = 33 hours.
Once the training process is finished, we use vllm5 for inference and serving. On a single A100 GPU (40GB memory), when responding to the top10 recommendation request, our implementation can process 20 requests per second (on average each request involves newly generated 112 tokens).
A.3 Prompts In this section, we show the prompt for the instruction used in the experiment. The prompt of I0, I1, I2, I3 is illustrated in Listing 1 to Listing 4 respectively. The prompt of positive and negative category control intentions is illustrated in Listing 5 to Listing 6 respectively
5https://github.com/vllm-project/vllm
Listing 1: Prompts of I0 Instruction: You are an expert recommender engine. You need to generate a
recommendation list considering user’s preference from historical interactions. The historical interactions are provided as follows: {history}. You need to generate a recommendation list with {item_count} different items.
Output: {item_list}
Instruction: You are an expert recommender engine. You need to select a recommendation list considering user’s preference from historical interactions. The historical interactions are provided as follows: {history}. The candidate items are: {candidate_titles}. You need to select a recommendation list with {item_count} different items from candidate items. Output: {item_list}
Listing 2: Prompts of I1 Instruction: You are an expert recommender engine. You need to generate a
recommendation list simultaneously considering user’s preference inferred from historical interactions and user’s intention. If user’s preference conflicts with his intention, you should comply with his intention. Here are user’s historical interactions: {history}, and user’s intention: {synthetic_intention}. You need to generate a recommendation list with {item_count} different items.
Output: {item_list}
Instruction: You are an expert recommender engine. You need to select a recommendation list from candidate items simultaneously considering user’s preference inferred from historical interactions and user’s intention. If user’s preference conflicts with his intention, you should comply with his intention. Here are user’s historical interactions: {history}, and user’s intention: {synthetic_intention}. The candidate items are: {candidate_titles}. You need to select a recommendation list with {item_count} different items from candidate items. Output: {item_list}
Listing 3: Prompts of I2 Instruction: You are an expert recommender engine. You need to generate a
recommendation list simultaneously considering user’s preference inferred from historical interactions and user’s intention. Here are user’s historical interactions: {history}, and user’s intention: In the recommendation list, the proportion of ’{target_category}’ items should be less than or equal to {category_proportion}. You need to generate a recommendation list with {item_count} different items.
Output: {item_list}
Instruction: You are an expert recommender engine. You need to generate a recommendation list simultaneously considering user’s preference inferred from historical interactions and user’s intention. Here are user’s historical interactions: {history}, and user’s intention: In the recommendation list, the proportion of ’{target_category}’ items should be more than or equal to {category_proportion}. You need to generate a recommendation list with {item_count} different items.
Instruction: You are an expert recommender engine. You need to generate a recommendation list simultaneously considering user’s preference inferred from historical interactions and user’s intention. Here are user’s historical interactions: {history}, and user’s intention: In the recommendation list, the proportion of ’{target_category}’ items should be approximately {category_proportion}. You need to generate a recommendation list with {item_count} different items. Output: {item_list}
Listing 4: Prompts of I3 You are an expert recommender engine. You need to generate a recommendation list
complying user’s intention. Here is user’s intention: {synthetic_intention}. Please generate a recommendation list with {item_count} different items.
Output: {item_list}
You are an expert recommender engine. You need to select a recommendation list complying user’s intention from candidate items. Here is user’s intention: {synthetic_intention}. The candidate items are: {candidate_titles}. Please select a recommendation list with {item_count} different items from candidate items. Output: {item_list}
Listing 5: Prompts of positive intention in I1 and I3 I like ’{target_category}’ products Please recommend some ’{target_category}’ items I’m interested in ’{target_category}’ I would like to buy some ’{target_category}’ products I would like to browse some ’{target_category}’ products I prefer in ’{target_category}’ item
Listing 6: Prompts of negative intention in I1 and I3 I don’t like ’{target_category}’ products Please exclude any ’{target_category}’ item I’m not interested in ’{target_category}’ Don’t recommend me any ’{target_category}’ products I don’t want to browse any ’{target_category}’ product I hate ’{target_category}’ items

heading: Future Work
text: This motivates us to include some complex instructions in the alignment process in future work