Factual and Tailored Recommendation Endorsements using Language Models and Reinforcement Learning
Authors: Jihwan Jeong
Yinlam Chow
Guy Tennenholtz
ChihWei Hsu
Mohammad Ghavamzadeh
Craig Boutilier
Venue: COLM (2024)
Keywords: Large language model, reinforcement learning, conversational recommender systems, recommender systems
TL;DR: Develop a conversational recommender system that uses natural language and reinforcement learning to make precise, appealing, and personalized item recommendations based on user preferences, showing promising results on major datasets.

Abstract: Recommender systems (RSs) play a central role in matching candidate items to users based on their preferences. While traditional RSs rely on user feed-back signals, conversational RSs interact with users in natural language. In this work, we develop P4LM, an _aPpealing, Precise, Preference-comprehensive and Prioritized_ language model which endorses recommended items by emphasizing specific item characteristics and their coverage to a user’s preferences. P4LM uses an _embedding_ representation of a user’s preferences to generate responses that are appealing, factually-grounded and tailored to the user’s preferences. P4LM employs a joint reward function to measure precision, appeal, preference coverage and prioritization of preferences, which are used as AI-based feedback in a reinforcement learning-based language model framework. On the MovieLens 25M and Amazon Product Review datasets, P4LM delivers more appealing and tailored endorsements to users, as determined by auto-critic and rater evaluations.

Published as a conference paper at COLM 2024
Factual and Tailored Recommendation Endorsements using Language Models and Reinforcement Learning
Jihwan Jeong' University of Toronto
Yinlam Chow Google Research
Guy Tennenholtz Google Research
Chih-wei Hsu Google Research
Aza Tulepbergenov Google Research
Mohammad Ghavamzadeh x Amazon
Craig Boutilier Google Research
Abstract
Recommender systems (RSs) play central role in matching candidate items to users based on their preferences. While traditional RSs rely on user feed- back signals, conversational RSs interact with users in natural language. In this work, we develop LLLM, an aPPealing, Precise, creference-compreheette and Prioritized language model which endorses recommended items by emphasizing specific item characteristics and their coverage to a user's 's preferences. L*LM uses an embedding representation of a user's preferences to generate responses that are appealing, to the user's preferences. P'LM employs joint reward function to measure precision, appeal, preference coverage and prioritization of preferences, which are used Al-based feedback in reinforcement learning-based language model framework. On the Movielens 25M and Amazon Product Review datasets, PLLM delivers more appealing and tailored endorsements users, as determined by auto-critic and rater evaluations.
and tailored
1 Introduction
Recommender systems (RSs) have emerged dominant way in which users discover content, products, and services (Resnick & Varian 1997). Traditional RSs match candidate items to users based on estimates of their item preterences. However, these estimated preferences are often based on user behavioral signals (e.g., clicks, consumption, ratings,
'Wok performed while at Google Research. *Corresponding author: vinlamchow@google..com

---

Published as a conference paper at COLM 2024
purchases). Unfortunately, this provides little opportunity for RS to elicit preference information from users or to explain its recommendations. Conversational RSs employ natural language to facilitate more effective communication between RSs and their users (Sun & Zhang 2018 Lei et al. 2020 Shen et al. 2023).
The emergence of language models (LMs) as powerful paradigm for user interaction (Li et al. 2018 Friedman et al. 2023) suggests their use in conversational RSs. However, this requires LMs to engage with users personalized manner, consistent with their preferences. In this paper, we explore the use of LMs to enrich the user's 's RS experience. We develop techniques that allow an LM communicate the nuances of recommended items (as determined by back-end RS) to a user, detailing their features and benefits, and explaining their alignment with the user's preferences. Such personalized LMs are not meant to "convince" users in the traditional sense, but rather to articulate the genuine, relevant merits of a recommended item relative to the user.
Although existing RS technologies are adept at predicting a user's favored items, a personalized LM can provide an enriched experience by tailor- ing suggestions to what a user genuinely needs and values. However, number of challenges must be addressed in this endeavor:
Toy Story
Plot: TToy Story" an animated
festure fim produced by Pixar (...)
Reviow:
From the moment the first
lights up
(...
(i) the in- tegrity and accuracy of item's 's information is paramount; (ii) the LM's endorsement should be appealing; (iii) the LM should present a rea- sonably comprehensive portrayal of the item by articulating its merits and drawbacks, with focus on coverage of the user's 's preferences; and finally (iv) the endorsement should prioritize its focus on user preferences based on aspects that matter most to the user. With these four criteria, RS endorsements can facilitate informed user decisions and thus improve user engagement. In this work, develop LM framework for RS endorsements centered on these principles.
Embedding:
[-0.488, 0.089, -03344 0.198)
User Embedding [0.077, 0,471, 0.386 0.041]
PLLM
Recommendation Endorsement
Our contributions three-fold. First, we quan- tify the four criteria above to allow systematic evaluation. Second, leveraging recent advances in reinforcement learning (RL) from Al feedback (RLAIF) (Lee al.) (202)) and reward aggrega- tion, we develop an LM fine-tuning methodol- ogy better align an LM with the four crite- ria. The resulting P*LM framework not only em- bodies semantic skill, but also understands user preferences as encoded by an RS embedding, providing factual, appealing, and tailored dorsements. Finally, using the MovieLens 25M (Harper & Konstan (2015) and Amazon Product Review et al. 2019) datasets, we show that L'LM can power RS endorsements that promote customized, relevant, and holistic interactions with users.
Dive into "Toy Story," Pixar's visually mesmerizing cdyssey that coptures rich stmosphere where loys life,
Precision
Proference Priortiization
Reward Model
Rewarr Model
Appeal
Proforance Coverage
Reward Model
Reward Model
Figure 1: P*LM produces precise, appealing. comprehensive, and tailored endorsements from user embedding and item description, as demonstrated with Toy Story. It relies solely the user embedding, not requiring textual user profile.
We begin with an introduction of RSs, and the endorsement task description in Sec. 2 We develop the supervised fine-tuned RS endorsement LM in Sec. 3) In SSc. 4 we describe P'LM, our RL framework, which leverages Al feedback to optimize the four endorsement criteria. Finally, in Sec. 5 we demonstrate the effectiveness of P'LM generating factual, appealing, and user-tailored recommendation endorsements on the MovieLens 25M and Amazon Product Review datasets, using both automatic and rater evaluations.
2

---

Published as a conference paper at COLM 2024
2 Problem Formulation
We provide background on RSs and formulate the recommendation endorsement problem through our four principal criteria.
Recommender Systems. To model user preferences over items in an RS, we assume standard collaborative filtering (CF) model (Su & Khoshgoftaar 2009), where user behavioral data (e.g., clicks, ratings, etc.) are used to learn user and item representations which are combined to predict user's s preference for a given item. We assume users U, items I, and (usually sparse) |Z] x [2[] ratings matrix R = { stars) is user u's rating of item A CF method learns both user and item embeddings from R, where the embedding representation i of E I captures its latent attributes, and the representation u of u E 21 reflects their utility (or preference) function over these attributes. Suitable CF methods include matrix factorization (Mnih & Salakhutdinov 2007), or neural CF (Rendle et al. 2020 He et al. 2017 Beutel et al. 2018), whose Ewo-tower model processes users and items with separate, but co-trained, networks to produce respective embeddings u and
'uui F 0}, where ru, (e.g., 1-5
Endorsement Task. A key question when using LMs for endorsing personalized recommen- dations is how to effectively exploit the information captured by the RS embedding space to generate factual, appealing, and user-tailored endorsements. To measure the LM's efficacy on this task, we develop four metrics: precision, appeal, preference coverage, and preference prioritization. These assess the LM's ability be factual, appeal to users, align with user preferences, and prioritize specific user preferences, which adheres to the narralive paradigm (Fisher 1985), which emphasizes storytelling, coherence, and fidelity as building blocks for meaningtul communication. We elaborate on the four criteria below.
Precision (Prec). Recommendation endorsement can be viewed as a form of abstractive summarization (Zhang et al., 2020a; Liu et al. 2022): its text should capture item charac- teristics that explain why a user would benefit from the item. Factuality is critical to RS integrity- an endorsement must describe genuine merits and drawbacks of the item, rather than persuasive distortions or hallucinations.
Appeal (App). Increasing attention has been paid to enriching recommendations to appeal to users (Felfernig et al. 2007 Zhang et al. 2020b). To the extent that we do not sacrifice user benefit and well-being, appealing endorsements have value if they encourage users to accept recommendations with significant utility. The Appeal of an endorsement depends on various factors including stylistic variation (eloquent pitches vs. dry factual summaries) or breadth/ /deth of explanation, which may be challenging to explicitly quantify.
Preference Coverage (Peov), An endorsement should describe both the positive and negative attributes of an item that the user cares about. This ensures the user can make appropriate and informed decisions about the recommended items, and promotes the interpretability and overall integrity of the RS. For example, an endorsement that only articulates positive attributes of an item w.r.t. user's preferences, while ignoring relevant negative aspects, might be factual and appealing to the user, but could easily mislead the user into making a poor decision. Preference coverage measures the correlation between an endorsement and user's 's interpretable preferences. An LM that excels at preference coverage relates recommended items to users by matching facts with system's 's belief about user preferences.
Preference Prioritization (Ppr). A recommendation endorsement should prioritize communicat- ing aspects of an item that matter most to the user given their preferences. Preference prioritization focuses on the RS's assessment of the "most important" attributes to the user. This can be captured by the concept of a user's 's perceived utility for an item, i.e., after reading an effective endorsement text, the user's belief about the item's utility should be better than merely reading the full item descriptions. An effective endorsement is one that focuses on attributes that user cares most about, and conveys how the item matches their preferences. Quantify- ing preference prioritization is challenging, as it depends on user interpretation of the item endorsements, not merely on their preferences. Finally, though preference prioritization overlaps with coverage, it emphasizes the relevance to user's preferences.
3

---

Published as a conference paper at COLM 2024
3 Endorsement using Supervised Learning
We first describe a supervised learning approach to solving the RS endorsement task. We assume a dataset D = { (T(K) (,()), u(k), )
where each is a textual description of item i E I (e.g., description, user reviews); i is the CF embedding of i; u is the CF embedding of user u E Ut and Y is the endorsement text. (SeeAAppendis for details on D's generation.)
k=1
Given a set of items, the role of LM is to predict the probability of response Y given the item and user contexts. A standard way to apply an LM to this task is to use a transformer (Wolf et al., 2019) T to encode an item's text as an Ny-legghh sequence of embeddings (Z0,..., -1) induced by T's attention layers, where Nj is an upper- bound on the length (number of tokens) of an item description I. The endorsement text 1N-1 is sampled token-by-token in an auto-regressive manner using a decoder Y= = {yn n 0
= I Yy ,o. -nn.;/2)) 1;2), where yo is fixed start-of-sentence token (Chien & Kuo 2019). J. However, this architecture assumes all inputs are textual. In order to generate personalized endorsements, we wish to include the item and user CF embeddings as inputs, as they capture important behavioral information.
Y: Y Y= Y((
Personalizing the LM via User Profile Prompting. One approach to overcome this limita- tion is to assume access to a textual user profile, which maps a user u (or their CF-embedding u) to textual preference description [Ux(u)) | (e.g., K bullet-point preference texts that reflect u). One may extend this approach to incorporate personalization by encoding both the item and user texts with the text-only transformer T. However, the quality of this per- sonalized LM hinges on how well user preferences can be represented in natural language. Indeed, crafting effective user summaries is highly non-trivial task (Radlinski et al. 2022). Instead, we directly inject the user CF embeddings into the LM, as described below.
Injecting Embeddings into the LM. We directly inject user-item behavioral information into seq2seq LM (Vaswani et al. 2017) to generate personalized endorsements. Specifically, augmentt the standard LM with adapters (Pfeiffer et al. 2020) W,, Wu :V - Z, to induce a new LM 4- := Y o (T X W, x Wu) (Tennenholtz et al. 2024). . Here, T maps text-input tokens to Z, whereas W, (resp., Wu) maps item (resp., user) CF-embedding vectors V to Z. Importantly, T, W,, and Wy map tokens and CF vectors to a common space so that their relationship can be captured by the transformer's s attention mechanism.
Supervised Training. Treating the LM & as a factored distribution of item-user information over endorsement tokens, one way to train Φb is with behavioral cloning (BC) (Sasaki & Yamashina 2020), which maximizes the conditional log-likelihood w.r.t. to dataset D:
PBC(- Iyo u) E arg min Lcond (Φ) == - E (Li,,,)) log ( yor...,, V......u-iiiiii
Σ
Here, & predicts P(Y| yo, L, of the endorsement Y conditioned on item context (1, and user embedding u. The text-based sub-models within & (encoder T and decoder W) can be warm-started with pre-trained LM checkpoints (e.g., PaLM2) which encode rich semantic information. The adapter layers W, and Wi are trained from scratch.
Two-Stage Training. As we use distinct sub-model initialization schemes, the pre-trained LM modules have established embeddings in the language space, while the newly initialized adapters require more training to map CF embeddings to another latent space. Thus, training the full model may cause the LM to disregard the embedding contexts (with diminished adapter weights) and fall back to the pre-trained token generation probabilities. Effectively, & degenerates to a non-contextual LM that ignores user embedding input. To alleviate this, al. (2024): we first perform cold-start
we use the two-stage BC procedure Tennenholtz training of the adapters W, and Wu on an auxiliary task, fixing transformer parameters (T, V) as non-trainable; we then train the entire model using the maximum likelihood objective. Alternatively, we can leverage parameter-efficient training approaches (e.g., low- rank adaptation (Hu al. 202T) for greater training efficiency at the second stage). This two-stage training is critical for LM convergence (Ténnenholtz et al. 2024).
4

---

Published as a conference paper at COLM 2024
4 Optimal Endorsement using PLLM
While the supervised approach above may generate seemingly good endorsements, it may not adhere to the four principles outlined in Sec. 2 We now develop rigorous learning framework, namely aPpealing, Precise, Preference-oompreeennstee and Prioritized Language Modeling (P+LM), to optimize an endorsement LM using RLAIF. (See Algorithm 1 and Fig. 1 for an overview of LM.) Recall that our objective is to train an LM to articulate factual and relevant item nuances that align with the user's 's preferences. Here, the LM acts as a policy which maps text inputs and user /ste/ behavioral embedding vectors endorsements.
In RLAIF, rewards are computed using feedback from another L.M. RLAIF can effectively align LMs with specific metrics, using labels gener- ated by off-the-shelf LMs rather than by human raters (i.e., RL fine-tuning with human feedback (RLHF)). That said, recent work has shown that hy- brid human-Al preference models, to- gether with impooiig fine-tuning, outperforms traditional supervised fine-tuned baselines and offers addi- tional benefits (Lee et al., 2023 Bai et al.) 2022 Zhu et al. (023). Using the four criteria for LM-based RSs outlined in Sec. 2. we develop four reward models to train and evaluate LMs w.r.t. precision, appeal, prefer- ence coverage, and preference prioritization. We then devise an RLAIF technique to fine- tune such LMs with joint reward model that embodies these four components.
Algorithm 1 P'LM Learning Framework
1: Input: Off-the shelf NLI model; pretrained LM.
2: Data Generation: Generate supervised data us- ing self-critiquing (App. C for precision, appeal, preference coverage and prioritization.
3: Supervised Learning: Finetune LM using data from previous step and user embeddings (Sec. 3).
4: Reward Models: Generate preference labels us-
ing LM for appeal, preference coverage, and pref- erence prioritizatinn and train reward models (Sec. 4.1).
5: RLAIF: Finetune LM with RL, using the aggre- gated RM from previous step (Sec. 4.2). Use finetuned supervised model of step 3 as anchor model.
4.1 Reward Models
We describe the reward models (RMs) corresponding to each of the criteria defined in Sec. 2
Precision (Prec) RM Inspired by Roit et al. (2023) and Honovich et al. (002)), we evaluate factuality in our LM-based RS using an entailment reward (Bowman et al., 2015). Unlike widely-used metrics ROUGE Lin 2004)), which are ineffective at hallucination de- tection, we adopt a textual entailment, or natural language inference (NLJ) metric to measure factuality of our generated text, viewing it partial summary of an items's description. Specifically, we define the NLI score, i.e., Prec(Y; as the probability of entailment w.r.t. classifier trained on entailment datasets MacCarreyy & Manning (2007),
Appeal (App) RM. To assess appeal, we use a dataset of pairwise human machhne demon- strations (see Appendix for details on its construction). We develop an appeal model which scores the generated text Y to assess its appeal, using learning from human/ AI feedback
} [Papp) be the labeled dataset reflecting the relative appeal of two texts Yp, Y,, where Yo is more appealing given item de- scription I, i.., Y, > I. Assuming these relationships are governed by a model AYp(Y; we parameterize it via Bradley-Terry (Huang et al. 2006), giving an appeal distribution: (Ap(Ap((Y;;)))))
(LAIF) (Crisstaano et al. 2017). Let Dapp {(Y()) y(k), ;1)
}=)
Papp (Y > Yj: exp(App(Y,,)))/xxppppp (ex)))))xx((ppp)) (Ym: 1) + (Ap(App(Y) I))]. By formulating the problem as a binary classification task, the learned latent model minimizes the negative logelineldhood loss: (Ym, log or( (Ypp((Yw:))) I) I)) To reduce variance, we
subtract this model with its population mean so that (y. [App(Y:1)] = 0.
Here, "Precision" denotes factual articulation of item information, distinct from its common usage in information retrieval MI..
5

---

Published as a conference paper at COLM 2024
Preference Coverage (Pcov) RM. To explicitly relate item features to user preferences, for each coan-ssr-endorsemeee tuple (I, u, Y), we assume access to user profile {Ux( (a mapping from a user's CF embedding to K textual preference descriptions). Quantifying preference coverage can then be accomplished by andorsement-prefereenn
similarity match- ing. Specifically, for each tuple, the preference coverage score s can be defined as either the average cosine similarity of the encoded endorsement T(Y) with K encoded user preference texts (T(U;(u))) (u))) Alternatively, we use black-box estimated likelihood, egg, via prompting an LLM with query: "Does endorsement Y sufficiently cover the preferences depicted by user profiles (Uz(u)) and predicting the likelihood of the answer.
We train scoring model Pcov(Y; u) by minimizing an l regression loss sL. Pcov (Yov(Y,,,u)) u)) We can also employ LAIF (Christiano et al.) 2017) to learn a pref- erence coverage model. Assuming the endorsement text Y covers more preference attributes in the user profile than item descriptions themselves (i.e., Y > I[u), the model Poov IY; u) can be learned by minimizing the regularized negative log-likelihood: (L Pcov(Y;L, for À >
(yy log or (PCooo(V u, (cov(I;II, u)) which also enforces the equality constraint s = uY,
Preference Prioritization (Ppr) RM. We define a scoring model Ppr u) to quantify per- ceived utility, or how an endorsement connects an item with the most important preferences of the user. One special case is when the endorsement is complete item description, i.e, Y= I. In this case, we may assune that Ppr is simply the item utility, iee, Ppr(I;i, = -i u. Alternatively, we can employ LAIF (Christiano et al. 2017), leveraging pairwise feedback to learn Ppr. Using the dataset D of item descriptions I, user-item CF embeddings u, i, and endorsement texts Y, and assuming this text has greater preference prioritization than item a Bradley-Terry model Ppr(Y;i, u) learned by minimizing the negative log-likelihood: - (Y log (Ppr(Y,,, (Y; Ppr(I;i, u)). This loss can be further regularized with the term A - (Ii..)) (Ppr(I;i, u) - i. u),, for À > O, which enforces the constraint:
descriptions (i.e., Y I u P
4.2 Reward Aggregation and RL-based Fine-tuning
In multi-objective RL, reward models can often be aggregated using linear scalarization (Peschl et al. 2021), which solves for an optimum on the convex Pareto frontier. Given endorsement text Y, item description I, and user-item embeddings iu, we define the multi-objective reward as:
1 Hat 0
yx= FOS]; O.W.,
r r(Y;L,,,
(1)
where f/1, 1/2, 113, 174 2 are the importance weights for the component rewards.
Wang et al. (0224) show that, when the reward models are either pointwise logit or pairwise logit-difterence models (as in our case), one can simply employ the reward aggregation technique with all importance weights set to 1, without losing optimality. To show this, Wang et al. (2024) first apply a log-sigmoid transformation to all reward models. In our case, the posnt-wise Prec reward (learned with logistic regression) becomes log o (Prec(YY; and corresponds to the log-probability of the positive entailment, i.e,, log PEntail 1), where Entail is the entailment random variable. Similarly, for the pairwise rewards that characterize appeal, preference coverage, and preference prioritization (trained to maximize logit differences between a pair of endorsement scores), the log difference between the score of endorsement Y and its baseline counterpart can be written as log I, This quantity corresponds to the log-likelihood of
R RYYrefi i, u)), ), for R E (App, Prr,,rroo positive preference, i.e., log P(Prre = 1), where Rpref is a random variable which indicates endorsement Y is preferred over the reference text Yru under the criterion of interest.
Instead of comparing the endorsement text with item description, one could construct a dataset with two texts and labeled rating order (see Appendix (fo details).
Response Yfef can either be the label in offline data 2 the output of pre-trained model.

---

Published as a conference paper at COLM 2024
Finally, assuming conditional independence of random variables (Entail, Appreef PpTpref PcoVPret), adding the transformed rewards is equivalent to a logical-AND operation, ie,
rY == logoPrec(Y; +
Σ
log o (R); -
Re (App.Ppr..coor
log ((ntail Ap 1, Pp 1,PcoVrref 1),
which implies that the joint reward aligns across all aspects. This, in turn, shows that all importance weights 7/1, 1/2, //3, 1/4 can indeed be set to 1.
Given the LM (Y L, and joint reward model r(),,,i,u) the goal of LM fine-tuning is to maximize the overall quality of the generated text: max E(Li,)) Es(y)ti (T(,,,,)(((YY)))) (r(Y;,, I, u)). Using the MDP framework (Puterman 2014), this problem can be solved using on-policy RL, e.g,, REINFORCE (Williams) 1992,, with trajectories generated by the current LM.
One risk of RLAIF fine-tuning is overfitting the LM to the reward model, thereby de- grading the LM's inherent "semantic skill." " To alleviate this, we add a KL regularization term (Ouyang et al., 2022; Stiennon et al., 2020) to the MDP objective, comparing the LM D and the supervised fine-tuned model (B u). Leveraging the auto- regressive nature of LMs, KL regularization is applied over the entire MDP trajectory, reducing the objective to
P E arg max [(Φ) := E(Li.u) EoyyLL
((X:I,,
Blog
ΦB (Y
corresponding to a KL-regularized MDP, which can be solved by policy gradient methods.
5
Experiments
We conduct empirical evaluation of LLM, assessing whether our reward models can sig- nificantly increase the precision, appeal, preference coverage, and preference prioritization of item endorsements. We conduct experiments on two recommendation datasets: (i) MovieLens 25M (Harper & Konstan 2015), which contains ratings of 62,423 movies by 162,541 users, and (ii) Amazon Product Review (Ni et al. 2019), from which we use the "Clothing, Shoes and Jewelry" category, containing 5.7M ratings of 1.5M products. We use user-item interactions to generate item descriptions, user-ereeerence texts, and recommen- dation endorsements by prompting PaLM2-L LM (Google al. 2023) (see Appendix C for details). The resulting datasets have four components: (1) item descriptions I, (2) user-item behavioral embeddings ui, (3) user preference texts U(u), and (4) endorsements Y.
We experiment with PaLM2-XS (Google et al., 2023) and incorporate user and item embed- dings (see 3) by augmenting the model with adapter layers. To simplify hyper-parameter optimization, we use reward aggregation (Fquation 2 to combine the rewards. We also experiment with rewards combined with various importance weights (see Appenddi A).
To demonstrate the efficacy of P*LM, we compare it with the following SOTA baselines: (i) PaLM2-L, pre-trained text-only large LM, prompted using item descriptions, user preference texts and instructions to generate a response that adheres to our four endorsement principles; (ii) Supervised Fine-Tuned with Text (SFT-Text), a PaLM2-XS model fine-tuned with the datasets above, with explicit user-item texts as input; (iii) Supervised Fine-Tuned
(SFT), a PaLM2-XS model fine-tuned to use user-item embedding vectors We measure performance of all methods using both model- and Juman evaluation.
Model-Based Evaluation. We conduct model-based (MB) evaluations on both the movie and product endorsement tasks using our four RMs from Section 4.1 i.e. Prec, App, Ppr and Pcov.
"Text-based approaches are inherently limited by the specificity of textual user profiles. Defining optimal textual profiles is beyond this work's 's scope. On the other hand, by directly injecting user embeddings into the word-token embedding space, P*LM can leverage diverse user data, reducing re- liance on specific prompts. The output from adapter Wi efficiently represents user details, minimizing dependence on profile formatting.
7

---

Published as a conference paper at COLM 2024
Table Evaluation of MB metrics on Amazon dataset: Prec scores from NLI model entailment probabilities; average logits for other metrics from three RMs.
Table 2: Human evaluation: Model vs. SFT win rates, covering Prec, Ppr, App Ratios, Pcov in- crease, and overall quality (see Appendix E.
Evaluation Metrics
Evaluation Metrics
Method
Prec
Ppr
App
Pcov
All
Method
Prec
Ppr
App
Peov
Amazon MLens
PaLM2-L
0.31
0.35
0.31
8.3%
0.33
PaLM2-L
0.34
-1.73
-5.47
-5.48
SeT-Text
0.5
0.5
0.50
2.1%
0.49
SFT-Text
0.37
-1.70
-5.48
-5.42
P*LM (ours)
0.77
0.64
0.69
35%
0.71
SFT
0.37
-1.59
-5.54
-5.33
PaLM2-L
0.43
0.40
0.49
-17%
0.47
PLLM (ours)
0.79
-1.47
-5.27
-4.93
SFT-Text
0.52
0.27
0.37
-32%
0.44
LLLM (ours)
0.73
0.67
0.64
17%
0.81
We report the scores of endorsements Y generated by each model on a held-out, unlabeled dataset Diest consisting of 73 (Amazon) or 96 (Movielens) non-overlapping user-item pairs. The average results are presented in the main text, with detailed results and confidence intervals available in Appendix[A] We also assess the relative improvement each LM over the standard SFT baseline by computing (a) the win rate (number of occurrences on which the tested LM outperforms SFT), and (b) the absolute increase (of LM's score improvement).
To validate the efficacy of our four reward models, in reflecting corresponding preference values in the context of recommendation endorsement, we first assess their accuracy, We start by prompting the Gemini Ultra LLM (Google 2023) (see Appendix A. for details) to generate synthetic test dataset of endorsement pairs with clear preferences on precision, appeal, preference coverage and prioritization respectively. Then the RMs are evaluated based on their abilities to distinguish the endorsements with respect to the corresponding principles. The results presented in Table (Appendix (A) illustrate that the RMs can effectively classify all the test cases, thus certifying the validity of these models.
Table depicts the MB evaluation results based on the four principles of endorsements on the Amazon dataset. It highlights the ro- bust performance of LM in all four pivotal dimensions: precision, appeal, preference cov- erage and prioritization. P*LM attains the highest precision score by a large margin, un- derscoring its ability to mitigate the risk of hal- lucinating information about recommended items. It also fares well all other categories, corroborating the hypothesis that RL-based fine-tuning, when paired with reward aggre- gation, manages to align the endorsement LM with respect to all the reward models. Simi- lar evaluation on the MovieLens dataset can be found Table 5 (Appendix A). Though similar trends (especially in the precision cate- gory) are observed, PaLMM-Lachieves achieves the best performance in appeal, preference coverage and prioritization. This is unsurprising be- cause (i) PaLM2-L is much larger than LLMM; (ii) Different from the specialized user-item in- formation associated with Amazon products, there is ample public data about movie con- texts and generic user preferences available for pre-training such that PaLM2-L can adeptly generate personalized endorsements. These observations are also correlated with the findings highlighted in Figures 5 in Appendix A) for MovieLens, Figures] and 4 (Appendix A) for Amazon, which elucidate
Prec
Ppr
0.8
0.7
0.5
0.4
App
Pcov
0.9
0.7
0.6
0.4
PalM2-L
SFT-Text
P4LM (ours)
PaLM2-L
SFT-Text
PILM (ours)
Figure 2: Win rates of MB scores against SFT from the Amazon dataset. Dotted lines at 0.5 represent ties. The win rate, as defined Ap- pendix B is calculated by comparing the MB score from output against the SFT score.
8

---

Published as a conference paper at COLM 2024
both the win rates and absolute score increases of each method when compared with the SFT baseline.
Comparing SFT and SFT-Text, both methods exhibit comparable performance across most metrics, except for the preference coverage and prioritization scores. SFT extracts user preferences from the behavioral embedding to generate personalized endorsements, while SFT-Text generate outputs that attends to specific user preferences in the input prompt. How- ever, the dependency on text inputs introduces limitations, omitting subtle user preferences captured by their behavioral embedding. This difference suggests that text descriptions are unable fully reflect the intricate user preference information in embeddings.
Human Evaluation. Table showss the human evaluation on models trained with Movie- Lens and Amazon datasets, in terms of their relative preferences over the SFT baseline by human raters. We asked raters to compare two endorsement texts, one from SFT and another from a test model, to assess relative performances. Raters evaluated these endorsements using the same four criteria as in the model-based evaluation, with an additional criterion for overall quality comparison (see Appendix E for details).
Raters assess the endorsements generated by different models, including LLM For the LLMs trained with Amazon data, L'LM unanimously outperforms other baselines across all the four endorsement principles, in most cases with significant margins. In particular, human evaluation much favored the precision and preference coverage of LLM, both consistent with the model-based observations. Interestingly, in the MovieLens experiment human evaluation highlighted P*LM's superior performances across all the metrics (see Table 2 while model-based evaluation only indicated the same trend on factuality (see A 5 Table These discrepancies may be attributed to the approximation errors of the reward models (Ppr App, Pcov), which are sensitive to their preference training data. Nevertheless, raters showed a strong preference for L'LM in terms overall text quality.
Human evaluation also favored SFT over SeT-Text in the Amazon domain, but is indifferent in the MovieLens Lens domain. This may likely due to SFT-Text, which shares inputs with PaLM2-L, is also trained with its generated dataset and thus limited by PaLM2-L's semantic skills. Contrarily, SFT, while trained with the same data, can also leverage additional information from user's behavioral embeddings to enhance the LM's understanding about user's preferences, giving it an edge over SFT-Text, especially in preference coverage and prioritization scores. This observation is more pronounced in the Amazon experiment, of which the user-item information is less available in the public data.
Ablation Studies Our ablation studies (Table 7 for MovieLens, Table 8 for Amazon) show that LLLM trained using RLAIF with a single RM scores higher MB evaluation for the corresponding RM of interest. Intriguingly, in both domains, models trained solely preference prioritization or preference coverage excel on both metrics, suggesting correlation between these two categories, which is somewhat expected (see Sec. 2 Human evaluations on models trained with individual RMs are detailed in Appendix x A Table 9 Notably, ranking of these models by raters differs from the MB ablation. For example, on MovieLens the Pcov-model (which targets preference coverage) and on Amazon the Ppr-model (which targets preference prioritization), surprisingly score highest on all categories in human evaluations. Human raters are somewhat biased towards endorsements that relate item features to user preferences, heavily influencing their judgment of overall quality.
Related Work
Our work connects research from personalized RSs, LMs, RL, recommendation integrity.
Personalized RSs. RSs permeate e-commerce, content systems, social media, etc. CF
methods still play a prominent role (Schafer et al., 2007 Mnih & Salakhutdinov) 2007). ,
with deep learning methods (e.g., neural CF (He et 2011), , dual encoders (Yi al. 2019
Yang et al. 2020)) extending traditional CF methods. RSs are increasingly modeling more

---

Published as a conference paper at COLM 2024
nuanced and often sequential aspects of user behavior (van den Oord et al. 2013 Covington et al. 2016 Gauci et al. 2018: le et al. 2019 Chen et al.) 2019a).
Conversational RSs & LMs. Conversational RSs augment traditional RSs with conversa- tional agent that supports user interaction through natural language dialogue (Chen et al.
2019b Zhou et al. 2020 Lei et al. 2020 Li et al. 2018 Sun & Zhang 2018 Christakopoulou
et al. 2016). Such systems provide a richer means of understanding a user's preferences,
allowing the natural refinement of recommendations and more user control. Conversational RSs are a ready target for the use of LMs, t leveraging LMs in RSs is relatively recent effort. With the advance of transformer architectures (Vaswani et al. 2017 Wolf et al./ 2019), LMs have found use cases beyond typical NLP tasks, including the synthesis of textual data with user preferences enhance RS personalization and expressiveness (Jaech & Ostendorf 2018 Xia et al. 2023). Our work fits into this space, aiming to generate appealing narratives that effectively communicate a recommendation's coverage to the user.
Transparency and Factuality in RSs. Maintaining integrity in RSs is vital, but challenging, given the potential for RSs to inadvertently mislead users or reinforce biases (Abdollahpouri
et al. 2019 Shen et al. 2023 Cabello et a 2023). Increasingly, research has delved into the fairness, transparency, and interpretability of RS algorithms (Beutel et al. 2019 Ghazimatin
et al. 2020-Chen Chen et al., 2023). Our work emphasizes factual and precise recommendations that articulate genuine merits rather than distortions designed to appeal to users.
appealing strategy for refining models beyond supervised tine-tuning (Williams, 1992 Ranzato et al. 2016). The RLHF methodology (Christiano et al. 2017 Bai al. 2022) allows model responses to be first rated by human evaluators, and then used to fine-tune models (with RL). Inverse RL (Abbeel & Ng. 2004) has been used to acquire objectives from expert demonstrations in textual settings (Daniels-Koch & Freedman 2022 Sun 2023). Finally, there is growing interest Al-based teedback, where responses are labeled by models rather than human raters (Lee et al. 2023 Bai et al. (2022). Our work collectively shows tremendous potential for using RL drive LMs for better alignment with user preferences and RS objectives.
RLHF & RLAIE. The integration of RL with LMs has emerged
7 Conclusion
We have studied language modeling for generating recommendation endorsements, propos- ing novel reward models which quantify critical aspects of effective personalization, and training our LM using RLAIF with these rewards. The resulting LM, L'LM, not only sup- ports language interaction with a user, but understands latent user preferences as encoded in CF embedding space. provides item endorsements that are factual, appealing, preference comprehensive and prioritized, connecting relevant item attributes with user's 's prefereenss, and increasing the odds of users accepting high-value recommendations. Experiments on benchmark datasets demonstrated the efficacy of L'LM across all these dimensions.
Our work is a step toward creating robust conversational RS that can identify the intricate (and most relevant) connections between an item's features and a user's 's preferences and explain them in an appealing fashion. Future work includes: improving PLLM's ability to generate longer responses, e-g., by moving beyond single-shot, autoregressive decoding; extending RL fine-tuning to handle multi-turn conversational recommendations; devel- oping better reasoning capabilities that trade off user-item preferences and constraints; and expanding LM's functionality beyond recommendation, to include technical support, negotiation, etc.
10

---

Published as a conference paper at COLM 2024
References
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, Pp- 2004.
Himan Abdollahpouri, Masoud Mansoury, Robin Burke, and Bamshad Mobasher. The impact of popularity bias on fairness and calibration in recommendation, arXio preprint arXiv:1910.05755, 2019.
András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with bellman-residual minimization based fitted policy iteration and single sample path. Machine Learning, 71:89-129, 2008.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitu- tional ai: Harmlessness from feedback. arXiv preprint arXio:2212.08073, 2022.
Alex Beutel, Paul Covington, Sagar Jain, Can Xu, Jia Li, Vince Gatto, and Ed H Chi. Latent cross: Making use of context in recurrent recommender systems. In Proceedings of the cleventh ACM international conference on web search and data mining, pp. 46-54, 2018.
Alex Beutel, Jilin Chen, Tulsee Doshi, Hai Qian, Li Wei, Yi Wu, Lukasz Heldt, Zhe Zhao, Lichan Hong, Ed H Chi, al. Fairness in recommendation ranking through pairwise comparisons. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2212-2220 2019.
Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, pp. 632-642, 2015.
Laura Cabello, Anna Katrine Jorgensen, and Anders Søgaard. On the independence of association bias and empirical fairness in language models. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pp. 370-378, 2023.
Salvatore Carta, Anselmo Ferreira, Alessandro Sebastian Podda, Diego Reforgiato Recupero, and Antonio Sanna. Multi-dqn: An ensemble of deep q-learning agents for stock market 2021.
forecasting. Expert systems with applications,
Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. Bias and debias in recommender system: A survey and future directions. ACM Transactions on Information Systems, 11(3):1-39 2023.
Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed Chi. Top-k off-policy correction for reinforce recommender system. In 12th ACM International Conference on Web Search and Data Mining (WSDM-19), pp. 456-464, Melbourne, Australia, 2019a.
Qibin Chen, Junyang Lin, Yichang Zhang, Ming Ding, Yukuo Cen, Hongxia Yang, and Jie Tang. Towards knowledge-based recommender dialog system. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Join! Conference on Natural Language Processing, EMNLP-IJCNLP 2019, pp. 1803-1813, 2019b.
Jen-Tzung Chien and Che-Yu Kuo. Markov recurrent neural network language model. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 807-813. IEEE, 2019.
Konstantina Christakopoulou, Filip Radlinski, and Katja Hofmann. Towards conversational recommender systems. In Proceedings of the 22nd ACM SIGKDD international conference knowledge discovery and data mining, pp. 815-824, 2016.
11

---

Published as a conference paper at COLM 2024
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for YouTube rec- ommendations. In Proceedings of the 10th ACM Conference on Recommender Systems, pp. 191-198, Boston, 2016.
Wojciech M Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant Jayakumar, Grzegorz Swirszez, and Max Jaderberg. Distilling policy distillation. In The 22nd international conference on artificial intelligence and statistics, pp. 1331-1340. PMLR, 2019.
Oliver Saniels-Koch and Rachel Freedman. The expertise problem: Learning from special- ized feedback. Xixiv preprint arXiv:2211.06519, 2022.
Alexander Felfernig, Gerhard Friedrich, Bartosz Gula, Martin Hitz, Thomas Kruggel, Ger- hard Leitner, Rudolf Melcher, Daniela Riepan, Sabine Strauss, Erich leppan, et al. Persua- sive recommendation: serial position effects in knowledge
recommender systems. In Persuasive Technology: Second International Conference on Persuasive Technology, PERSUA- SIVE 2007, Palo Alto, CA, USA, April 26-27, 2007, Revised Selected Papers 2, pp. 283-294. Springer, 2007.
Walter R Fisher. The narrative paradigm: An elaboration. Communicationss Monographs, 52 (4):347-367, 1985.
Luke Friedman, Sameer Ahuja, David Allen, Terry Tan, Hakim Sidahmed, Changbo Long, Jun Xie, Gabriel Schubiner, Ajay Patel, Harsh Lara, al. Leveraging large language models in conversational recommender systems. arXiv preprint ar kie:2305.07961, 2023.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Jason Gauci, Edoardo Conti, Yitao Liang, Kittipat Virochsiri, Yuchen He, Zachary Kaden, Vivek Narayanan, and Xiaohui Ye. Horizon: Facebook's 's open source applied reinforce- ment learning platform. arXiv preprint arXiv:1312.5602, 2018.
Azin Ghazimatin, Oana Balalau, Rishiraj Saha Roy, and Gerhard Weikum. Prince: Provider- side interpretability with counterfactual explanations in recommender systems. In Pro- ceedings of the 13th International Conference on Web Search and Data Mining, pp. 196-204, 2020.
Gemini Team Google. Gemini: A family of highly capable multimodal models, 2023.
Rohan Anil Google, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexan- dre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette- Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Diaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby,
12

---

Published as a conference paper at COLM 2024
Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. PaLM technical report, 2023.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In Interna- fional conference on machine learning, pp. 1861-1870. PMIR,, 2018.
Maxwell Harper and Joseph Konstan. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tris,, 5(4):1-19, 2015.
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In Proceedings of the 26th International Conference on World Wide Web (WWW) 2017, pp. 173-182. ACM, 2017.
Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. TRUE: Re-evaluating factual consistency evaluation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies, (NAACL-2022), pp. -905-3920, 2022.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
Tzu-Kuo Huang, Ruby C Weng, and Chih-Jen Lin. Generalized bradley-terry models and multi-class probability estimates. Journal of Machine Learning Research, 7(1), 2006.
Eugene le, Vihan Jain, Jing Wang, Sanmit Narvekar, Ritesh Agarwal, Rui Wu, Heng-Tze Cheng, Tushar Chandra, and Craig Boutilier. SlateQ: A tractable decomposition for reinforcement learning with recommendation sets. In Proceedings of the Twenty-eighth International Joint Conference on Artificial Intelligence (IJCAI-19), pp. 2592-2599, Macau, 2019.
Aaron Jaech and Mari Ostendorf. Personalized language model for query auto-completion. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume Short Papers, pp. 700-705. Association for Computational Linguistics, 2018
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00456, 2019.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hattield-Dodds Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language models (mostly) know what they know, 2022.
Wang-Cheng Kang, Jianmo Ni, Nikhil Mehta, Maheswaran Sathiamoorthy, Lichan Hong, Ed Chi, and Derek Zhiyuan Cheng. Do Ilms understand user preferences? evaluating Ilms on user rating prediction, 2023.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy g-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
13

---

Published as a conference paper at COLM 2024
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXio preprint ar arXio:2399.00267 2023.
Wenqiang Lei, Xiangnan He, Maarten de Rijke, and Tat-Seng Chua. Conversational recom- mendation: Formulation, methods, and evaluation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2428, 2020.
Raymond Li, Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. Towards deep conversational recommendations. Advances in neural information processing systems, 31, 2018.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summariza- fion branches out, pp. 74-81, 2004.
Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig, BRIO: bringing order to abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL-2022), pp. 2890-2903, 2022.
Bill MacCartney and Christopher D Manning. Natural logic for textual inference. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pp. 193- 200, 2007.
Andriy Mnih and Russ R Salakhutdinov. Probabilistic matrix factorization. Advances in neural information processing systems, 20, 2007.
Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly- labeled reviews and fine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, pp. 188-197, 2019.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
Markus Peschl, Arkady Zgonníkov, Frans Oliehoek, and Luciano C Siebert. MORAL: aligning Al with human norms through multi-objective reinforced active learning. In 21st International Conference on Autonomous Agents and Multiagent Systems, 2021.
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulié, Sebastian
Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers. ar Xiv preprint ar arXit:2007.0777 2020.
Martin L Puterman. Markou decision processes: discrete stochastic dynamic programming. John
Wiley & Sons, 2014.
Filip Radlinski, Krisztian Balog, Fernando Diaz, Lucas Dixon, and Ben Wedin. On natural language user profiles for transparent and scrutable recommendation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2863-2874, 2022.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. In 4th International Conference on Learning Representations, ICLR 2016, 2016.
Steffen Rendle, Walid Krichene, Li Zhang, and John R. Anderson. Neural collaborative filtering vs. matrix factorization revisited. In RecSys 2020: Fourteenth ACM Conference on Recommender Systems, Virtual Event, Brazil, September 22-26, 2020, pp. 240-248. ACM, 2020.
Paul Resnick and Hal R Varian. Recommender systems. Communiccatios of the ACM, 40(3): 56-58, 1997.
14

---

Published as a conference paper at COLM 2024
Paul Roit, Johan Ferret, Lior Shani, Roee Aharoni, Geoffrey Cideron, Robert Dadashi, Matthieu Geist, Sertan Girgin, Léonard Hussenot, Orgad Keller, et al. Factually consistent summarization via reinforcement learning with textual entailment feedback. In Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics (ACI-2023), PP- 6252-6272, 2023.
Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. Lamp: When large language models meet personalization, 2024.
Fumihiro Sasaki and Ryota Yamashina. Behavioral cloning from noisy demonstrations. In International Conference on Learning Representations, 2020.
J Ben Schafer, Dan Frankowski, Jon Herlocker, and Shilad Sen. Collaborative filtering recommender systems. In The adaptive web: methods and strategies of web personalization, pp- 291-324. Springer, 2007.
Tianshu Shen, Jiaru Li, Mohamed Reda Bouadjenek, Zheda Mai, and Scott Sanner. Towards understanding and mitigating unintended biases in language model-driven conversa- tional recommendation. Information Processing & Management, 60(1):103139, 2023.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Xiaoyuan Su and Taghi M Khoshgoftaar. A survey of collaborative filtering techniques. Advances in artificial intelligence, 2009, 2009.
Hao Sun. Offline prompt evaluation and optimization with inverse reinforcement learning. Xiv preprint ar 0rXiv:2309.000 366335533 2023.
Yueming Sun and Yi Zhang. Conversational recommender system. In The 41st international acm sigir conference on research & development in information retrieval, pp. 235-244, 2018.
Guy Tennenholtz, Yinlam Chow, Chih-Wei Hsu, Jihwan Jeong, Lior Shani, Azamat Tulep- bergenov, Deepak Ramachandran, Martin Mladenov, and Craig Boutilier. Demystifying embedding spaces using large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https:://ppeerwwieeeee wwwp:/ee/fforruue met//forrmdddxooookkIIz (To appear).
Aaron van den Oord, Sander Dieleman, and Benjamin Schrauwen. Deep content-based music recommendation. In Advances in Neural Information Processing Systems 26 (NIPS-13), pp. 2643-2651 Lake Tahoe, NV, 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances neural informa- fion processing systems, 30, 2017.
Zihao Wang, Chirag Nagpal, Jonathan Berant, Jacob Eisenstein, Alex D'A Sanmi Koyejo, and Victor Veitch. Transforming and combining rewards for aligning large language models, 2024.
Ronald J Williams.
Simple statistical frallint algorithms for connectionist rein- forcement learning. Machine learning, 8:229-256, 1992.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019.
Xue Xia, Pong Eksombatchai, Nikil Pancha, Dhruvil Deven Badani, Po-Wei Wang, Neng Gu, Saurabh Vishwas Joshi, Nazanin Farahpour, Zhiyuan Zhang, and Andrew Zhai. Transact: Transformer-basee realtime user action model for recommendation at pinterest. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining 2023.
(KDD-2023) pp.
15

---

Published as a conference paper at COLM 2024
Ji Yang, Xinyang Yi, Derek Zhiyuan Cheng, Lichan Hong, Yang Li, Simon Xiaoming Wang, Taibai Xu, and Ed Chi. Mixed negative sampling for learning two-tower neural networks in recommendations. In Proceedings of the Web Conference (WWW-20), pp. 441- 447, Taipei, 2020.
Xinyang Yi, Ji Yang, Lichan Hong, Derek Zhiyuan Cheng, Lukasz Heldt, Aditee Kumthekar, Zhe Zhao, Li Wei, and Ed Chi. Sampling-bias boorecede neural modeling for large corpus item recommendations. In Proceedings of the Thirteenth ACM Conference on Recommender Systems (RecSys19), pp. 269-277, Copenhagen, 2019.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference Machine Learning, pp. 11328-11339 PMLR, 2020a.
Yongfeng Zhang, Xu Chen, et al. Explainable recommendation: A survey and new perspec- tives. Foundations and Trends@ in information Retrieval, 14(1):1-101, 2020b.
Kun Zhou, Yuanhang Zhou, Wayne Xin Zhao, Xiaoke Wang, and Ji-Rong Wen. Towards topic-guided conversational recommender system. In Proceedings of the 28th International Conference on Computational Linguistics (COLING-2020), pp. 4128-4139, 2020.
Banghua Zhu, Jiantao Jiao, and Michael I Jordan. Principled reinforcement learning with human feedback from pairwise k-wise comparisons. In International Conference Machine Learning (ICML-2023), volume 202 of Proceedings of Machine Learning Research, pp- 43037-43067, 2023.
16

---

Published as a conference paper at COLM 2024
A Additional Results
Table 3: The accuracies of the RMs on the test set generated with Gemini Ultra Google (0023).
Prec
Ppr
App
Pcov
MovieLenss
0.875
1.0
1.0
1.0
Amazon
0.9686
1.0
1.0
1.0
A.1 Validation of Reward Models
In the development of LLL,, we employed four distinct Reward Models (RMs). The efficacy of these RMs is critical, they need to accurately reflect human values and preferences in the context of endorsement text generation.
The core objective of the evaluation in this part is to determine whether the RMs can reliably distinguish between varying qualities of endorsement texts in terms of factual consistency, appeal, preference coverage, and preference prioritization, respectively. This involves assessing if a specific RM, such as the Prec RM for factual consistency or the Appeal RM for textual appeal, correctly score texts in alignment with their actual quality.
To validate the effectiveness of these RMs, we generated synthetic examples using Gemini Ultra Google (2023). Our approach involved arbitrarily selecting 32 pairs of movies and users. For each pair, Gemini Ultra was prompted to generate two endorsement texts. These texts were designed have distinctly different qualities in terms of specific measure (e-g., one text being more appealing or covering more user preferences than the other, or one being more factually consistent). See the previous page for the prompts we used for the generation of these texts.
The evaluation criteria were straightforward: for each pair texts, we already knew which one was superior based on the targeted measure. The task for the RMs was to correctly classify these texts - identifying the one with higher quality in the context of the specific measure being evaluated. The accuracy of the RMs in correctly classifying the superior text was calculated, providing a clear metric their effectiveness. The outcomes of this evaluation presented in Table 3 for both the MovieLens and the Amazon datasets.
Prec
Ppr
App
Pcov
0.6
0.4
PaLM2-L
SFT-Text
P4LM (ours)
PaLM2-L
SFT-Text
PLLM (ours)
PaLM2-L
SFT-eex
PLLM (ours)
PaLM2-L
SFT-ext
P4LM (ours)
Figure 3: Win rates of model-based scores SFT (MovieLens)
A.2 Model-based Evaluation
We extend our MB evaluation to the Movielens dataset, focusing on the performance of PLLM and other models across the four reward metrics. Table provides these details,
17

---

Published as a conference paper at COLM 2024
Table 4: MB Evaluation Results (Amazon)
Evaluation Metrics
Method
Prec
Ppr
App
Pcov
PaLM2-L
0.34 + 0.04
-1.73 + 0.04
-5.47 + 0.07
-5.48 + 0.06
SFT-Text
0.37 + 0.04
-1.70 0.04
-5.48 - 0.08
-5.42 0.06
SFT
0.37 + 0.04
1.05
- 0.07
- 0.07
P4LM
0.79 + 0.03
-1.47
-5.27 + 0.04
-4.93 0.05
Table 5: MB Evaluation Results (MovieLens)
Evaluation Metrics
Method
Prec
Ppr
App
Pcov
PaLM2-L
0.14 1 0.02
-1.80 + 0.03
-1.43 + 0.04
-2.03 0.03
SFT-Text
0.28 + 0.03
-2.07 0.02
-1.68 i 0.04 0.04
-2.36
SFT
0.29 = 0.02
-201
-1.65 0.03
-2.34
PLLM
0.58 + 0.02
-1.61 + 0.03
-2.16 0.03
Table 6: Single RM Model-based Ablations (Amazon)
Evaluation Metrics
RM
Prec
Ppr
App
Pcov
Prec
0.84 0.03
-1.74
0.05
-5.59 + 0,05
Ppr
0.29 + 0.03
0.03
-4.96
-4.79 0.05
App
0.36 0.03
+.05
-4.95
0.04
-5.01 + 0.06
Pcov
0.47 + 0.02
-1.37
-5.39 0.02
-4.97 + 0.03
Table 7: Single RM Model-based Ablations (MovieLens)
Evaluation Metrics
RM
Prec
Ppr
App
Pcov
Prec
0.87 0.01
-2.21 0.01
-1.95 + 0.03
-2.30 0.04
Ppr
0.41 0.03
-2.08
-1.80 0.04
-2.15 + 0.03
App
0.32 1: 0.02
-.00
0.03
-2.33 0.03
Pcov
0.37 0.02
-2.04 0.02
-1.95 0.02
-1.88 0.03
Prec
Ppr
App
Pcov
0.5
0.2
-0.1
-0,2
PaLM2-L
SFT-Text
P4LM (ours)
PaLM2-L
SFT-exx
PLLM (ours)
PaLM2-L
SFT:Text
P4LM (ours)
PaLM2-L
5FT-Text
PLLM (ours)
Figure 4: The absolute MB score increases vs. SFT (Amazon)
18

---

Published as a conference paper at COLM 2024
Prec
Ppr
App
Pcov
0.4
0.2
-0.2
PaLM2-L
SFT-Text
P4LM (ours)
PaLM2-L
SFT-Text
('LM( (ours)
PaLM2-L
SFT-Text
P'LM (ours)
PaLM2-L
SFT-Text
P'LM (ours)
Figure 5: The absolute MB score increases vs. SFT (MovieLens)
Win Rate: P (ours) SFT-Text
Win Rate: P*LM (ours) vs. SFT-Text
0.8
0.2
0.4
Prec
Ppr
App
Pcov
Prec
Ppr
App
Pcav
Figure 6: Win rate of P*LM over SFT-Text: Amazon (Left) & MovieLens (Right)
Nowe: 1,0 2.0
Neppi 0.25 2.0
Neeayi 2.0
Prec
Prec
Ppr
Ppr
App
Pcov
Pcov
Prec
Ppr
Pcov
0.40
0.45
0.50
0.55
0.600.3
0.5
0.7
0.8
0.40
0.45
0.50
0.35
0.60
Win Rate
Win Rate
Win Rate
Figure 7: Win Rates While Changing the Mixing Weights of Reward Models
previously summarized in the experimental section (Section While PLLM showcases notable precision, it is PaLM2-L that lead in App, Ppr, and Pcov, benefiting from its extensive dataset and larger model size. Interestingly, human evaluations displayed strong preference for LLLM over others, including PaLMM2L, especially in the aspects of precision and preference coverage, suggesting that P'LM's endorsements resonated more with human raters despite the mixed results in MB evaluations.
The absolute increases and win rates offer deeper look into model performances beyond mere score comparisons. While win rates are derived from direct comparisons on individual examples, absolute increases provide averaged perspective across the test dataset. These metrics (see Appendix B for the precise definitions), detailed in Figures and align with the overall trends noted in our results section.
Comparing P*LM to SFT-Text via win rates (Figure 6) reveals that PLLM consistently surpasses SFT-Text across all rewards within both the Amazon and MovieLens datasets. Notably, in Preference Prioritization, L'LM's enhanced performance underscores its superior
19

---

Published as a conference paper at COLM 2024
Table Evaluation of MB metrics in the Amazon dataset via individual RMs. Scores, such as Prec in the first row, result from RL fine-tuning an LM with single RM, then assessing across the four metrics.
Table 9: Human evaluation: Single RM- trained Model vs. SFT output win rates. See Appendix for metric definitions.
Evaluation Metries
RM
Prec
Ppr
App
Pcov
Evaluation Metrics
Prec
0.64
0.55
0.57
6.0%
RM
Prec
Ppr
App
Peov
MLens
Ppr
0.75
0.68
0.75
20%
Prec
0.84
-1.74
-5.65
-5.9
App
0.52
0.60
0.5
1.9%
Ppr
0.29
-0.93
-4.96
-4.79
Pcov
0.81
0.72
0.80
143%
App
0.36
-1.38
-4.95
-5.01
Amazon
Prec
0.64
0.61
0.62
35%
Peov
0.47
1.37
-5.39
-4.97
Ppr
0.82
0.78
0.84
118%
App
0.70
0.52
0.66
0.0%
Pcov 0.64
0.64
0.63
29%
capability for user preference prioritization over SFT-Text, despite SFT-Text's higher win rate relative to SFT. L'LM's high win rates across all the metrics also highlight its proficiency in effectively utilizing RS embeddings, contrast text-based methods, allowing for more nuanced capture and representation of user preferences. This distinction serves to spotlight the potential of embedding-centrr approaches in advancing personalized recommendation generation.
A.3 Ablations
Our ablation studies on the MovieLens domain, detailed in Table examine the model's performance when trained exclusively with individual RMs. This approach mirrors our analyses for the Amazon dataset, as discussed in Section S (Table Training with singular RMs allows us to isolate and understand the impact of each reward on the endorsement quality across different metrics. As alluded to, models trained with focus preference prioritization or preference coverage not only excelled their respective metrics but also showed commendable performance in other areas. This outcome highlights the intricate re- lationships between different aspects of endorsement quality, suggesting that improvements in one dimension can positively influence others.
Figure 7 shows the win rate of policies trained with specific reward mixing weights, com- pared base policy trained with another reward mixing weights. The left figure illustrates the scenario when the Prec reward weight is increased from 1.0 to 2.0. Here, we computed the ratio of instances where the policy with an Prec weight of 2.0 achieved higher RM scores than the one with 1.0 by analyzing 96 pairs of (movie, user) and their corresponding endorsement texts. This win rate was calculated for all four RMs: Prec, Ppr, App, and Pcov, offering nuanced view of how altering a single RM weight influences the overall policy performance across different metrics.
Further exploration into the impact of varying the mixing-weigh. combinations, presented in 7 reveals several noteworthy trends:
1. Increasing the focus on App positively influences Ppr, suggesting a symbiotic relationship between these two metrics.
2. Enhancing certain metrics often results a decrease others, highlighting the inherent trade-offs in our multi-objective optimization problem.
3. Emphasizing Prec slightly increases the Pcov score, though not significantly.
Despite these complexities, our main experiments leverage reward aggregation technique Wang al. (2024) to simplify the optimization process, sidestepping the challenges of manu- ally tuning reward mixing weights. This ablation study, therefore, offers additional insights into the potential interdependencies between different RMs, enriching our understanding of how to best align model training with multiple endorsement criteria.
20

---

Published as a conference paper at COLM 2024
A.4 Example Outputs
In this part, we conduct a comparative analysis of example decoded output texts derived from SFT and P'LM to qualitatively understand their performances and the behaviors learned. Specifically, our focus is on instances where one or more scores attributed to P*LM are significantly higher than those derived from SFT. This approach aimed deter- mining whether the scores effectively capture the correct information from the generated endorsements.
Example 1:
LP*LM achieving much higher Precision score than SFT)
Movie: Up (2009)
Up (2009) is an animated adventure comedy-drama film released by Pixar. The film centers on grumpy widower named Carl Fredricksen, who, having finally had his fill living in modern society, ties thousands balloons his house and sets off to fulfill his childhood dream of exploring the wilds of South America. However, he soon finds that he not as alone on his journey as he had originally thought. Up explores the themes of childhood, growing old, and the importance of following one's dreams.
User Preferences
"hhyy like action movies.
They like movies with lot of special effects.
They like movies with dark and gritty tone.
They like movies with lot of violence.
They like movies with lot of suspense.
They don't like animated movies.
*
They don't 't like romantic comedies.
They don't 't like movies with lot of dialogue.
They don't 't like movies with a slow pace.
They don't like movies with lot of philosophical themes.
SFT
Up is great choice for viewers like you. It is an action-packen adventure comedy- drama with a strong male lead, stunning visuals, and an inspiring story about following one's dreams. It also explores the themes of childhood, growing old, and the importance of family. highly recommend it!
PLLM
Up (2009) is a Pixar animated adventure comedy-drama film that follows grumpy widower who sets off on journey to fulfill his childhood dream of exploring South America. The film is visually stunning and features strong performances from its voice cast. It's also thought-provoking themes of childhood, growing old, and the importance of following one's dreams. Up is must-watch for fans of animation, Pixar, and coming-of-age stories.
and emotionally resonant, exploring the
In this example, P4LM strives to replicate and paraphrase the content from the provided movie synopsis uphold high precision. Conversely, SFT crafts an endorsement while unfortunately incorporating misinformation, as indicated in red.
Example 2: (P+LM achieving much higher Appeal score than SFT)
21

---

Published as a conference paper at COLM 2024
Movie: Sherlock Holmes: A Game of Shadows (2011)
Sherlock Holmes (Robert Downey Jr.) and Dr. Watson (Jude Law) are on the trail of criminal mastermind Professor Moriarty (Jared Harris) in this sequel to the 2009 film "Sherlock Holmes." " As the duo follows series of clues across Europe, they discover that Moriarty is planning terrorist attack that could change the course of history. The movie full of action, suspense, and humor, and features strong performances from its leads.
User Preferences
They enjoy movies with a lot of action and adventure.
They like movies that are visually stunning.
Thhe appreciate movies with a strong story line.
They enjoy movies that are
fantasy worlds.
They like movies that are dark and gritty.
They dislike movies that are slow-paced.
They dislike movies that are too predictable.
* They dislike movies that are too sentimental.
They dislike movies that are too light-hearted.
SFT
"Sherlock Holmes: A Game Shadows" is an action-packed movie full of suspense and humor. If you're looking for movie with strong performances from its leads, then this is the movie for you. It sequel to the 2009 film "Sherlock Holmes" and is set in the past. The movie is not too violent or dark, and it is not too long or slow-paced. It is the perfect movie for viewers who are looking for a light-hearted and entertaining film.
P'LM
Sherlock Holmes: A Game of Shadows visually stunning action-packed thriller that will keep you on the edge of your seat. Robert Downey Jr. and Jude Law deliver strong performances as the legendary detective and his partner, Dr. Watson, as they track down the criminal mastermind Professor Moriarty. The film is full of suspense, humor, and Sherlock Holmes stories.
themes, and it's sure to please fans of the original
In this instance, While the SFT's output is well-rounded and may appeal to those seek- ing light-hearted entertainment, the one generated by P*L.M is more likely to generate excitement and interest among broader audience, including fans of the actors and the Sherlock Holmes series. Specifically, the latter provides specific details about the plot, the characters, and the actors' performances, which make it more engaging and informative. Also, mentioning Robert Downey Jr. and Jude Law leverages their star appeal, which is significant draw for many viewers.
Example 3: (P*LM achieving a much higher Preference Prioritization score than SFT)
22

---

Published as a conference paper at COLM 2024
Movie: Scream (1996)
Scream (1996) follows high school student Sidney Prescott, who becomes the target of a mysterious killer who wears a white mask and robes inspired by the Edvard Munch painting "The Scream". The killer uses horror films as part of deadly game and becomes known Ghostface. The film is notable for its self-referential deconstruction of the slasher film genre.
User Preferences
"They enjoy movies that are well-made and have strong performances.
* "They appreciate movies that are visually appealing and have a strong sense of style. They like movies that are funny and have a light-hearted tone.
They enjoy movies that are romantic and have strong emotional core. They appreciate movies that are sage.
and have strong social mes-
They dislike movies that are poorly made and have weak performances.
"hhe dislike movies that are visually unappealing and have a weak sense of style.
They dislike movies that are unfunny and have a heavy-handed tone.
They dislike movies that are unromantic and have weak emotional core.
* They dislike movies that are bathought-provookin and have a weak social message.
SFT
Scream (1996) is classic slasher film that is sure to keep you on the edge of your seat. The film follows Sidney Prescott, a high school student who becomes the target of a mysterious killer. The film is full of suspense and horror, and it is sure to leave you feeling unsettled. The film also features a strong female protagonist, which you are sure to appreciate.
LL
Scream (1996) is self-referential slasher film that follows high school student Sidney Prescott, who becomes the target of mysterious killer who wears white mask and robes inspired by the Edvard Munch painting "Thh Scream". The film is notable for its dark humor, complex characters, and thought-provoking themes. It's also visually appealing and well-made, with a strong story and unique style. If you're 're looking for horror film that's ' more than just mindless slasher, Scream is great choice.
Here, LM offers a more nuanced and appealing narrative for the given viewer profile. While the SFT output leans heavily into the thriller genre's 's conventional appeal with sus- pense and horror, it makes somewhat narrow assumption about the viewer's 's preferences, focusing the genre's intensity and strong female lead without directly aligning with the viewer's interests specified in the profile. In contrast, P*LM asserts its appeal by highlighting the film's 's dark humor, storytelling style. These elements cater the user's preferences for movies that are not only well-made and visually striking but also carry strong emotional or intellectual impact. This endorsement suggests "Scream" transcends its genre's typical boundaries, offering depth and variety that align more closely with the user's select tastes, from humor and aesthetics thematic substance, thereby making the more compelling recommendation.
themes, and unique visual and
Example 4: (P+LM achieving a much higher Preference Coverage score than SFT)
23

---

Published as a conference paper at COLM 2024
Movie: Ocean's Eleven (2001)
Ocean's Eleven (2001) is a slick, stylish heist movie directed by Steven Soderbergh and starring George Clooney, Brad Pitt, Matt Damon, Don Cheadle, Andy Garcia, and Julia Roberts. The film, a remake of the 1960 Rat Pack film of the same name, follows group of thieves and con artists who plan an elaborate robbery of three Las Vegas casinos. Ocean's Eleven, which oozes cool from its star-studded cast to its jazzy soundtrack, was critical and commercial success, grossing $450 million worldwide.
User Preferences
They enjoy movies that are light-hearted and fun.
They appreciate movies with a strong sense of adventure.
They like movies that are well-paced and exciting.
They enjoy movies with a strong sense of humor.
They appreciate movies with a unique and interesting story.
They dislike movies that are dark and depressing.
*
They dislike movies that are slow-paced and boring.
They dislike movies that are too violent gory.
* They dislike movies that are too sci-fi or fantasy-oriented.
They dislike movies that are too sports-oriented.
SFT
Ocean's Eleven (2001) a remake of the 1960 Rat Pack classic, following a group of thieves and con artists who plan elaborate robbery of three Las Vegas casinos. With an all-star cast, a stylish director, and jazzy soundtrack, it's a slick heist film that oozes cool. If you like dark, gritty, and violent movies, Ocean's Eleven may not be your first choice. However, if you're re looking for stylish, well-made heist film with great cast, you'll probably enjoy it.
PLLM
Ocean's Eleven (2001) is a slick, stylish heist movie that oozes cool from its star- studded cast to its jazzy soundtrack. The film follows group of thieves and con artists who plan an elaborate robbery three Las Vegas casinos. Ocean's Eleven is well-made film with strong story and plenty of action. It's also funny and entertaining, with a unique visual style. you're looking for movie that's visually appealing, well-made, and thought-provoking Ocean's Eleven is great choice.
The endorsement generated by P*LM more effectively caters to the user's preferences by highlighting aspects of the movie that align closely with their interests. For example, explicitly mentions the film being 'funny and entertaining" and possessing "unique visual style", which resonates with the viewer's appreciation for movies that are light-hearted, fun, and have strong sense of humor. Additionally, by describing the movie as "well-made" with "plenty of action", addresses the viewer's preference for films that are well cell-paced and exciting, as well as their interest in unique and interesting stories.
On the other hand, the output from SFT, while emphasizing the stylish and well-made nature of the film, might inadvertently deter the viewer by stating, "if you like dark, gritty, and violent movies, Ocean's Eleven may not your first choice." " This statement could be misconstrued implying the film lacks in excitement, potentially conflicting with the user's aversion to dark and depressing movies. In contrast, P+LM avoids any implication that the film might contain elements the user dislikes and affirms qualities they value, such as humor, visual appeal, and a captivating storyline.
24

---

Published as a conference paper at COLM 2024
B Experimental Details
SOTA Baselines We compared the performance of our recommendation endorsement LMs with the following baselines:
1. PaLM2-L: We prompted PaLM2-L with item descriptions, user preference texts, and instructions to generate a response that suits the four endorsement principles.
2. Supervised Fine-Tuned with Text (SFT-Text): We fine-tuned PaLM2-XS with the aforementioned endorsement dataset but explicitly takes user-item texts as inputs.
3. Supervised Fine-Tuned (SFT): We fine-tuned a PaLM2-XS model that utilizes user- item embedding vectors.
Model-Based Evaluation Metrics We evaluate the methods with held-out unlabeled test dataset Deest = { (I(8), u ( which consists of 96 (MovieLens) or 73 (Amazon) unique user and item pairs. Let 0RM {Prec, App, Ppr, Pcov} denote specific reward model (k),
used for scoring and & be the parameters of the LM. Then, we evaluate FRM (Y(²), ((),, ((k
for each sample in the test set and we report the average score per RM.
To better examine relative performances of the methods, set SFT the common baseline and compare the performance improvements the other models against it. To this end, let
ylt) denote the response sampled by SFT given (I(A), (*)) as an input. Then, we compute Presil which are defined as the win rate, and absolute increase of an LM relative to (YSFT)R=1 k=l follows:
Win rate:
win win_rate (RM)))
Doest
[00RM(Y > ORM (YSF))
Σ k==
DDeetl
where y
denotes the kth textual response sampled by the model +b
Absolute increase
Drsse
(Y r((),,, (k)) PRM (Y()), (()) (K)))
Dtest
Human Evaluation Metrics
Please check Appendix F for more information the metrics and procedures for human evaluation.
B.1 Details of Modeling and Pre-training
In this part, we discuss the details of modeling and pre-training process, focusing on both LLLM and SFT. We specifically elaborate on the integration of user and item behavioral embeddings into a unified latent space interpretable by L.M.
To facilitate the learning of this intricate mapping, we have conceptualized a series of tasks, orthogonal to the primary problem addressed in this study. First, note that to interpret embedding vectors, we require some semantic information about the entities to which they correspond. For instance:
Item embeddings: Consider movie(item) i represented by its text-form descrip- tion, denoted as I(), A supervised learning task designed with the movie(item) embedding i E V as input and (i)) as the target label. This approach enables the construction of varied tasks utilizing elements like critical reviews or movie(item)
summaries to train the LM.
25

---

Published as a conference paper at COLM 2024
User embeddings: A user u is associated with set of rated movies(items), In- In other words, I, =i: A 0. To textually describe a user, an LLM can be provided with the rating history Tu Vi In to encapsulate the user's 's preferences. Given the extensive nature of [Iu], we selectively filter movies(items) and feed them to an
LLM for summarization F
The user's 's rating history is then summarized into user preference profile text output (U)( by the LLM. Consequently, a supervised learning task is developed with the user embedding V as the input and {Uy(w)} as the corresponding target.
Architecture We construct our LM by augmenting a pre-trained transformer architecture T with additional adapter layers designed map continuous behavioral embedding vectors to a common word embedding space. It's crucial to note that we are not training u or i; rather, we focus on optimizing the adapter layers W1 and Wy. This ensures that the nuanced information encapsulated in the CF embeddings in V is effectively translated into the word embedding space, Z. Each of these adapter layers incorporates a 3-layer feed-forward network, interconnected with ReLU non-linearity. The conventional method is employed for mapping text tokens to word embedding space, whereas the adapter layers are utilized to map movie(item) and user embeddings to the latent space.
Training Procedure Our observations indicate that the simultaneous training of newly initialized adapter layers and the transformer parameters does not yield optimal results. This can be intuitively understood as the pretrained embedding layer has an established mapping to the language space, and the freshly initialized adapter layers necessitate extensive updates to achieve comparable mapping. To mitigate this challenge, we employ two-stage training approach (Section3)) Initially, we exclusively train the adapters Wu, W/ with the transformer parameters (I) set frozen, promoting more effective convergence in the subsequent stage. Following this, we proceed to fine-tune the complete model, engaging all the parameters of the LM. As alternative, we leverage parfmeter-efficient training approaches like the one proposed by Hu et al. (2021). This bifurcated training methodology proves pivotal in ensuring the convergence of LM.
C Data Generation
In this part, we discuss the data generation processes of various datasets used for evaluation. (1) We first elaborate on how we construct the dataset 'D used for supervised fine-tuning, as described in Section 3 Specifically, we describe the prompting techniques we used to generate the recommendation endorsement texts Y. (2) In order for an LM to interpret and utilize user and item behavioral information stored in their corresponding embedding vectors (u, i), we insert adapter layers W1 and Wy 6 These adapter layers map either u or to the latent space that the LM can comprehend; that is, the token embedding space. We follow Tennenholtz et al. (2024) in training the adapter layers and the construction of the necessary datasets, which we detail below. (3) Finally, we also expand on how we have generated the Al labels for reward model training.
Generation of the supervised fine-tuning dataset via self-critiquing and revision As- suming we can represent user / viewer with some text description, we can use various prompting techniques to query a very large LM to generate a personalized endorsement of an item to the user. To ensure that the endorsement outputted by the LM abides by the four
*In the MovieLens domain, we can safely assume that an LLM should have obtained vast implicit knowledge about most of the movies in the dataset, which is an unlikely assumption for the Amazon dataset. Hence, we first summarize the information of the items that user has rated, and these descriptions are provided an L.M along with the rating history.
'Note that the user-item behavioral embeddings u, i can be obtained via standard technique such neural collaborative filtering Rendle et al. (2020) simpler matrix factorization Mnih & Salakhutdinov 2007
26

---

Published as a conference paper at COLM 2024
principles set out in this work, we use self-critiquing and self-reoision similar to the approach introduced in Bai et al. (2022).
In this approach, we first use few-shot prompting by querying PaLM2-L with an instruc- tion, an item description 1(8), and user profile text Ug (u(*) to generate a personalized endorsement text.
Instructions: Write short and to-the-point personalized endorsement of movie in a single paragraph. You are given a synopsis the movie and the user preference profile information. Given these, the endorsement text should solely based on the provided information. That is, extra information not explicitly mentioned in the provided synopsis should not appear in the endorsement. Assume the viewer has not seen the movie yet. Hence, do not include any spoilers. A factual and personalized endorsement text is a brief statement that expresses positive approval or recommendation of the film in a way that appeals to specific viewer, based on specific details or elements mentioned in its synopsis.
## Here follows few-shot examples. ##
Answer:
Then, the model generates the first endorsement text following the style presented in the few-shot examples. After that, we sample a critique request and query the model with the request, with the entire texts including the just-generated endorsement being given as the context. Here is the list of critique requests we used in the generation:
Please comment on whether the last response in ways appear to be not based on the given synopsis, not aligned well with the viewer's preferences, or lacking appeal. Keep your response in single paragraph.
Does the last response seem off-base, not aligned with the viewer's preferences, or lacking appeal? Please comment in single paragraph.
Does the last response fail to be based on the synopsis, disregard the viewer's preferences, or lack appeal? Please keep your response in single paragraph.
Would you say that the last response unrelated to the plot synopsis, misaligned with the viewer's preferences, or lacking appeal?
Identify specific ways in which the last response not factually consistent with the provided synopsis. Keep your response in single paragraph.
Discuss ways in which the endorsement deviates from the provided synopsis. Keep your response in single paragraph.
Imagine the only information about the movie is the one in the synopsis. Analyze what additional information is discussed in the last response.
Identify there are any hallucinated information in the last response compared to the given synopsis.
Explain ways in which the last response may not align well with the preferences of the viewer. Keep your response in single paragraph.
Discuss whether the last response appeals to contradicting user preferences. Keep your response in single paragraph.
Identify in which ways the last response may be less appealing to the viewer. Keep your response in single paragraph.
Explain whether the last response tried too hard to meet all the viewer's preferences. Keep your response in single paragraph.
27

---

Published as a conference paper at COLM 2024
Discuss the last response and check whether it sounds sufficiently compelling an endorsement. Keep your response in a single paragraph.
Discuss whether the last response may contain spoilers of the movie. Keep your response in a single paragraph.
Explain whether the last response would make someone to watch the movie. Keep your response in a single paragraph.
Analyze the appeal and compellingness of the last response. Keep your response in a single paragraph.
Our strategy of diversifying critique requests creates dataset containing wide range of responses different styles. When a request made, the model will autonomously assess its previous endorsement generation based on the specified criteria.
Following this evaluation, we instruct the model to revise its response in order to produce endorsement that better aligns with the four criteria. Here are the prompts used:
Please rewrite the response to remove any hallucinated information.
Please rewrite the endorsement such that all information is firmly grounded in the synopsis.
Please revise the endorsement such that it is solely based on the information in the synopsis.
Please rewrite the endorsement to refrain from mentioning information that cannot be inferred from the synopsis.
Please rewrite the endorsement text to better fit the viewer's preferences. Please rewrite the endorsement text such that it is coherent and consistent. Please rewrite the endorsement text to look more personally appealing to the viewer.
Please rewrite the endorsement text so that it appeals to specific parts of the viewer's preferences.
Please rewrite the response to sound more appealing and compelling.
Please rewrite the response to remove any and all spoilers while keeping it sounds appealing.
Please revise the response such that one would want to watch the movie after reading the endorsement.
Revise the response to make it appealing and compelling, while keeping it in the same style.
Please rewrite the endorsement text improve it for better precision r..r.t. the synopsis), better personalization and better appeal.
Rephrase the endorsement text to make it more precise (in regards to the synopsis), personalized, and appealing.
Please edit the endorsement text make it more precise, personalized, and appealing.
Please edit the endorsement text make it more accurate, personalized, and appealing.
Data generation for the two-stage BC procedure as in Tennenholtz et al. (2024) The adapter layers project user and item behavioral embeddings to the token embedding space of an LM, i.e., is mapping from V to Z. In order to train these additional layers, we construct dataset consisting of behavioral embeddings and their corresponding text descriptions. For example, to construct user profile in text, a typical approach in conversational RS is to
28

---

Published as a conference paper at COLM 2024
provide the history of the user to an LLM prompt Kang et al. (2023) Salemi al. (2024). For the MovieLens domain, we selected a few movies per each user based on the ratings. Then, asked PaLM2-L to describe the characteristics of the user in a few bullet points. Here is an example:
Example: user profile generation
Prompt: In a few bullet points, describe the attributes and characteristics of a viewer who likesthe movies: Catch Me You Can (2002), Cellular (2004), National Treasure (2004), DieHard 2 (1990), and The Matrix (1999) but dislikes the movies: Half Past Dead (2002), Predator (1987), In the Valley of Elah (2007), The Legend of Zorro (2005), and Mortal Kombat: Annihilation (1997).
Sample Output: They enjoy movies that are fast-paced and action-packed. * *They prefer movies with strong plot and well-developed characters. * They appreciate movies with a sense of humor. *They are not interested in movies that are too violent gory. They do not enjoy movies that are too slow-paced or boring. 'They prefer movies that are visually appealing. "They appreciate movies with a good soundtrack. "hhy are not interested in movies that too predictable. "hhy enjoy movies that are thought-provoken and challenging.
Next, we employ the two-stage training procedure. In the initial stage, we fine-tune the adapter layers, and subsequently, we fine-tune the entire network. For additional details, we refer the readers to Tonneenhol et (2024)
Generating AI feedback labels In Section 4.1 we explain how establish and evaluate score (i.e., reward) according to four key principles: Precision, Appeal, Preference Cov- erage and Preference Prioritization. For the Appeal, Preference Coverage, and Preference Prioritization, we employ the LAIF methodology as suggested in Lee et al. (2023). Although, in principle, it is feasible use AI feedback for training the Prec RM, we choose to use an existing off-the-shelf NLI model that has been trained on a substantially larger dataset of textual entailment examples. The other RMs, however, are tailored to our specific applica- tion. Therefore, we use another LM to generate and train these RMs in accordance with the loss functions outlined in the main text.
Below is the example prompt for eliciting Al feedback on Preference Prioritization:
[QUESTION]
Options:
(A). [ENDORSEMENT A]
(B) [ENDORSEMENT B]
User's preferences: [USER_PROFILE]
Let's think step-by-step:
Answer: [ANSWER]
In our approach, we select a question from predetermined question pool and repeat this process multiple times for each example. This method is designed to obtain more robust labels from the model. The labels for the two responses are derived directly from line with the findings of Bai et al. (2022), we observed that using hard binary labels tends to lead to overfitting in the developed RMs. To mitigate this, we utilize the average normalized probabilities of the two responses labels Kadavath et al. (0022) This average is calculated across the ensemble of sampled questions.
the supervised fine-tuning dataset, as previously described. Additionally,
Below, we present the lists of questions we ask for Appeal, Preference Coverage and Preference Prioritization RMs, respectively. We show the examples for the MovieLens domain, but the adaptation to the Amazon product review domain is straightforward.
29

---

Published as a conference paper at COLM 2024
Questions for eliciting Preference Prioritization labels from PaLM2-L
Which of the following two responses resonates more deeply with the key elements of the viewer's s preferences?
2. Choose the response that engages more effectively with specific aspects of the viewer's preferences.
3. Select the option that appears delve more deeply and connect more ingfully with certain aspects of the viewer's 's interests.
4. From the two options presented, which one do you think dives deeper into aligning with the viewer's 's specific preferences?
5. Decide which of these two alternatives more effectively hones in on and matches key aspects of the viewer's 's preferences.
6. Identify the option out of these that best delves into and reflects the core interests of the viewer.
7. Pick the response that appears to engage more thoughtfully with the viewer's specific tastes.
8. Among the given choices, select the one you feel offers a more insightful connection with what the viewer strongly prefers.
9. Which option, out of the two provided, seems to engage more deeply with the viewer's personal preferences?
10. Choose the response that you think demonstrates a more focused alignment with the viewer's 's individual preferences.
11. Determine which of these two choices appears to be more engaging and relevant to the specific likes and interests of the viewer.
12. From the two given options, identify the one that seems to offer a more nuanced and focused approach tailored to meet the viewer's 's unique prefer- ences.
30

---

Published as a conference paper at COLM 2024
Questions for eliciting Appeal labels from PaLM2-L
Which of the following two responses has more appeal and reads more compelling?
2. Choose one of the following two options that sounds more compelling and persuasive.
you find more engaging and convincing from 3. Please select the option that you the two provided.
4. Between these two options, which one strikes you as more convincing and engaging?
5. Can you determine which of the two given responses holds greater appeal and more compelling?
6. Of the following two possibilities, which one do you find more persuasive and appealing?
7. From the two options presented, which one do you believe is more com- pelling and persuasive?
8. Considering the two options below, which one appeals to you more as being persuasive and compelling?
9. Please indicate which of these two options you perceive as more engaging and convincing.
10. Please evaluate and choose the more compelling and persuasive option from the two presented.
11. Reflect on the two provided responses and select the one that you find more engaging and persuasive.
12. Analyze these two options and decide which one you believe to be more compelling and persuasive. preferences.
Questions for eliciting Preference Coverage labels from PaLM2-L
1. Between these two responses, which one more comprehensively addresses the diverse preferences of the viewer?
2. Which of these two responses provides broader coverage of the viewer's preferences?
3. Considering the viewer's profile, which of the two movie endorsements more comprehensively addresses their listed preferences?
4. Given the specific preferences in the viewer's profile, which endorsement text among A and B aligns more closely with the majority of these prefer- ences?
5. In relation to the viewer's detailed profile, which of the endorsement texts demonstrates greater coverage of the viewer's movie preferences?
6. Taking into account the viewer's preferences, which endorsement more effectively caters to broader range of their interests in movies? Select between À and B.
D Fine-tuning LMs with Reinforcement Learning
Recall the LM O = N- u) = II, (y 1in- with item text I, item and user CF embedding vectors (i, u) and the reward model i, that measures the quality of factuality, appeal, preference prioritization, and preference coverage of a given
31

---

Published as a conference paper at COLM 2024
recommendation endorsement. Also recall the generation process of LMs can be modeled using the following N-horizon context MDP:
c (I,i,
Sn y0un-11 an Yu,
50 yo, P(S 1 Su, an) 8f5a+1 11 (Sn, an)),
r(sn ani ( [((9==1:)==r((mml (:(::)))((())11)))((0000 L, (0
if n= otherwise , T(nn Snic) = / (yn Yo:n- 1,
where ó: denotes the Dirac distribution z. As result, optimizing RL policy π is equivalent to fine-tuning the underlying LM. The system starts from the start-of-sentence token yo. equipped with user-item context Given the MDP state Sn, the policy takes the action at token-step иf as the next generated token yn. As a result of this action, the system transition deterministically to the state which corresponds the updated token sequence. The reward is zero, except at the final step in which measures the overall quality of the texts at the end of the rugo-regressvve generation process.
A common goal in fine-tuning the LM is to maximize the average over- all quality of the generated text response given the context distribution, ie,, max E(Li.) I-liyo--lliaa ( The gradient of this objective function can be obtained as follows: VolE (
EG (x((() ((swic) - V log (sn (n:))) This is equivalent to applying the popular policy gradient algorithm REINFORCE the aforementioned context MDP for endorsement text generation. The gradient of the objective function is estimated using trajectories I T( (n;))) generated by the current policy, and then used to update the LM policy in an online fashion.
Adding KL regularization: The risk of fine-tuning purely based on the reward model learned from human or AI feedback is that it may overfit to the reward model and degrade the "skill" of the initial LM. To avoid this phenomenon, similar to (Ouyang et al.) 2022 Stiennon et al. 2020), we add the KL between the fine-tuned and pre-trained models the objective function. Leveraging the auto-regressive nature of LMs one can compute the KL regularization over the entire sequence/ pesuencce/rajecctoonn trajectory (of tokens), The resulting objective function is follows:
as a regularizer
ie., KL (L(((yNN-1 i,, u) (ww.pp/ccooooolll
(oauuu-Blggg max Ι(() := (((i.)) Eo r(yo.N i;I, - BBoo Φ i
(3)
It can be shown that this problem is equivalent to the KL-regularized objective in the CoMDP.
Denote by D a replay buffer of trajectories {[I, u, Vo) generated by arbitrary "off- policy" LMs < W, (e.g,, the LM 4 does not necessarily equal to the "on-policy" LM &) over various contexts (L, u). Below we aim to leverage the abundance of offline text-token sequence trajectories for more efficient LM policy learning. Denote by T =
(n, ~ D trajectory sampled from the offline data D, where (sn, An Sp+1) tuple of state, action, and next state of the context MDP, respectively. The addition of KL regularization (Haarnoja et al. [0188] Carta et al.) 2021), which was originally intended to avoid overfitting to the reward model and discounting the "skill" of the initial LM, has also been shown to alleviate the out-of-distrrbuttin action data generalization issues arisen from off-line RL (Kumar et al., 2019). With this KL regularization we can utilize the soft actor critic framework (Haarnoja et al. 2018 to develop RL updates for the value function {vn (s; N-1 state-action value function (Qn a;))) N-1 and LM policy [T T (n)(ani) (initialized with
32

---

Published as a conference paper at COLM 2024
I TBC (sn (nnic)
that minimizes the following losses:
= =
(Man ( (((())nn:))))))))
QN 188 IaN 1:c))²
(4)
(Qtara(Sn, - a log [[nn(nniC) - Vn((nic))
Ly=E,wD
(5)
TBC (0m(snic))
N-1 Qn ani log
c)
L7 B
(6)
TBC (An (nni
where the critic On and Vn take any token sequences at step и input and predict the corresponding cumulative return; & > is the entropy temperature; (Viar, Qtar,n) are the target value networks.
Besides iteratively updating the LM policies and their critic functions, consider the closed- form optimal solution of the Bellman equation of this reguoiyy RL problem:
V (5;c) = a log F ( (exp( Oi (s, )}, Vn,
a
(s, = r(s;c), (s, = E2 [V (s':c)], An < < - 1,
(8)
(als;)) TBC (aBc(als;)) exp( (2,s, E [epp(
Oi (5,
(exp(
}], Vn,
(9)
where the time-dependent optimal policy (at time n),, i.e., 24 is a softmax policy w.r.t. the optimal state-action values Q; over different actions sampled from the pre-trained LM твC- Therefore, value-based approach for RL-based LM fine-tuning would be to first learn the optimal value functions {(*} via the Bellman residual minimization procedure (Antos et al.,
2008) applied to Eq. 7 and Eq. 8 and then solve the following policy distillation (Czarnecki
et al. 2019) problem: π E arg min, EN~D E (L((T-IIs:c))ll (-1sn;c) with respect to
the optimal value }- Notice that this amounts to updating the LM model π via the gradient update
N-1 Σ
<(a
① - ① 7 ETWD
E Ea
V log 72 (Ils;;))
TBC(41 (a];c)
&
(10)
with learning rate y > 0. Further techniques in value-function parameterization have been employed to tackle the overestimation bias. (Fujimoto et al. 2018) proposed maintaining two Q functions, and dual Q function chooses the minimum value between them to avoid overestimation. (Jaques et al. 2019) applies dropout in the Q function to maintain an ensemble of Q values, and outputs the minimum value to avoid overestimation.
E Rater Evaluation
Our test set comprised 96 instances from the MovieLens dataset and 73 instances from the Amazon dataset. For the MovieLens dataset, evaluators were presented with a movie synopsis alongside two endorsements for comparison Their task was to determine which endorsement was more accurate and which was more appealing, as illustrated in Figure 8 For the Amazon dataset, evaluators were shown a description of an item accompanied by two endorsements. In a manner akin to the win rate metric introduced in Appendix B for MB evaluation, we calculate the ratio at which model outputs were preferred over the baseline outputs (i.e., SFT), More concretely,
Diesl 1l ((((K) >
(k) SFT
-==1
win_rate( > &
(11)
D
To minimize bias in the evaluation by raters, we randomized the presentation order of the two endorsements.
33

---

Published as a conference paper at COLM 2024
"MELCA YOUR PRISE SIED WITH snncers MOUI HUMEI DAMES EIT1
HAAL
moussai DAITE SONN (nvit
Quadio
tumge Dimes 210T2, votettyy den fhoe, colora fibo lims or rovo
ahoa onyydeicc
take carcalle
dan rollows Katiaa Cundens.
who
es ber pames order zow atee The lims eep ora trmes ancs pouetty protrp
The sunger dhone 21012 in sanan la monving service sntee prosicn otyin anlive poided enneg land, somnus neer hen expiioe reneee desenaa pouery provding fimm
who
We two andersmenn "carppllng??
Consade bow each ous portays therrode whetber timates move swen
The lange Games 2012 is actoor parcad zo film paay ol crtogia hos viters eisiiic und thai une ancend forinc annpishid segan soe soi the ariila consonaryy on al touar touth kush lg mivea mach eciliig and Boughh plonking
Pleuss actcct witar Aci beln))
Gumas cotiriae
Figure 8: Sample Form for Human Rater Evaluation for Pairwise Comparison
Now, consider vewer who has the following preference attribute:
hhhy enjoy movies with a strang musical elemnnt nescription (pptiona))
Question:
Does Endorsement cover or deal with the viewer's preference? (Please answer Yes or No.)
Yes
Question:
Does Endorsement desl with the viewer's preference? (Please answer Yes or No.)
Question
align preference, aligns closely with the viewer's preference? (Plcase select either or below.)
O
O
B
Figure 9: Sample Form for Human Rater Evaluation for Preference Coverage and Prioritiza- tion
where ((2)
denotes the kth item endorsement sampled by the model c; E c;
(k) (PALMM-L,SST--Tett P*LM), and SFT the corresponding endorsement generated by SFT.
To assess Preference Prioritization and Preference Coverage, we developed specific questions aimed at more accurately quantifying relative preferences. Remember that the user profile
34

---

Published as a conference paper at COLM 2024
text use consists of a bullet-point list detailing various user attributes and preferences. Initially, in our human evaluation process, we presented raters with the complete user profile text, asking them to determine which endorsement better aligned with the user's 's preferences. However, this approach yielded noisy and unreliable responses, and feedback indicated that the raters found it challenging to process the extensive information provided.
To address this issue, we refined our approach by selecting a subset of user attributes from the bullet-point list in each user profile. We then asked raters to evaluate these attributes individually. Specifically, we inquired if each of the endorsement texts addressed the presented user attribute and, so, which one did so more effectively (see Figure 9 For Preference Prioritization, we asked raters to directly compare the outputs from our model with baseline outputs, and we calculated the proportion of instances where our model's outputs were preferred (i.e., win rate). For Preference Coverage, we compared how often the chosen user attributes were addressed in both the model and baseline outputs. We then quantified the improvement of our model's outputs over the baseline as a percentage.
Formally, let (() = (4) represent the user preference texts for the user in the
kth example of the test set, which includes |/| bullet points with J being the corresponding
index set. We randomly select subset of |M| attributes from this list, forming an index set (()), For each attribute m E (R)), we inquire raters if the attribute is addressed by both Q and SFT. Subsequently, we tally the instances where test model outputs and SFT outputs encompass the chosen attributes throughout all examples in the test set. Following this, the Pcov score is calculated as the percentage increase in the frequency of attributes covered by the model compared to those covered by the SFT. That is,
Emc M(H) 11 [Y (K) d. (Am) = 1] EK= EmeM 1 [Y(k)) (Am) = 1]
Pcov((;)
#SST
(12)
ΣK- EMEM?:
II
(K)) dhSFT (Am) -1]
where 1I [Y(]) (Am) = 1] denotes that the kth model output addresses the user attribute Am sampled from )
As illustrated in Figure 9 raters were also tasked with identifying which of the two endorse- ment texts more closely matched the given user attribute. For calculating the Preference Prioritization score, we only considered instances where raters indicated that both en- dorsements addressed an attribute. The preferences expressed by raters were then used to determine the win rate test model compared to SFT, employing the same metric outlined in equation 11
F Prompts
We put the prompts that used when evaluating the RMs discussed in Appendix A.1
35

---

Published as a conference paper at COLM 2024
Prompt to Gemini Ultra for Evaluating the Precision (Prec) RM
Craft two endorsements of the movie [TITLE] whose plot is provided below:
[PLOT]
he endorsement should be factually consistent, relying solely on the information from the plot. Avoid copying the plot verbatim. The endorsement should be single paragraph. Do NOT use any information that is not explicit in the plot, even if you know other fact about the movie.
Endorsement: [GOOD ENDORSEMENT]
Now create another endorsement for the same movie. This new endorsement should be less factually consistent with the plot. That is, this endorsement should have lower quality than the previous in terms of precision. Also, feel free to hallucinate, or to add some untrue fact in the endorsement w.r.t. the plot. It should be easy to tell this endorsement is worse than the previous. The length of this endorsement should be similar to the length of the previous one. Non-Precise Endorsement: [BAD ENDORSEMENT]
Prompt to Gemini Ultra for Evaluating the Preference Prioritization (Ppr) RM
Craft a personalized endorsement of the movie [TITLE] whose plot is provided
below:
[PLOT]
The endorsement should be personalized to a user with the following preferences:
The endorsement should be factually consistent, relying solely on the information from the plot. Avoid copying the plot verbatim. The endorsement should be single paragraph. Do NOT use any information that not explicit in the plot, even if you know other fact about the movie.
Tailor the endorsement to align with the user's preference profile, highlighting aspects of the movie that would particularly appeal to them. Ensure each appeal is directly tied to the plot elements.
Endorsement: [GOOD]ENTE ENDORSEMENT]
Now create another endorsement for the same movie. This endorsement should be less personalized the user's profile. It should easy tell this endorsement is worse than the previous in terms of personalization. Less-Personalized Endorsement: BBAD_ENDORSEMENT
Prompt to Gemini Ultra for Evaluating the Appeal (App) RM
Craft an appealing and compelling endorsement of the movie [TITLE] whose plot is provided below:
[PLOT]
The endorsement should be factually consistent, relying solely on the information from the plot. Avoid copying the plot verbatim. The endorsement should be single paragraph. Do NOT use any information that not explicit in the plot, even if you know other fact about the movie.
Ensure each appeal directly tied the plot elements. Make the endorsement as appealing and compelling as possible.
Appealing Endorsement: [GOOD ENDORSEMENT]
Now create another endorsement for the same movie. This endorsement should be less appealing and compelling than the first one. It should be easy to tell this en- dorsement is worse than the previous in terms of appealingness and compellingness. Less-Appealing Endorsement: [BAD_EN BAADEENORSEMENTI
36

---

Published as a conference paper at COLM 2024
Prompt to Gemini Ultra for Evaluating the Preference Coverage (Pcov) RM
Craft a personalized endorsement of the movie [TITLE] whose plot is provided
below:
[PLOT]
The endorsement should be personalized to user with the following preferences:
The endorsement should be factually consistent, relying solely on the information from the plot. Avoid copying the plot verbatim. The endorsement should be single paragraph. Do NOT use any information that not explicit the plot, even if you know other fact about the movie.
Tailor the endorsement to align with the user's preference profile, highlighting aspects of the movie that would particularly appeal to them. Ensure each appeal is directly tied to the plot elements.
Importantly, make sure the endorsement covers both positive and negative prefer- ences w.r.t. the plot. For example, if the user does not like action movies, then the endorsement should cover the fact that the movie is an action movie. Preference Covering Endorsement: Now create another endorsement for the same movie. This endorsement should not cover all of the user's s preferences like the first one. It should be easy to tell this endorsement is worse than the previous terms of preference coverage.. Less-Covering Endorsement: [BAD ENDORSEMENT]
37


Reviews:
[{"review_id": "83tQ0hsBEJ", "reviewer": "Reviewer_VTxA", "summary": "The paper introduces innovative reward models designed to quantify key elements crucial for effective personalization. By employing Reinforcement Learning with Augmented Intrinsic Rewards (RLAIF) and these novel reward structures, The Language Model (LM) generates item endorsements that are not only factual and appealing but also comprehensive in addressing user preferences. Furthermore, the prioritization mechanism enhances the connection between relevant item attributes and user preferences, boosting the likelihood of users accepting high-value recommendations.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. Please find brief responses below.\n\n1. __many articles on combining large models with user information … should be compared with one or two traditional models.__\n\nIndeed, there is recent work that combines user information with large language models (e.g., [1, 2, 3], listed below). __However, these papers focus on the recommendation task itself, whereas our work addresses the novel task of generating endorsement text for recommended items that satisfy the criteria of precision, appeal, preference coverage, and prioritization.__ As such, these methods do not admit a direct comparison with ours. Traditional recommenders are also unsuitable for our task as they do not generate natural language responses.\n\n[1] CoLLM: Integrating Collaborative Embeddings into Large Language Models for Recommendation (Zhang et al., 2023)\n\n[2] Representation Learning with Large Language Models for Recommendation (Ren et al., 2024)\n\n[3] Collaborative Large Language Model for Recommender Systems (Zhu et al., 2024)\n\n2. __The abscissa in Figure 2 is not shown...__\n\nWe appreciate the feedback on Figure 2 and will revise it to include a clear abscissa and ordinate, as well as combine the four indicators on a common coordinate axis to enhance clarity.\n\n3. __The main experiments of the article should be reflected in the main text, such as ablation experiments, etc., and should not be found in the appendix…__\n\nWe acknowledge the importance of having main experiments in the primary text. Unfortunately, due to the strict 9-page limit excluding references, we were forced to place some details in the appendix. If accepted, in the camera-ready version, which allows an additional page, we will incorporate key results from the appendix into the main text. \n\nWe discuss ablation studies on page 8 in the last paragraph and refer readers to the corresponding figures and tables in the appendix. We would definitely welcome any suggestions you may have on specific results that should ideally be included in the main text.\n\n4. __...not mentioned at all in the text, such as Figure 2__\n\nWe reference Figure 2 on page 8, second line. We will ensure that all figures and tables are explicitly mentioned in the text to improve readability.\n\n5. __too many references in the appendix; no anonymous code link__\n\nWe believe these references provide comprehensive context and support for our work. We plan to release the code once the blind review period is over.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Use the collaborative information of users and products to join the large model and fine-tune it, and use a two-stage training method to enable better integration.\n2. Four aspects of criteria are used to measure the quality of the results, and an RLAIF technique is designed to fine-tune the model. \n3. The model and formula derivation part is quite clear, and the specific details are included in the appendix.", "reasons_to_reject": "1. In the experimental part, apart from SFT fine-tuning, only compare PaLM2. There are already many articles on combining large models with user information. I think it is very important to compare with them. In addition, the model should be compared with one or two traditional models.\n2. The abscissa in Figure 2 is not shown and the ordinate on the right side is not shown. The four indicators should be put into a common coordinate axis in one picture to display, which will make it clearer.\n3. The main experiments of the article should be reflected in the main text, such as ablation experiments, etc., and should not be found in the appendix. It gives people a feeling that the things in the main text are not as important as in the appendices.\n4. Some figures and tables are not mentioned at all in the text, such as Figure 2. Moreover, there are too many references in the appendix of the article, and no anonymous code link is published.", "questions_to_authors": "See the weakness above.", "extra_scores": {}}, {"review_id": "cRpSq4EH90", "reviewer": "Reviewer_wcse", "summary": "# Summary\n\nThe paper focuses on generating good natural language endorsements for recommendations provided to a user. They introduce four criteria: 1) factuality (referred to as precision) 2) appeal 3) coverage of user's preferences and 4) prioritization of aspects that matter most to the user. They quantify these criteria, learn a reward model for each, and then use RLAIF to align LM with the criteria.\n\nA separate contribution of the work is to learn projections from CF embeddings of users and items to a common space with the token embeddings, so the CF embeddings can be used directly in the model instead of having to generate natural language profiles of users.\n\nThey compare their model with 1) a bigger LLM prompted with text only to generate a response adhering to the four criteria 2) a text-only smaller SFT model and 3) a smaller SFT model fine-tuned with embeddings. Their results show that their RLAIF trained model does better than the other models evaluated.\n\n# Overall evaluation\n\n* The paper is written clearly and is easy to follow. The data collection and experimental evaluation makes sense. \n* Decomposition of the objective into the four criteria makes intuitive sense, but I would have liked to see more evaluation, more details below.\n* The part about injecting CF embeddings directly into the model seems like a tangent from the main point of the paper, more below.\n* I do have an issue with the use of the word precision for the criteria of factuality. Precision has a very specific meaning as a metric in the IR community, it'll be good to use a different word here.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive feedback. We appreciate your insights and have addressed each point as follows:\n\n1. __CF embeddings...__\n\nOur main contribution is identifying the four criteria for recommendation endorsements and using RLAIF to achieve them. Defining optimal textual user profiles is beyond the scope of this work. In this regard, mapping from a CF embedding space to a word-token embedding space allows us to use diverse sources of user data, reducing reliance on specific prompts. The output from adapter $W_{\\mathcal{U}}$ compactly represents collaborative-filtering-based and other user details, minimizing dependence on specific profile formatting.\n\n2. __text-based RLAIF baseline__\n\nWe agree that text-based RLAIF is an important baseline. However, we aimed to avoid relying on specific user profile formats. Preliminary trials showed this baseline struggled to maximize the rewards, possibly due to increased context lengths. We are re-evaluating this baseline and will include the results in the final version.\n\n3. __the word precision__\n\nWe understand the concern, and we will adopt a different term in our revised paper.\n\n4. __conflict between criteria 3 and 4__\n\nAlthough we acknowledge a potential conflict between these criteria, these scores are positively correlated in practice. Tables 6 and 7 show that training with only the Ppr or Pcov RM resulted in high scores for both. This is because Pcov RM training does not exclude prioritization. We will expand on this in the final version.\n\n5. __human vs. model-based (MB) evaluation__\n\nAs noted on page 8, MB and human evaluations align well on Amazon, but differ on MovieLens. Raters preferred P4LM much more w.r.t. Pcov, Ppr, and App scores than MB evaluations suggest. This likely arises because these RMs were trained with PaLM2-L data. Using a different LLM for data generation might help, but won't fully resolve the issue. We will evaluate RM alignment with raters' responses, comparing RM and rater preferences for each endorsement text.\n\n6. __overall improvement by P4LM__\n\nWe do have overall quality comparison from the raters (ratios of preferred responses over SFT's):\n| Amazon | Ratio |\n|-|-|\n| PaLM2-L     | 0.466 |\n| SFT-Text    | 0.438 |\n| Ours | 0.808 |\n\n| MovieLens | Ratio |\n|-|-|\n| PaLM2-L | 0.333  |\n| SFT-Text | 0.490 |\n| Ours | 0.708 |\n\nThese results show significant overall quality improvement with P4LM as evaluated by human raters. We will include these in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is clearly written and is good technically. The choices made for the different reward models make sense. \n* Decomposition of the reward into easily understandable criteria makes it easier to train and debug when a model is producing unsatisfactory results.\n* Human evaluation is performed in addition to the model based evaluation, which increases my trust in the claims.\n* The experimental plan, though restricted to a single family of LLMs, is mostly appropriate for the task.", "reasons_to_reject": "* The whole section about user profile prompting is tangential to the main thrust of the paper. If it is included, it needs to be evaluated as well. E.g. an RLAIF trained PALM2-XS model that uses text based user representation instead of embedding.\n* There is some conflict between criteria 3 and 4, e.g. if we prioritize focus on some aspects (criteria 4), this might mean leaving some aspects out, which would reduce coverage of user's preferences (criteria 3). Could the authors' talk a bit more about this?\n* In the experimental section, the human evaluation and reward based evaluation have some discrepancies (attributed to approximation errors in the reward models). This needs to be discussed more, as noise in the reward models impacts the whole RLAIF methodology.\n* Finally, it's not clear to me if the division into the four criteria actually helps increases the overall quality of the endorsements. It would've been good to have human evaluation for the overall endorsements and establish that we see improvement in this score as well.", "questions_to_authors": "* Consider using a different term for factuality than precision.", "extra_scores": {}}, {"review_id": "Oad1Z5RyAL", "reviewer": "Reviewer_kDyJ", "summary": "In this paper, the authors develop a novel P4LM model to endorses recommended items by emphasizing specific item characteristics and their coverage to a user’s preferences. It takes user & item embeddings as inputs to generate responses that are appealing, factually-grounded and tailored to the user’s preferences. P4LM employs a joint reward function as AI-based feedback for reinforcement learning. Through extensive experiments on offline datasets, the authors demonstrate that P4LM delivers more appealing and tailored endorsements to users, as determined by auto-critic and rater evaluations.\n\nOverall, the paper is very nicely written, the research idea is novel and technically sound, and I especially commend the authors for conducting a comprehensive set of ablation studies to demonstrate the generalizability of their proposed method. I will definitely recommend acceptance of this paper. Meanwhile, I also offer a few suggestions below, which I hope the authors will find them useful:\n\n(1)\tI wonder what is the exact role of LM in the designed method Since the authors directly inject the embeddings into LM (a seq2seq LM in particular) and generates the response directly, it seems that LM is equivalent to a hidden interaction layer between user & item in this case.\n\n(2)\tI also wonder the improvements of the endorsement quality (demonstrated through a couple of evaluation metrics in this paper) will or will not eventually lead to the improvements of business metrics related to recommendations (CTR for example).", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the value of our work and for your constructive feedback. We appreciate your insightful comments and suggestions.\n\n1. __what is the exact role of LM in the designed method__\n\nThanks for the interesting comment. Although we directly inject the embeddings into the LM, without using a pretrained LLM, we cannot guarantee the language quality of responses generated by the trained model. The generalization capability of pretrained LLMs helps improve response quality even for specific downstream domains, as observed in prior work. \n\nIn domains with large, diverse, high-quality domain-specific textual datasets, training an LLM from scratch followed by RLAIF training may be beneficial. However, using a pretrained (or instruction-tuned) LLM will likely significantly enhance language quality. The tradeoff may include potential hallucinations, as information from the pre-training dataset can be ingrained in the model, causing it to include information not supported in the given context in responses. In this case, P4LM's training framework, which ensures factually grounded responses through the Precision RM, can mitigate the impact of hallucinations.\n\n2. __the improvements of the endorsement quality will or will not eventually lead to the improvements of business metrics__\n\nThis is indeed a very interesting question. We plan to explore this as future work, where P4LM can be integrated into a deployed ConvRec system to generate endorsement text for the items selected by the recommendation engine. For the scope of this work, we focus on the endorsement task itself, isolating this problem to better evaluate endorsement quality through extensive model-based and human evaluations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Please refer to the summary section.", "reasons_to_reject": "Please refer to the summary section.", "questions_to_authors": "Please refer to the summary section.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper develops a novel method for training LLMs to present natural language item endorsements that explain why a recommender system has selected a particular item. If done well (in particular, if properly grounded and not just confabulated), this could help enhance users' agency. The reviewers agree that the technique is well-motivated and well-implemented, and the results demonstrate that the method developed by the authors is effective as compared with some more simplistic alternatives. The reviewers raised some helpful suggestions, to which the authors responded effectively.
Decision: Accept