Multi-Objective Intrinsic Reward Learning for Conversational Recommender Systems
Authors: Zhendong Chu
Nan Wang
Hongning Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Conversational Recommendation
Reinforcement Learning
Meta Learning
TL;DR: We propose a multi-objective bi-level optimization algorithm to learn intrinsic rewards for conversational recommender systems.

Abstract: Conversational Recommender Systems (CRS) actively elicit user preferences to generate adaptive recommendations. Mainstream reinforcement learning-based CRS solutions heavily rely on handcrafted reward functions, which may not be aligned with user intent in CRS tasks. Therefore, the design of task-specific rewards is critical to facilitate CRS policy learning, which remains largely under-explored in the literature. In this work, we propose a novel approach to address this challenge by learning intrinsic rewards from interactions with users. Specifically, we formulate intrinsic reward learning as a multi-objective bi-level optimization problem. The inner level optimizes the CRS policy augmented by the learned intrinsic rewards, while the outer level drives the intrinsic rewards to optimize two CRS-specific objectives: maximizing the success rate and minimizing the number of turns to reach a successful recommendation}in conversations. To evaluate the effectiveness of our approach, we conduct extensive experiments on three public CRS benchmarks. The results show that our algorithm significantly improves CRS performance by exploiting informative learned intrinsic rewards.

Multi-Objective Intrinsic Reward Learning for Conversational Recommender Systems
Zhendong Chu University of Virginia zc9uyvvirginia. edu Charlottesville, VA, USA
Nan Wang' Netflix Inc. nanw@netflix com Los Gatos, CA, USA
Hongning Wang University of Virginia hw5x@virginia, edu Charlottesville, VA, USA
Abstract
Conversational Recommender Systems (CRS) actively elicit user preferences generate adaptive recommendations, Mainstream reinforcement learning-base -based CRS solutions heavily rely on handcrafted reward functions, which may not be aligned with user intent in CRS tasks. Therefore, the design of task-specific rewards is critical to facilitate CRS policy learning, which remains largely under- explored in the literature. In this work, we propose novel approach to address this challenge by learning intrinsic rewards from interactions with users. Specifically, we formulate intrinsic reward learning as multi-objective bi-level optimization problem. The inner level optimizes the CRS policy augmented by the learned intrinsic rewards, while the outer level drives the intrinsic rewards to optimize two CRS- CRS-specific objectives: maximizing the success rate and minimizing the number of turns to reach successful reconmendation in conversations. To evaluate the effectiveness of our approach, we conduct extensive experiments on three public CRS benchmarks. The results show that our algorithm significantly improves CRS performance by exploiting informative learned intrinsic rewards.
Introduction
Conversational recommender systems (CRS) leverage interactive conversations to delineate a user's preferences [Zhang et al. 2018 Lei al. 2020a Chu et al. 2023 The conversations revolve around questions aimed at discerning users' preferences on specific item attributes (e.g., music genres). Through interactive process of questions and answers, a profile about user's intended item can be depicted. Numerous CRS formulations have been proposed [Chen et al., 2019 Christakopoulou al. 2018 2016). In this work, investigate prevalent CRS setting known the multi-round conversational recommendation [Lei et al., 2020a Deng et al., 2021]. where a CRS agent can ask question or recommend an item in consecutive rounds of conversations. The conversation continues until the user accepts the recommendation (indicating a successful conversation) or quits the conversation (considered
failed conversation).
CRS fundamentally embodies a sequential decision making problem. for which numerous reinforce- ment learning (RLe- solutions have been proposed (Lei et al. 2020a Chu al. 2023 However, as the users only provide textual or binary responses (e.g., accepting or rejecting the inquired at- tributes), existing RL-based solutions heavily rely heuristic reward functions that manually defined to train CRS policies. These reward functions, such as promoting attributes accepted by a user and penalizing those rejected, may not accurately reflect user intent due their heuristic nature. This becomes problematic since the effectiveness of CRS policy learning largely depends on the quality of pre-defined reward function an inadequately designed reward function can lead solutions that
Work was done while at the University of Virginia.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

significant deviates from optimality. Additionally, these arbitrary reward functions can inadvertently distort the modeling of conversation states, influencing the subsequent actions taken by the RL agent.
Arguably, an effective reward function should promote actions that lead to more precise modeling of users' preferences. As result, different attributes items, including those rejected, can each uniquely contribute to user preference modeling. As an example illustrated in Figure [ even though Heavy metal rock rejected by the user, still, to certain extent, contributes identifying the target item, Hey Jude, However, existing handcrafted heuristic reward functions fall short in delivering information at this granularity, as they assign uniform rewards to all accepted or rejected actions. This motivates us to learn reward function that enables more fine- thee-graind policy learning.
Instead of manually define reward func- tions, we introduce a principled approach to reward learning for CRS, where we learn a intrinsic reward for each action taken by the agent utilizing the optimal rewards framework (Singh et al. 2010). This framework delineates the optimal intrinsic reward function as the one that, when employed by an RL agent, fosters behaviors that optimize the task-specific or extrinsic rewards- the case of CRS, successful recommendations.
Do you Pop music?
Accepted, but too general Preference: (Pop)
Yes
Do you like Britpoe
Acoepted, helpful
Preference: (Pop, Srrppop
Yes!
Do you like Bock music-
Accepted, and helpful Preference: (Pop, Britpop, Rock]
Yes!
Do you like Heavy metel rock?
Rejected, but helpful Preference: (Rock, Brilpop, Rock, -Hoavy metol Rock]
No!
Actions should encouroged
suggest Hey Jude
Correct recommandation
Two notable technical challenges stand out when learning intrinsic rewards for
like
CRS. First, explicit extrinsic rewards in CRS are extremely sparse, which com- plicates the intrinsic reward learning. Despite that the agent interacts with the user in each round, the only clear extrinsic reward signal, which is whether the overall conversation is successful or not, is only revealed from the user the conclusion of the conversation. The significance of each accepted or rejected attribute/item prior to the conversation's 's ends remain ambiguous. For instance, an inquired attribute that is rejected by the user does not necessarily imply a negative reward for policy learning, as it can signify what the user is not looking for. Second, the assessment of CRS multi-dimensional, entailing various factors that contribute to the overall effectiveness and user experience, such recommendation quality and user effort. On the one hand, asking more questions may be necessary to accurately profile user preferences to facilitate a successful recommendation. On the other hand, reducing user effort in conversations (i.e., fewer conversation turns) is essential to ensure users' engagement and maintain their satisfaction. Balancing these factors is a delicate task.
Figure
Motivating example of intrinsic reward learning.
To tackle the challenges for improving CRS from a reward learning perspective, we develop an online algorithm for learning intrinsic reward functions via multi-objective bi-level optimization. We name the proposed solution CRSIRI, meaning CRS with Intrinsic Reward Learning. In the inner loop of CRSIRL, the policy is optimized with the learned intrinsic reward function. In the outer loop, the intrinsic reward function is updated to satisfy two specific criteria designed for CRS. The first criterion aims to maximize the sparse extrinsic reward, augmented by reward shaping strategy to encourage actions that promote the target item as quickly as possible. The second criterion involves tailoring the learnt reward function to promote successful trajectories over the failed ones. The results of our extensive experiments demonstrate that CRSIRL not only improves the success rate of CRS but also achieves it with shorter conversations.
2 Related Work
Conversational Recommder Systems. Christakopoulou et al. [2016] pioneered the concept of Conversational Recommender Systems (CRS). Their approach primarily focused on determining which items to solicit feedback on and applied off- the-hhh- metrics such as the upper confidence bound [Auer 2002] for this purpose. This laid the groundwork for reinforcement learning (RL) based methods, which have recently become the prevalent solutions for CRS. For example, [unn and Zhang [20T8) developed a policy network to decide whether to recommend an item or inquire about an item attribute at each conversation turn. However, these initial studies terminated the conversation upon

---

making a recommendation, regardless of user acceptance. Lei et al. conversational recommendation, where CRS can ask question or recommend an item in multiple rounds before the user accepts the recommendation or quits. This is also the setting of our work in this paper. To better address multi-round CRS. Lei et al. [020b] leveraged knowledge graphs to select more relevant attributes to ask across turns. Xu et al. [202T] extended [Lei et al.) [020aa by revising user embeddings dynamically based on users' feedback on attributes and items. And Deng et al. [2021] unified the question selection module and the recommendation module in an RL-based CRS solution. However, all the aforementioned works depend on heuristically crafted reward functions, which may lead policies deviate from the optimal solution. In this work, propose to learn intrinsic rewards which can maximize the recommendation performance.
[220al] studied multi-
Intrinsic Reward Learning in Reinforcement Learning, Intrinsic reward learning has emerged as a promising approach to enhance the performance and efficiency of reinforcement learning algorithms. Singh et al. [20TO] introduced the Optimal Reward Framework which aims to find good reward function that allows agents to solve a distribution of tasks using exhaustive search. PPahhak al. 2017] introduced the concept of curiosity-driven intrinsic rewards, where the agent is rewarded for actions that lead to novel states, improving its ability to explore complex environments. ZZeng et al. 2018
proposed meta-gradient method named LIRPG to learn intrinsic rewards via bi-level optimization framework. Zheng et al. [020] extended LIRPG by learning intrinsic rewards on a distribution of tasks. Liu et al. [222] developed another meta- gradient method to learn intrinsic rewards from trajectory preferences. In this work, we propose a novel intrinsic reward learning framework designed for CRS, where we learn intrinsic reward functions to satisfy multiple CRS- CRS-sepcific objectives from users' extremely sparse explicit reward feedback.
3 Preliminaries
In this section, we define the notations to be used in our technical discussions and some basic notions in multi-objective optimization.
3.1 Problem Definition
Similar to traditional recommender systems, CRS serves a set of users U with a set of items V; and we denote specific user as and item as Each item is associated with set of pre- defined attributes Pe. Attributes describe basic properties of the items, such as genres in movie recommendations and cuisine type in restaurant recommendations.
We formulate the CRS problem using a Markov decision process (MDP) [eng et al. 2021 Lei et al. 2020b] Chu and Wang 2023), which can be fully described by tuple (S,A, T, R). S denotes the state space, which summarizes the conversation between the system and user so far. denotes the action space for the system, which includes recommending particular item or asking for feedback on a specific attribute. T : S x A + S is the state transition function, and R : x A R is a reward function.
With this formulation, conversation in CRS can be represented as {(a1, r1), ... (ar, pT)]. where T is the maximum number of allowed turns. A conversation (or an episode in the language of RL, which we will use exchangeablely) terminates when: (1) the user accepts the recommended item; or (2) the CRS agent runs out of maximum number of allowed turns. At each time step t, the CRS agent, which follows a policy Te (o(ur(s)) parameterized by 0, selects an action d, based on the current state 8t. The training objective of CRS policy is to maximize the expected cumulative rewards over the
set of observed episodes D, i.e., minimizing the loss
[(T)= = E
R
d~P(D)
where R₁ = (((((()) is the accumulated reward from turn f to the final turn T, and 7 [0, 1] a discount factor to emphasize rewards collected in near term.
Instead of using handcrafted reward functions R as in previous works Lei et 2020a Deng et al. 2021 Chu et al. 2.23), we learn an intrinsic reward function defined as r (s, parameterized by enhance CRS policy learning. In this context, the original CRS-specific reward is referred to
3

---

Outer Loop
Pair-wise
Objective
Suc. Buffer
0L((0)
00
rap
L(0) (0')
00 0eb
Falled Buffer
Point-wise
Objective
Inner Loop
HPs
y
8, a 8
a.,
0'
Policy
T (a|s)
1 C.
Te (al
+ An
Update
Figure 2: Overview of URSIRL,,hich consists of two modules, policy parameterized by and an intrinsic reward function parameterized by Φ. The optimization of CRSIRL has two levels. In the inner level, a policy is trained to maximize the return defined by both intrinsic and extrinsic rewards. In the outer level, the intrinsic reward function trained to optimize two CRS-specific objectives realized by the learnt policy's behaviors.
as the extrinsic reward, denoted as r (s, a). The extrinsic reward is inherently sparse, the only discernible and useful reward signal is the success or failure of an episode, with the intermediate actions' contributions remaining ambiguous. We assign positive extrinsic reward at the conclusion of a successful episode and a negative reward otherwise. All intermediate actions are assigned zero extrinsic reward.
3.2 Multi-Objective Optimization
We utilize multi-objective optimization (MOO) to achieve the multi-dimensional goal of CRS, i.e., maximizing the success rate and reducing the length of conversations. MOO aims to simultancously optimize multiple objectives, possibly conflicting ones. This results in a trade-off among objectives, making the CRS problem more complex and challenging to solve. In these cases, the Pareto-optimal solutions represent different optimal trade-offs between the objectives (Deb and Deb 2013
Consider M objective functions (L), .....y L*),, model parameterized by 0 is optimized towards them. We specify the multi-objective optimization using vector valued loss L.,
min (0) min (0, T
The goal of multi-objective optimization achieving Pareto optimality. Definition (Pareto optimality).
(a) A solution f dominates solution 0 if (0) S (0) under all objectives and L(O)) + L(0).
(b) A solution 0* is called Pareto optimal if there exists solution 0 that dominates 0"
The set of Pareto optimal solutions is called the Pareto front Pa.
Multi-Objective Intrinsic Reward Learning for CRS
In this section, elaborate the structure of CRSIRL, illustrated in Figure CRSIRL operates on two tiers of optimization: the inner optimization, which improves the policy using both the extrinsic reward and the learned intrinsic reward function, and the outer optimization, which refines the intrinsic reward function based on the policy assessment derived from the inner optimization. Given the absence of supervision for the intrinsic reward function, between it and the refined policy via gradient descent in the inner optimization. Specifically, we compute meta- graddeerriet for the intrinsic reward function using chain rule in the outer optimization. In the outer optimization, we design point-wise objective striving to enhance extrinsic rewards in the learnt intrinsic reward function, through use of hindsight reward shaping (HRS). This objective aids in pinpointing pivotal actions that significantly improve the target item's 's ranking, thereby shortening the conversation length. parallel, we introduce a pair-wise objective that favors successful trajectories over unsuccessful ones, which assists in identifying actions that result in
establish the relationship

---

preferred conversations. This objective is named as recommendation preference matching (RPM). Finally, we introduce a holistic multi-objective bi-level optimization framework that optimizes intrinsic rewards to meet both objectives.
4.1 Hindsight Reward Shaping
As we discussed before, the extrinsic reward is extremely sparse in CRS. The only clear and informative signal from the extrinsic reward is whether the conversation is successful, making it hard judge the progress of user preference elicitation during the conversation. Reward shaping. as proposed by Ng et al. [199)], serves as valuable tool for incorporating task-specific knowledge estimate the reward function. We leverage reward shaping within the outer loop of our model to imbue the process of intrinsic reward learning with more nuanced, task- guidance. We use the following hindsight reward shaping to augment the extrinsic reward,
(s 0e) r* ($1, + (u(s141, v) w(s(, v),
where w is a scoring function, v is the target item and is a discount factor. f az) encourages actions which promote the target item. In turn, it helps shorten the conversation length. In our experiments, we use w = log(p(s;, v) + 1), where p(5e, v) is the rank of target item v under state st.
Lemma 1. Consider any reward shaping function : AxS - R. we say F potential-basee reward shaping function (PBRS) if there exists real-valued functiooo T S - R satisfying,
a, s') = >b((s)) +(s).
(4)
Then F being PBRS is a necessary and sufficient condition for it to guarantee the consistency of the optimal policy, i..,, the optimal policy of (S. A, T, R+FF is the same (S, A. T, R.).
The proof is based on Ng ct al. and 3..
1999 and omitted in this paper. By matching the form of Eq. conclude the hindsight reward shaping satisfies the PBRS condition, and thus the optimal policy is consistent. The resulted objective induced by HRS is
T
_ (0) -E E
(5)
where R -i= TTx' pen. Note that the information of target item is unknown beforehand, we can only use HRS after the target item is hit, which is why we call it hindsight. Otherwise HRS degenerates to the original extrinsic reward.
4.2 Recommendation Preference Matching
Even though the contributions of intermediate actions to conversation are undefined in the extrinsic reward, it is still feasible discern valuable intermediate actions that could potentially lead a successful conversation, and the learned intrinsic reward should help us identify them. We realize this by contrasting successful and failed episodes by the learnt intrinsic reward: preferred episode should have a higher likelihood under the optimal policy, comparing to less preferred one; and this optimal policy should be achieved by the correct reward function. Given policy ma, the probability of conversation 7 is preferred over 7 is computed based on the likelihood of the trajectories,
exp ElE log no (aolse)
Po Po > + exp Σ ElEro a log To (8:) exp Eeer log To (8))
(6)
Assume 7 preferred over 71, the resulting loss function given by,
C² = -E Σ log log P [=]
where r-,7' E B are sampled from buffer storing past trajectories. This follows the Bradley-Terry model [Bradley and Terry 1952] for estimating score functions from pairwise preferences. In the context of CRS, the preference is naturally defined by whether the recommendation is successful or not; and among successful recommendations, we prefer the one shorter. We truncate the failed trajectory to match the length of the successful trajectory.
5

---

4.3 Multi-Objective Bi-Level Optimization
The intrinsic reward function is expected to lead policy satisfying the above two objectives. This translates to a bi-level optimization procedure for policy learning: first update the policy with learned intrinsic rewards, and then improve the intrinsic rewards to help the resulting policy better satisfy the above two objectives. More formally, we define,
min L(0"),
(8)
s.t. 8' = arg min Cex+in (0,
where L((0) (C' and Cextim (0, 6) is the negative cumulative reward calculated with weighted sum yr + Ar À is papeer- to balance two rewards. In the inner loop, we optimize the policy with both the extrinsic reward and the learned intrinsic reward function. In the outer loop, we optimize the intrinsic reward function minimize the vector value loss. To derive the gradients for optimization, we first build the connection between 0 and Φ in the inner loop, and then derive the gradients on the outer loop.
Inner Loop: Optimizing 0, building the connection between and Φb We update 0 as follows,
8' = - 17. VoLertin o).
where VoLertin (0, o) can be calculated by the policy gradient theorem [Sutton al. 1999 and fj is the learning rate used in the inner loop. In this way, the updated parameter 0 becomes a function of c. With the built connection, we are able compute the gradient of by taking the gradient of gradient on f', i.e., the meta- pretaagradient..
Outer Loop: Optimizing Φ. In the outer loop, optimize the vector value loss L(0") to satisfy aforementioned two CRS-specific objectives. Even though we do not have supervision on s, the gradient of can still be derived using the chain rule,
g aL(0) 00
(10)
00'
Different from single objective optimization, the first part of [3) is the derivative w.r.t. the multi- objective function ((0'). Sener and Koltun [2018) adopt the multiple gradient descent algorithm
(MDGA) [Désidéri] 2012 to find Pareto stationary point for MOO problem. We follow their
approach to solve the following optimization problem,
VV + (1 (1-1)) a) (xx
min c-E[(.11]
(11)
where a has the following analytical solution,
a V LP - (0)
(12)
(er Ve. (P
+
where [++,; represents clipping to [0, 1] as [a]+,) = max(min(a, 1), 0). The resulted meta-gradient of Φ becomes,
OL (0)
08'
(CP(8")
08'
g(
(1 a)
(13)
00
00
Thus @ is updated by,
== g. (b).
(14)
where B is the learning rate used the outer loop. We can conclude the optimization in the outer loop as an automatic trade-off between two objectives, and thus the resulted intrinsic reward function expected to strike good balance between two CRS objectives.
Training procedure. Now we are finally equipped to illustrate the complete learning solution for CRSIRL in Algorithm In the inner loop, first rollout episode to caleulate Le In the outer loop, we also rollout an episode to calculate eav and sample a pair from the trajectory buffer B calculate LP. We run the inner loop and outer loop alternately until the model convergence.
5 Experiments

---

In this section, we conduct exten- sive experiments on three widely-used CRS benchmarks to study the fol- lowing research questions: (1) Can CRSIRL achieve better performance than state-of-the-art CRS solutions? (2) How does each proposed com- ponent contribute to the final perfor- mance of CRSIRL? (3) How does the degree of intrinsic rewards affect the policy learning?
Algorithm 1: Optimization algorithm of CRSIRI.
Initialize and d; Initialize the trajectory buffer B <- 23;
while not Done do # Inner Loop
Collect 7i by executing wp:
Compute updated parameters 0' with intrinsic reward
,ax Ar using (9)
# Outer Loop
Collect To by executing :e:
Compute _ with shaped reward he using (5.(5)
Sample 70, 7 ~ B; Compute L using Eq.
5.1 Setup
Update with gradients computed by Eq. 13
end
We evaluate CRSIRL on three multi-round CRS benchmarks
Datasets.
[Lei et al. 2020a, Deng et al., [021]. The LastFM dataset is for music artist recommendation.
Lei et al. 2020a] manually grouped the original attributes into 33 coarse-grained attributes. The
L dataset is the version where attributes are not grouped. The Yelp* dataset is for local business recommendation. We summarize their statistics Table
User simulator. Training and evaluating CRS through direct user interactions can be prohibitively expensive at scale, We ad- dress this by employing the user simulator approach from [Lei et al., 2020a), simulat- ing a conversation session for each observed user-item interaction pair (u, v). In this sim- ulation, item v is considered the target item, and its attribute set P., is treated as the ora- cle set of attributes preferred by user u. The session begins with the simulated user specifying attribute, randomly selected from P... This simulation adheres to the "System Ask, User Respond" paradigm CRS, as described in ZZang
Table 1: Summary statistics of datasets.
LastFM
L
Yelp
#UserS
1,801
1,801
27,675
#Items #Attributes
7,432
7,432
70,311
33
8,438
590
#Interactions
76,693
76,693
1,368,606
al. 2018).
Baselines. We consider rich of state-of-the-as CRS solutions. Max Entropy chooses to select an attribute with maximum entropy based on the current state, or to recommend the top ranked item. Abs Greedy (Christakopoulou et al. 2016] continues to suggest items until it either makes successful recommendation or reaches the maximum number of allowed attempts. CRM (Sun) and Zhang 2018 is an RI.-based solution. integrates user preferences into belief tracker, which then guides the pecision-making process regarding when to ask which attribute. EAR [Lei et al. 2020a] proposes a three-stage RL solution consisting of estimation, action and reflection. SCPR Lei et al. 2020b reconceptualizes the CRS problem an interactive path reasoning process within a user-item-attribuut graph. It selects candidate attributes and items based on their relationship to attributes that have already interacted with users within this graph. FPAN [Xu et al.) 2021 extends the EAR model by utilizing a -ttm-attributee graph to enhance offline representation learning. User embeddings revised dynamically based users' feedback on items and attributes in the conversation. UNICORN (Deng et al.) 2021 merges the conversation and recommendation components into unified RL agent. To streamline the RI. training process, it proposes two heuristic strategies for pre-selecting attributes and items at each turn.
Evaluation metries. We follow previous works on multi-round CRS to evaluate the performance of CRS with success rate at turn T' (SR @T) and average turns (AT) of conversations. SR @ is the average ratio of successful episodes with T turns, while AT is the average number of turns for all conversations. We also report the two-level hierarchical normalized discounted cumulative gain Deng al. 2021] defined as
T
ADCG@(T,K)
Σ Σ
ktt
1 log, (t 2)
1 + (1ogs(++1) +1 logz(t + 2) logz(k + 1)
k=1
7

---

Table 2: Main results. For SR@ 15 and hDCG, higher is better. For AT, lower is better. represents improvement over all baselines is statistically significant with p-value < 0.01.
LastFM
LastFM
Yelp*
SR@ 15
AT
hDCG
SR@ 15
AT
hDCG
SR @ 15
AT
hDCG
Abs Greedy
0.222
13.48
0.073
0.635
8.66
0.267
0.189
13.43
0.089
Max Entropy
0.283
13.91
0.083
0.669
9.33
0.269
0.398
13.42
0.121
CRM
0.325
13.75
0.092
0.580
10.79
0.224
0.177
13.69
0.070
EAR
0,429
12.88
0.136
0.595
10.51
0.230
0,182
13.63
0,079
SCPR
0,465
12.86
0.139
0.709
8.43
0.317
0,489
12.62
0.159
FPAN
0.630
10.16
0.224
0.667
7.82
0.407
0.236
12.77
0.116
UNICORN
0.535
11.82
0.175
0.788
7.58
0.349
0.520
11.31
0.203
CRSIRL
0.772
10.12
0.231
0.913
6.791
0.431
0.622
10.61
0.228!
Table 3: Abalation study of different components of CRSIRL.
LastFM
LastFM*
Yelp*
SR@15 15
AT
hDCG
SR @ 15
AT
hDCG
SR@ 15
AT
hDCG
PG
0.724
10.42
0.217
0.882
7.41
0.401
0.598
10.95
0.186
HRS
0.732
10.58
0.219
0.898
7.16
0.401
0.602
11.21
0.196
-RPM
0.754
10.14
0.227
0,904
6.89
0,415
0.606
10.81
0.213
MTL
0.768
10.09
0.224
0.908
6.92
0.426
0.613
10.73
0.207
CRSIRL
0.772
10.12
0.231
0.913
6.79
0.431
0.622
10.61
0.228
where T and K represent the number of conversation turns and recommended items in each turn, r(t, k)) denotes the relevance of the results the turn and position Intuitively, successful sessions with fewer turns are preferable for CRS. Also, the target item is expected to be ranked higher on the recommendation list at the success turn. We report hDCG@ (15, 10) by default.
Training details. All datasets are split by 7:1.5:1.5 ratio for training, validation and testing. We used the transformer-bssee based state encoder proposed in [Chu et al.) [023]. We adopt TransE (Bordes et al. 2013) to pretrain the node embeddings on the training set, and use the user simulator described before for online policy learning on the validation set. We first pretrain the policy with only extrinsic reward using policy gradient and then apply CRSIRL to fine-tune the pretrained policy. The learning rates in the inner and outer loop are searched from {le 5e 1e 4) with Adam optimizer. The coefficient of intrinsic reward À is searched from {0.05, 0.1, 0.5, 1.0]. The discount factor y is set 0.999, All experiments are run on an NVIDIA Geforce RTX 3080Ti GPU with 12 GB memory. RL-based baselines rely on handcrafted rewards, we follow Lei et al. [2020a] to set (1) Trec 1 for successful recommendation; (2) Frec fail -0.1 for failed recommendation; (3) Pask 0.1 when the inquired attribute is confirmed by the user; (4) Tree_aii = 0.11 when the inquired attribute dismissed by the user; (5) ruuit 0.3 when the user quits the conversation without successful
recommendation. We set the maximum turn 7 as 15 and the size K of the recommendation list as 10. We provide more implementation details in the supplementary material.
5.2 Results & Analysis
We present the main results in Table 2. We can clearly observe the CRSIRL outperformed all baselines with a large margin. Both FPAN and EAR are policy gradient based methods, but they pretrain their policies using conversation history generated by rule-based strategy via supervised learning. This training approach biases policies towards pre-set rules, limiting the performance of policy learning on datasets with larger action spaces (like LastFM and Yelp*), where more exploration is necessary. SCPR and UNICORN have relatively stable performance on all the datasets. Our CRSIRL outperforms baselines significantly with learned intrinsic rewards. Rather than arbitrarily assigning the reward values, we dynamically optimize them in CRSIRL. Any action that contributes to a final successful recommendation should receive credit and thus be promoted by policy learning, regardless of whether it involves rejected attribute or a failed recommendation.
8

---

5.3 Ablation Study
Contributions of each component in CRSIRL. We evaluate different variants of CRSIRL to study the contributions of each proposed component. Firstly, disable the fine-tuning with CRSIRL and directly report the results after the policy gradient pretraining with only extrinsic rewards, denoted as PG. Secondly, remove HRS and RPM to evaluate their individual effectiveness. In both variants, the outer loop degenerates to a single objective optimization problem. Finally, we conduct an experiment where instead of updating the objectives with MOO, we treat the two objectives as distinct tasks and assign them equal weights, a process referred to as Multi-Task Learning (MTL),
We present the results in Table 3 Interestingly. we observe that directly optimizing the extrinsic rewards with policy gradient already outperformed most of baselines. It is worth noting that PG uses sparser rewards than other baselines in Table with saauulll-ddffnned rewards for intermediate actions. However, the exploratory behavior of PG enables it to outperform these baselines. We can observe that without HRS, the AT metric degenerated on all three datasets. HRS prefers actions which can increase the rank of the target item, which is the most direct metric of action utilities in CRS. Even though the asked questions could still be helpful without HRS, HRS provides explicit hints about how to ask the most useful questions, leading to smaller AT. Besides, the performance decreases after removing RPM, which finds actions leading to successful recommendations by comparing successful and failed trajectories. Lastly, MTL shows a significant improvement compared to PG, and it occasionally outperforms CRSIRL (e.g. AT on LastFM). However, MTL has difficulty balancing the two objectives, generally resulting in worse performance than CRSIRL.
Analysis of X. We use the hyper-parameter À to control the influence of intrinsic rewards. It is im- portant to study how it affects the policy learning. In this experiment, we evaluate CRSIRL with X {0.05, 0.1, 0.5, 1.0} on the LastFM dataset. The re- sult is shown in Figure 3 We can clearly observe that the performance peaks when an appropriate is cho- sen. With a small A, the intrinsic reward is not strong enough to affect the final performance. However, large value of À can unfortunately impair performance. This is because the estimation errors in the intrinsic reward could overwhelm the extrinsic reward. Even though it is sparse, the extrinsic reward can help calibrate the intrinsic reward.
SAg15
10.4
0.76
10.3
0.75
0.74
0.73
10.1
0.72
10.0
0.05
0.1
0.5
1.0
Figure 3: Performance with different A.
5.4 Case Study
Additionally, we performed a qualitative study to analyze the learned intrinsic rewards of CRSIRL (shown Figure the LastFM dataset. The natural language questions and user responses are generated by predefined templates. We observe that the intrinsic rewards depend not only on whether the user accepts or rejects the action, but also on how well the action contributes to the final recommenda- tion. Even though the user accepts pop, the intrinsic reward for this action remains negative. This is because pop is a very general attribute and contributes little to modeling the user's preference. Conversely, although vocalist is re- jected by the user, it still carries a small positive value as aids in identifying the target artist. Finally, Indie and Punk are two attributes that are accepted and best describe the target artist, Franz Ferdinana (a band known for in- die rock and post-punk revival). Consequently, they carry relatively large positive intrinsic rewards. This case shows the CRSIRL can provide more fine-grained reward signals, leading to better final performance.
Do you like Rock?
Yest
Intrinsic R: 0.022
you pog?
Yes!
-0.082
Do you like good vocalist?
0.043
Noi
Do you like good Indie?
0.167
Do you like Punk?
Yes!
0.224
suggest Frenz Ferdinand.
Figure 4: Conversations generated by CRSIRL. The values of the learned in- trinsic rewards are marked in red.
https://een wwwp::iwwiiiiiiiiooeeeeeeeeeeeee org/wikiirranzz Fordinand__ (band)

---

6 Conclusions
In this paper, we study an important but largely under-explored problem in CRS, i.c., reward function design. We present a principled solution for reward learning for CRS and formulate an online algorithm to learn intrinsic rewards via bi-level multi-objective optimization. In the inner loop of CRSIRL, we optimize the policy with the learned intrinsic reward function. And in the outer loop, we optimize the intrinsic reward function to satisfy two criteria designed for CRS: maximizing success rate and minimizing number of turns in conversations. The results on three CRS benchmarks demonstrated the effectiveness of learned intrinsic rewards.
CRSIRL sheds light on learning reward functions to improve CRS. Currently, we consider two directly quantifiable objectives for CRS, i.c., success rate and conversation length. Other perspectives. such as user satisfaction [Liang et al. 2006] and fairness [Lin et al. 2022]. are worth to be investigated and embedded into reward learning. Moreover, beyond the template-based conversation generation, it is important to integrate CRSIRL with advanced natural language-
toguage-based based conversational agents, such as [Touvron al., 2023 Zhang et al. 2023). to learn reward functions that satisfy multiple objectives favored by humans during natural language driven interactions, such as conversational persuasiveness, cohesiveness and explainability [Moon et al. 2019].
7
Acknowledgement
We thank the anonymous reviewers for their insightful comments. This work was partially supported by NSF IIS-2007492, IIS-2128019 and NSF IIS- 1838615.
References
Peter Auer. Using confidence bounds for exploitation-eexplraatinn trade-offs. Journal of Machine Learning Research, 3(Nov):397-422, 2002.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26, 2013.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(344):24--45, 1952.
Qibin Chen, Junyang Lin. Yichang Zhang, Ming Ding, Yukuo Cen, Hongxia Yang, and Jie Tang. 2019.
Towards knowledge-baat based recommender dialog system. arXiv preprint
Konstantina Christakopoulou, Filip Radlinski, and Katja Hofmann. Towards conversational recom- mender systems. In Proceedings of the 22nd ACM SIGKDD, pages 815-824, 2016.
Konstantina Christakopoulou, Alex Beutel, Rui Li, Sagar Jain, and Ed H Chi. Q&r: A two-stage ap- proach toward interactive recommendation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pages 148, 2018.
Zhendong Chu and Hongning Wang. Meta-reinforcemment learning via exploratory task clustering. arXiv preprint arXiv:2302.009958 2023.
Zhendong Chu, Hongning Wang, Yun Xiao, Bo Long, and Lingfei Wu. Meta policy learning for cold-start conversational recommendation. Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, pages 222-230, 2023.
Kalyanmoy Deb and Kalyanmoy Deb. Multi-objective optimization. In Search methodologies: Introductory futorials in optimization and decision support techniques, pages 403-449. Springer, 2013.
Yang Deng, Yaliang Li, Fei Sun, Bolin Ding, and Wai Lam. Unified conversational recommddttoooooo policy learning via graph- reinforcement learning. arXiv preprint arXiv:2105.097110 2021.
Jeto-Antoine Désidéri. Multiple-gradien descent algorithm (mgda) for multiobjective optimization. Comptes Rendus Mathematique, 350(5-6):313-113 2012.
10

---

Wenqiang Lei. Xiangnan He, Yisong Miao, Qingyun Wu, Richang Hong, Min- Yen Kan, and Tat- Seng Chua. ietimation-action-ree recommender systems. In Proceedings of the 13th WSDM Conference, pages 304-312, 2020a.
Towards deep interaction between conversational and
Wengiang Lei, Gangyi Zhang, Xiangnan He, Yisong Miao, Xiang Wang, Liang Chen, and Tat-Seng Chua. Interactive path reasoning on graph for conversational recommendation. In Proceedings of the 26th ACM SIGKDD, pages 2073-2083, 2020b.
Ting- Liang, Hung-Jen Lai, and Yi-Cheng Ku. Personalized content recommendation and user satisfaction: Theoretical synthesis and empirical findings. Journal of Management Information Systems, 23(3):45-770 2006.
Shuo Lin, Ziwei Zhu, Jianling Wang. and James Caverlee. Towards fair conversational recommender systems. arXiv preprint 0:Xiv:2208.03854, 2022.
Runze Liu, Fengshuo Bai, Yali Du, and Yaodong Yang. Meta-reward-net: Implicitly differentiable reward learning for preference-basee reinforcement learning. Advances in Neural Information Processing Systems, 35:22270 22284, 2022.
Seungwhan Moon, Pararth Shah, Anuj Kumar, and Rajen Subba. Opendialkg: Explainable conversa- tional reasoning with attention based walks over knowledge graphs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 845-854, 2019.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. leml, volume 99, pages 278-281 Citeseer, 1999.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity driven exploration by self-supervised prediction. In International conference on machine learning, pages 2778-2787. PMLR, 2017.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31, 2018
Satinder Singh, Richard L Lewis, Andrew G Barto, and Jonathan Sorg. Intrinsically motivated reinforcement learning: An evolutionary perspective. IEEE Transactions on Autonomous Mental Development, 2(2):70-82, 2010.
Yueming Sun and Yi Zhang. Conversational recommender system. In The 41st ACM SIGIR, pages 235-244, 2018
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397/ 2023.
Kerui Xu, Jingxuan Yang. Jun Xu, Sheng Gao, Jun Guo, and Ji-Rong Wen. Adapting user preference to online feedback in multi-round conversational recommendation. In Proceedings of the 14th ACM WSDM, pages 364-372, 2021.
Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Guoyin Wang, and Yiran Chen. Towards building the federated gpt: Federated instruction tuning. arXiv preprint arXiv:2305.055: 05644,, 2023.
Yongfeng Zhang, Xu Chen, Qingyao Ai, Liu Yang. and W Bruce Croft. Towards conversational search and recommendation: System ask, user respond. Proceedings of the 27th acm international conference on information and knowledge management, pages 177-186, 2018.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. Advances in Neural Information Processing Systems, 31, 2018
Zeyu Zheng, Junhyuk Oh, Matteo Hessel, Zhongwen Xu, Manuel Kroiss, Hado Van Hasselt, David Silver, and Satinder Singh. What can learned intrinsic rewards capture? In International Conference on Machine Learning, pages I 11436-11466 PMLR, 2020.
11


Reviews:
[{"review_id": "QhmaMVGSJ9", "reviewer": "Reviewer_reFg", "summary": "The paper addresses the problem of designing effective reward functions for conversational recommender systems (CRS), which is critical but largely under-explored in the literature. The paper proposes a novel approach to learn intrinsic rewards from user feedback, which can better capture the user intent and optimize multiple CRS-specific objectives, such as success rate and conversation length. The paper formulates intrinsic reward learning as a multi-objective bi-level optimization problem, and develops an algorithm to solve it. The paper evaluates the proposed approach on three public CRS benchmarks, and shows that it significantly improves CRS performance by exploiting informative learned intrinsic rewards.", "questions": "1) How does the user simulator approach compare to other methods of evaluating CRS solutions, such as human evaluation or simulation with real user logs?\n2) How does the proposed CRSIRL handle the exploration-exploitation trade-off in policy learning, especially when the intrinsic rewards are uncertain or noisy?\n3) How does the multi-objective bi-level optimization framework deal with the potential conflicts or trade-offs between the two objectives of maximizing success rate and minimizing number of turns? Authors mention this in section 3 but it's not analyzed in the results section. \n4) Can CRSIRL incorporate user feedback on the generated recommendations, such as ratings, reviews, or explanations?\n5) Can CRSIRL handle the cold start problem, when there is no or little prior information about the user preferences or behavior?\n6) Can CRSIRL cope with the dynamic and evolving nature of the user preferences and behavior, especially in long-term interactions?\n7) How does the CRSIRL leverage the contextual information, such as user profile, location, time, or mood, to improve the recommendation quality and user satisfaction?", "limitations": "NA", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) Originality: The paper proposes a novel approach to learn intrinsic rewards from user feedback, which can better capture the user intent and optimize multiple CRS-specific objectives. This represents a creative combination of existing ideas and a new problem formulation in the field of conversational recommender systems.\n2) Quality: The paper formulates intrinsic reward learning as a multi-objective bi-level optimization problem, and develops an online algorithm to solve it. The proposed approach is rigorously evaluated on three public CRS benchmarks, and shows significant improvement in CRS performance by exploiting informative learned intrinsic rewards.\n3) Clarity: The paper is well-written and clearly presents the research problem, methodology, results, and conclusions. The technical details are explained in a clear and concise manner, making it accessible to a broad audience.\n4) Significance: It is questionable if The paper makes a significant contribution to the field of conversational recommender systems. I am not sure if any of this will actually be used in practical systems because algorithms are evaluated on non practical scenarios/variants of datasets.", "weaknesses": "1) The paper does not provide any details on convergence of the algorithm or computations complexity of the proposed method against existing methods. \n2) Even though it seems like a useful strategy to use intrinsic rewards, It is unclear how the intrinsic rewards affect the policy learning process. This is not well motivated using experimental results. \n3) How does a user simulator affect overall results? Can authors justify the specific choice of user simulator? Why does the attribute set Pv be treated as the oracle set?\n4) It is also unclear how the hyperparameters of the model are tuned and how they affect the performance.\n5) Not exactly a weakness but this would be an interesting study too: The paper does not compare the proposed method with any baselines that use real user feedback, such as ratings, reviews, or explanations. It is possible that the user simulator may not capture the true user preferences or behavior, and that the intrinsic rewards may not reflect the real user satisfaction. It is also possible that the user feedback may provide additional information or guidance for improving the recommendation quality and user satisfaction.\n6) The paper does not evaluate the proposed method on a diverse set of datasets such as e-commerce, music, or movies. It is unclear how the proposed method would generalize to different domains, contexts, or user types. It is also unclear how the proposed method would handle practical challenges, such as data sparsity, noise, or diversity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive comments on our work, and the constructive questions to help enrich our work in the future.\n\n---\n\n**[Q1]** How does the user simulator approach compare to other methods of evaluating CRS solutions?\n\n**[A1]** Due to interactive nature of conversational recommendation, training and evaluating CRS with real users is costly. We use a user simulator, inspired by [1], for efficient training and evaluation. Assuming users favor all attributes of the target item, we treat the attribute set $P_v$ as the oracle, ensuring consistent feedback. While crafting a more realistic simulator is compelling, our paper focuses on reward learning. For fair comparisons, we choose to align our evaluation with established baselines\n\n\nOn the other hand, using logged data to learn or evaluate a RL algorithm is known to be challenging, due to the issue of distribution shift. Though various solutions have been proposed, they have different limitations (e.g., variance vs., bias trade-off, and computational complexity), which add additional complexity in our study. Given we are the first to study the reward learning problem in CRS, we choose simulations to evaluate our proposed solution. \n\n\n[1] Sun, Yueming, and Yi Zhang. \"Conversational recommender system.\" The 41st international acm sigir conference on research & development in information retrieval. 2018.\n\n---\n\n**[Q2]** How does the proposed CRSIRL handle the exploration-exploitation trade-off in policy learning?\n\n**[A2]** While intrinsic rewards can be noisy, our framework relies on noise-free extrinsic rewards in the outer loop for stable feedback. These extrinsic rewards offer a clear training signal, enabling CRSIRL to refine intrinsic rewards and guide the policy to more effective actions. As a result, despite the noisy intrinsic feedback, CRSIRL maintains a balance between exploration and exploitation during the training. Additionally, as suggested by [1], noisy feedback can enhance exploration, potentially benefiting early training stages.\n\n[1] Kannan, Sampath, et al. \"A smoothed analysis of the greedy algorithm for the linear contextual bandit problem.\" Advances in neural information processing systems 31 (2018).\n\n---\n\n**[Q3]** Potential conflicts or trade-offs between the two objectives\n\n**[A3]** Our multi-objective optimization framework adeptly navigates these trade-offs, ensuring convergence to a point on the Pareto frontier, as evidenced in [1]. CRSIRL does not merely establish an arbitrary trade-off between the two objectives by a manually-set hyper-parameter, but strives to optimize them concurrently (as explained between line 221-234). CRSIRL realizes it by dynamically adjusting the weights, denoted as $\\alpha$, for each objective throughout the training process. As Table 3 illustrates, while the multi-task learning variant can indeed benefit from the dual objectives, CRSIRL can still outperform it by establishing a more desirable trade-off between the two objectives.\n\n[1] Désidéri, Jean-Antoine. \"Multiple-gradient descent algorithm (MGDA) for multiobjective optimization.\" Comptes Rendus Mathematique 350.5-6 (2012): 313-318. \n\n---\n\n**[Q4]** Can CRSIRL incorporate user feedback on the generated recommendations?\n\n**[A4]** CRSIRL is capable of integrating various types of user feedback, such as ratings, reviews, or explanations, as signals of extrinsic reward. And they can be included as other optimization objectives within our multi-objective framework. However, it's worth noting that incorporating user feedback would not fundamentally alter the focus of this paper, which is to investigate effective reward learning strategies in CRS. We leave how to effectively utilize other types of user feedback for intrinsic reward learning as an important future work.\n\n---\n\n**[Q5]** Can CRSIRL handle the cold start problem, when there is no or little prior information about the user preferences or behavior?\n\n**[A5]** CRS is expected to handle cold-start problems in recommendation by profiling a new user via eliciting her preference about item attributes on the fly. And CRSIRL is expected to better handle the cold-start problem by learning more effective rewards to guide CRS policy learning.\n\n---\n\n**[Q6]** Can CRSIRL cope with the dynamic and evolving nature of the user preferences and behavior, especially in long-term interactions?\n\n**[A6]** The dynamic and evolving nature of user preferences actually signal a shift in the reward distribution, which poses a non-stationary learning environment. This is known as a significant challenge in reinforcement learning in general. Not only does the learned reward function but also the policy learning algorithm has to cope with it. Currently, the CRSIRL framework is not designed to handle such a non-stationary environment, particularly in the context of long-term interactions where user behavior and preferences may evolve considerably. The reviewer’s comment is very well taken, and we acknowledge this as an area for potential enhancement of our framework.\n\n---\n\n**[Q7]** How does the CRSIRL leverage the contextual information to improve the recommendation quality and user satisfaction?\n\n**[A7]** Currently, we learn the intrinsic reward function to maximize the recommendation quality. However, it is possible to further improve user satisfaction by learning context-aware reward functions by encoding contextual information into reward functions, i.e., context-aware CRS. For example, different context induces different conversation strategies. It would be an interesting research direction for us in the future.\n\n---\n\n**[Q8]** Hyper-parameter setting\n\n**[A8]** We provide the search ranges for hyper-parameters in Section 5.1 under \"Training Details.\" Meanwhile, an in-depth discussion regarding the impact of different $\\lambda$ is included in Section 5.3.\n\n---\n\n**[Q9]** Convergence analysis\n\n**[A9]** Please refer to the general response CQ1. \n"}, {"review_id": "cwNoMAPJll", "reviewer": "Reviewer_Kjom", "summary": "Mainstream reinforcement learning-based CRS solutions heavily rely on handcrafted reward functions, which may not be aligned with user intent in CRS tasks. Therefore, the design of task-specific rewards is critical to facilitate CRS policy learning, which remains largely under-explored in the literature. This paper proposes a novel approach to address this challenge by learning intrinsic rewards from interactions with users. Specifically, the paper formulates intrinsic reward learning as a multi-objective bi-level optimization problem. Extensive experiments on three public CRS benchmarks show that the developed algorithm significantly improves CRS performance by exploiting informative learned intrinsic rewards.", "questions": "(1) What are the comparison results, using the metric SR@K, where K is a small number (e.g., 3 or 5)?\n\n(2) Are the inner loop optimization and outer loop optimization in an alternating fashion during training? If so, is there any insight or theoretical analysis about the convergence of the training procedure?", "limitations": "Limitations seem not discussed in the paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) The paper is well motivated, with a convincing motivating example.\n(2) The develop technique is solid and has clear intuitions.\n(3) Clear improvements on multiple datasets are observed in the experiments.", "weaknesses": "(1) More experiment details can be included. Previous works usually show the curves reporting the performance with different training episodes and different turns of conversations, for example, in baselines\n\nYang Deng, Yaliang Li, Fei Sun, Bolin Ding, and Wai Lam. Unified conversational recommendation policy learning via graph-based reinforcement learning. arXiv preprint arXiv:2105.09710, 2021.\n\nWenqiang Lei, Xiangnan He, Yisong Miao, Qingyun Wu, Richang Hong, Min-Yen Kan, and Tat-Seng Chua. Estimation-action-reflection: Towards deep interaction between conversational and recommender systems. In Proceedings of the 13th WSDM Conference, pages 304–312, 2020a.\n\nUsually learning a more generic reward function may need more samples compared to learning a handcrafted reward function, although after convergence, learning the generic reward function can lead to better performance compared to learning a handcrafted reward function. It would be interesting if the paper can report SR@K, where K is a small number, because the user experience in early interactions is very important in conversational recommendations.\n\n(2) The way of user interactions in Figure 1 (System Ask, User Respond) may become less interesting, especially when nowadays where users can progressively and flexibly describe what they want by LLMs with decent accuracy of understanding the desired attributes. For example, as in Table 2, by all the algorithms in this setting (System Ask, User Respond), even after 15 interactions, very few algorithms can achieve satisfying recommendations. However, if the user can progressively and flexibly describe what they want by LLMs, it is likely that the user can get proper recommended items with only very few interactions. Even the user may not be progressive in some cases, a hybrid setting (where the user can either respond yes/no or progressively describe what they want during the conversations) is more practical.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the constructive comments on enriching experiment results and more practical CRS paradigm.\n\n---\n\n**[Q1]** What are the comparison results, using the metric SR@K, where K is a small number (e.g., 3 or 5)?\n\n**[A1]** We report the SR@5 of CRSIRL, compared with the strongest baseline UNICORN, \n\n|         | LastFM | LastFM* | Yelp* |\n|:-------:|:------:|:-------:|:-----:|\n| UNICORN |  0.104 |  0.215  | 0.068 |\n|  CRSIRL |  0.262 |  0.324  | 0.182 |\n\nThis notable performance gain can be attributed to CRSIRL's strategy of posing more informative questions early in the conversation, allowing it to make more accurate recommendations as the conversation progresses. We will include more experiment details in the final version.\n\n---\n\n**[Q2]** Are the inner loop optimization and outer loop optimization in an alternating fashion during training? If so, is there any insight or theoretical analysis about the convergence of the training procedure?\n\n**[A2]** The inner and outer loops are operated in an alternative fashion, as all other solutions for bi-level optimization. Please refer to the general response CQ1 for the convergence analysis of CRSIRL.\n\n---\n\n**[Q3]** Progressive CRS paradigm\n\n**[A3]** Progressive CRS presents a more reasonable paradigm, allowing users to take a more active role in the conversation by explicitly describing their preferences. The idea of learning CRS with such actively participative users is interesting. However, at the time of our paper's submission, creating a progressive simulator based on LLMs is still an emerging field of study [1]. It's important to clarify that designing such a progressive user simulator, or any specific type of user simulator, is not the focus of our paper. Instead, our primary objective is the effective learning of optimal reward functions for CRS. For evaluations, we employed the widely-accepted \"System Ask, User Respond\" [2] paradigm. While the fundamental mechanism of reward learning remains consistent, irrespective of the CRS setting, we believe that the integration of our proposed method within the progressive CRS paradigm will be a seamless process. Exploring this integration further is a promising direction and we leave it as an important future work.\n\n[1] Wang, Xiaolei, et al. \"Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models.\" arXiv preprint arXiv:2305.13112 (2023).\n\n[2] Zhang, Yongfeng, et al. \"Towards conversational search and recommendation: System ask, user respond.\" Proceedings of the 27th acm international conference on information and knowledge management. 2018.\n"}, {"review_id": "Qdr7I4wvAu", "reviewer": "Reviewer_vRxN", "summary": "This paper study the problem of reinforcement learning based conversational recommender systems. The paper claims that it is difficult to design a handcraft reward function for each step of the conversations. Thus the paper proposes a multi-objective bi-level optimization method that the inner level optimizes the policy with real rewards and the learned intrinsic rewards, and the outer level optimizes two CRS objectives: maximizing the success rate and minimizing the number of turns in successful conversations. The paper validates the effectiveness of the proposed methods in 3 CRS datasets.", "questions": "See the weakness.", "limitations": "See the weakness.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.The motivation of the paper is very clear.\n2.The idea of hindsight reward shaping is sound and interesting.\n3.The multi-objective bi-level optimization is also interesting and easy to follow.\n4.The paper provide detailed introductions of evaluation metrics, and the performance improvement is significant.\n5.The paper shows the ablation study of the HRS and the RPM component.", "weaknesses": "1.The major concern of the paper is that it is not clear that how the proposed techniques work together. First of all, the intrinsic reward function defined in Eq.(3) is related to the rank of the target item under the state. How is it related to the parameter \\phi?\n\n2.How does the RPM module work? It seems we train the policy with Eq.(7). Is it related to the intrinsic reward?\n\n3.The performance improvement of CRSIRL is marginal compared with MTL, which means that we do need to use the MDGA algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the motivation of our paper and our idea being interesting. We hope the following responses can address the reviewer’s concerns. \n\n---\n\n**[Q1]** Clarification about how the components of CRSIRL work together\n\n**[A1]** We develop a bi-level optimization framework to learn intrinsic rewards from users’ explicitly provided reward signals for CRS. Our solution consists of an inner loop and an outer loop for the induced bi-level optimization problem. In the inner loop, we update the policy $\\pi$ with the learned intrinsic reward (parameterized by $\\phi$) to obtain an updated policy $\\pi’$. In the outer loop, we update $\\phi$ by calculating the meta loss $\\mathbb{L}$ induced by $\\pi’$ over the users’ provided explicit reward signal. Intuitively, we hope the learned intrinsic rewards can help us find a better policy $\\pi’$, leading to a lower meta-loss. \n\nMore specifically, to obtain the gradient of $\\phi$, in the inner loop, we create a differentiable connection between $\\pi$ and $\\phi$ via gradient update (Eq.9). Such a connection allows us to calculate the gradient of $\\phi$ w.r.t. the meta loss using chain rule (Eq.10 and 13). After obtaining the gradient of $\\phi$ in the outer loop, we are able to update it using Eq.14. \n\nTo better leverage explicit reward from user feedback, we design two CRS-specific objectives HRS (Eq.3) and RPM (Eq.7), and construct the multi-objective meta loss $\\mathbb{L}$. Hence, we will not directly use HRS and RPM to update the policy, but use them to guide the learning of intrinsic reward. We will add the pseudo code of CRSIRL to facilitate better understanding in the final version.\n\n---\n\n**[Q2]** Performance comparison between CRSIRL and MTL\n\n**[A2]** While MTL is a strong baseline that leverages our proposed HRS and RPM objectives, its static weight setting between the two objectives makes it less adaptable in different environments. This rigidity can result in one objective dominating the other. For example, in Table 3, MTL is skewed towards optimizing the average turn on the LastFM dataset.\n\nIn contrast, CRSIRL does not merely establish a simple trade-off between the two objectives by a manually-set hyper-parameter, but strives to optimize them concurrently (as explained between line 221-234). By utilizing the MDGA algorithm, CRSIRL can flexibly adjust the weights between objectives, striving for a Pareto optimal solution, as supported by the findings in [1]. This ensures simultaneous optimization of both objectives without one excessively dominating the other. Our multi-objective framework not only theoretically avoids suboptimal outcomes but also demonstrates better empirical results over MTL, as observed in Table 3.\n\nFurthermore, our experiments were repeated three times with different random seeds, and the improvement brought by CRSIRL over MTL has been found to be statistically significant ($p<0.05$). We'll be incorporating the significance test results in the final version of the paper.\n\n[1] Désidéri, Jean-Antoine. \"Multiple-gradient descent algorithm (MGDA) for multiobjective optimization.\" Comptes Rendus Mathematique 350.5-6 (2012): 313-318. \n"}, {"review_id": "7F50ojc4wm", "reviewer": "Reviewer_5ZDc", "summary": "In this paper, the authors propose an algorithm for learning the intrinsic reward function in order to solve the problem of poor results due to improper design of the reward function of the dialogue strategy module in current conversation recommendation systems. Specifically, a multi-objective bi-level optimization problem is designed, where the inner layer uses the learned reward function to optimize the selection strategy, and the outer layer updates the reward function used to optimize the system metrics. Excellent results are achieved in two conversation recommendation datasets.\n\n", "questions": "Please refer to weakness. ", "limitations": "Please refer to weakness. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "It is interesting research direction to design an internal reward based on the results of each interaction, in addition to the external reward generated by the conversation results, to satisfy the results of two recommendation goals through a multi-objective bi-level optimization framework.\nA hindsight internal reward function was designed to calculate the reward score of the current strategy after each target item was hit.\nIn order to obtain successful recommendation results faster and thus reduce the number of conversation rounds, the authors propose the Recommendation Preference Matching module, which improves the possibility of selecting the right decision.\nThe authors designed a Multi-Objective Bi-Level Optimization strategy in which the inner loop, using two rewards, optimizes the decisions of the conversation, while the outer loop optimizes the reward function.\nThe experiment design is comprehensive and solid, and the result analysis is clear and convincing.\n", "weaknesses": "First of all, your baseline is not enough, the CRIF model published in SIGIR in 2022 is not compared, and from the results of CRIF, your performance is worse than it, which needs to be verified by adding more experiments.\nI can't understand the example you showed in Case Study, pop and rock are very common examples, and Franz Ferdinand also has the label of pop rock, why pop gave a negative score it, and the contribution of this article is mainly in the conversation policy module, in the second step to raise a completely useless question, is this a policy failure?\nYour work contains only the content of the dialogue policy module, and in the experimental part we can see that it has good recommendation performance, but the article does not describe how to get the recommendation results, please explain.\nIt is suggested to include a significance test to prove the validity of the experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive suggestions to strengthen the empirical support of our work by adding more advanced baselines and significance testing. \n\n---\n\n**[Q1]** Compare with CRIF\n\n**[A1]** We discussed the CRIF model in our appendix and reported the experiment comparisons with it in the appendix B. Please kindly check it in the supplementary material. CRIF depends on a heuristic to identify the best action at each round of interaction. However, this heuristic abuses the design of the user simulator that all attributes of the target item will be accepted. This creates a form of information leakage and an unfair advantage over other solutions that do not exploit this specific knowledge. Thus we did not include it in the main paper. \n\nHowever, it is still interesting to understand whether the learned intrinsic rewards can augment this strong heuristic. The experiment results in appendix B showed that learning intrinsic rewards directly from user interactions offers an effective augmentation to CRIF, resulting in better performance. We will explicitly refer to this study and its conclusions in the main paper.\n\n---\n\n**[Q2]** Recommendation results\n\n**[A2]** Building upon [1], we employ a unified policy in which the action space consists of both attributes and items. When the policy selects an attribute action, the agent inquires if the user prefers that specific attribute. Conversely, if an item action is chosen, the agent compiles a recommendation list for the user, with items ranked based on the scores assigned by the policy. A better policy is recognized by its ability to not only strategize questions more effectively (i.e., the average turn metric) but also produce an optimized recommendation list (i.e., the ranking quality). To evaluate the quality of the recommendation list, we utilize the hDCG metric. We will add a detailed description of how to obtain the recommendation results in the final version.\n\n[1] Yang Deng, Yaliang Li, Fei Sun, Bolin Ding, and Wai Lam. Unified conversational recommendation policy learning via graph-based reinforcement learning. arXiv preprint arXiv:2105.09710, 2021.\n\n---\n\n**[Q3]** Clarification of the Case Study\n\n**[A3]** Thank you for your insightful observation regarding the Case Study! In the provided example, even though \"pop\" was accepted by the user, it received a negative intrinsic reward because it is a broad genre, associated with too many artists. This means that while \"pop\" may be liked, it is still not the most informative for refining recommendations at the moment. As a result, the intrinsic reward function gives \"pop\" a slightly negative reward, despite Franz Ferdinand falling under \"pop rock\". While the asked questions may occasionally seem redundant, it’s expected as the policy tries to avoid jumping to conclusions based on limited information (e.g., recommending at the beginning of the conversation). We see it as laying a foundational understanding of a user's preferences.\n\n---\n\n**[Q4]** Significance test\n\n**[A4]** Thanks for your suggestion! All the experiments are repeated 3 times with different random seeds, and we report the final averaged metrics. We will add the results of the significance test to ensure the validity of the results. \n"}]

Author rebuttal:
We sincerely thank all the reviewers for their thoughtful comments and constructive suggestions, which will help us strengthen our paper. We are encouraged to find that the reviewers appreciate the clear presentation (reviewer reFg, vRxN), motivation of our study (reviewer reFg, Kjom, vRxN, 5ZDc), novelty of our approach (reviewer reFg, Kjom, vRxN), solid experiment design and improvement (reviewer reFg, Kjom, vRxN, 5ZDc). In the following, we will first provide the answer to the common question regarding the convergence analysis of our method, and then endeavor to provide individual responses to each reviewer. 

---

**[CQ1]** Convergence analysis of CRSIRL (Reviewer reFg and Kjom)

**[CA1]** We provide the proof sketch of the convergence rate as follows. 

Based on the results in [1], to prove the convergence of CRSIRL, we only need to prove our multi-objective meta loss function is Lipschitz smooth. Formally, we prove the following lemma,

*Lemma 1*. Given Lipschitz smooth loss functions $f$ and $g$, the following function is also Lipschitz smooth,

$\alpha f+(1-\alpha) g,$

where $\alpha \in [0, 1]$.

*Proof*. Let $f$ and $g$ be Lipschitz smooth with Lipschitz constants $L_f$ and $L_g$ respectively. This means:

1. $|f(x)-f(y)| \leq L_f|| x-y||$,  for all $(x, y)$,
2. $|g(x)-g(y)| \leq L_g || x-y||$, for all $(x, y)$.

Consider a function $h(x)=\alpha f(x) + (1-\alpha )g(x)$, we need to show $h(x)$ is Lipschitz smooth.

$
|h(x)-h(y)|=|\alpha f(x)+(1-\alpha)g(x)-\alpha f(y)-(1-\alpha)g(y)|
               =|\alpha (f(x)-f(y))+(1-\alpha )(g(x)-g(y))|.
$

Using the triangle inequality, we have

$
|h(x)-h(y)| \leq \alpha |(f(x)-f(y))|+|(1-\alpha)(g(x)-g(y))|
                       = \alpha |(f(x)-f(y))|+(1-\alpha)|(g(x)-g(y))|.
$

By using the inequalities we induced by Lipschitz smooth, we get 

$
|h(x)-h(y)| \leq \alpha L_f || x-y ||+(1-\alpha)L_g||x-y||
           = (\alpha L_f +(1-\alpha)L_g)||x-y||.
$   

Thus, the function $h$ is Lipschitz smooth with Lipschitz constant $L_h=\alpha L_f +(1-\alpha)L_g$.

*QED.*

By plugging *Lemma 1* into *Theorem 1* and *Theorem 2* of [1], we are able to prove the convergence guarantee of CRSIRL. We will provide the complete proof in the final version of our paper. 

[1] Liu, Runze, et al. "Meta-reward-net: Implicitly differentiable reward learning for preference-based reinforcement learning." Advances in Neural Information Processing Systems 35 (2022): 22270-22284.

Meta-review: The paper studies multi-objective intrinsic reward learning to improve conversational recommender systems with multi-dimensional goals, such as maximizing success rate and shorten conversation length. Reviewers are overall happy with the studied problem, the proposed method and clarity of the presentation. The ablation study offer some clarity into the importance of the different components introduced, but the discussion on these results are short and do not offer much more insights.
Decision: Accept (poster)