SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
g (\phi) = \alpha \cdot \frac {\partial \mathcal {L} ^ {e x} \left(\theta^ {\prime}\right)}{\partial \theta^ {\prime}} \cdot \frac {\partial \theta^ {\prime}}{\partial \phi} + (1 - \alpha) \cdot \frac {\partial \mathcal {L} ^ {p} \left(\theta^ {\prime}\right)}{\partial \theta^ {\prime}} \cdot \frac {\partial \theta^ {\prime}}{\partial \phi}. \tag {13}
$$
Thus  $\phi$  is updated by,
$$
\phi^ {\prime} = \phi - \beta \cdot g (\phi), \tag {14}
$$
where  $\beta$  is the learning rate used in the outer loop. We can conclude the optimization in the outer loop as an automatic trade-off between two objectives, and thus the resulted intrinsic reward function is expected to strike a good balance between two CRS objectives.
Training procedure. Now we are finally equipped to illustrate the complete learning solution for CRSIRL in Algorithm 1. In the inner loop, we first rollout an episode to calculate  $\mathcal{L}^{ex + in}$ . In the outer loop, we also rollout an episode to calculate  $\mathcal{L}^{ex}$  and sample a pair from the trajectory buffer  $\mathcal{B}$  to calculate  $\mathcal{L}^p$ . We run the inner loop and outer loop alternately until the model convergence.
In this section, we conduct extensive experiments on three widely-used CRS benchmarks to study the following research questions: (1) Can CRSIRL achieve better performance than state-of-the-art CRS solutions? (2) How does each proposed component contribute to the final performance of CRSIRL? (3) How does the degree of intrinsic rewards affect the policy learning?
Datasets. We evaluate CRSIRL on three multi-round CRS benchmarks
[Lei et al., 2020a, Deng et al., 2021]. The LastFM dataset is for music artist recommendation. Lei et al. [2020a] manually grouped the original attributes into 33 coarse-grained attributes. The LastFM* dataset is the version where attributes are not grouped. The Yelp* dataset is for local business recommendation. We summarize their statistics in Table 1.
User simulator. Training and evaluating CRS through direct user interactions can be prohibitively expensive at scale. We address this by employing the user simulator approach from [Lei et al., 2020a], simulating a conversation session for each observed user-item interaction pair  $(u, v)$ . In this simulation, item  $v$  is considered the target item, and its attribute set  $\mathcal{P}_v$  is treated as the oracle set of attributes preferred by user  $u$ . The
session begins with the simulated user specifying an attribute, randomly selected from  $\mathcal{P}_v$ . This simulation adheres to the "System Ask, User Respond" paradigm in CRS, as described in [Zhang et al., 2018].
Baselines. We consider a rich set of state-of-the-art CRS solutions. Max Entropy chooses to select an attribute with maximum entropy based on the current state, or to recommend the top ranked item. Abs Greedy [Christakopoulou et al., 2016] continues to suggest items until it either makes a successful recommendation or reaches the maximum number of allowed attempts. CRM [Sun and Zhang, 2018] is an RL-based solution. It integrates user preferences into a belief tracker, which then guides the decision-making process regarding when to ask which attribute. EAR [Lei et al., 2020a] proposes a three-stage RL solution consisting of estimation, action and reflection. SCPR [Lei et al., 2020b] reconceptualizes the CRS problem as an interactive path reasoning process within a user-item-attribute graph. It selects candidate attributes and items based on their relationship to attributes that have already interacted with users within this graph. FPAN [Xu et al., 2021] extends the EAR model by utilizing a user-item-attribute graph to enhance offline representation learning. User embeddings are revised dynamically based on users' feedback on items and attributes in the conversation. UNICORN [Deng et al., 2021] merges the conversation and recommendation components into a unified RL agent. To streamline the RL training process, it proposes two heuristic strategies for pre-selecting attributes and items at each turn.
Evaluation metrics. We follow previous works on multi-round CRS to evaluate the performance of CRS with success rate at turn  $T$  ( $\mathrm{SR} @ T$ ) and average turns (AT) of conversations.  $\mathrm{SR} @ T$  is the average ratio of successful episodes with  $T$  turns, while AT is the average number of turns for all conversations. We also report the two-level hierarchical normalized discounted cumulative gain [Deng et al., 2021] defined as
$$
h D C G @ (T, K) = \sum_ {t = 1} ^ {T} \sum_ {k = 1} ^ {K} r (t, k) \bigg [ \frac {1}{\log_ {2} (t + 2)} + \bigg (\frac {1}{\log_ {2} (t + 1)} - \frac {1}{\log_ {2} (t + 2)} \bigg) \frac {1}{\log_ {2} (k + 1)} \bigg ],
$$
Algorithm 1: Optimization algorithm of CRSIRL
Initialize  $\theta$  and  $\phi$    
Initialize the trajectory buffer  $\mathcal{B}\gets \varnothing$    
while not Done do #Inner Loop Collect  $\tau_{i}$  by executing  $\pi_{\theta}$  . Compute updated parameters  $\theta^{\prime}$  with intrinsic reward  $r^{ex} + \lambda r_{\phi}^{in}$  using Eq.(9); #Outer Loop Collect  $\tau_{o}$  by executing  $\pi_{\theta '}$  . Compute  $\mathcal{L}^{ex}$  with shaped reward  $\tilde{R}^{ex}$  using Eq.(5); Sample  $\tau^0,\tau^1\sim \mathcal{B}$  ; Compute  $\mathcal{L}^p$  using Eq.(7); Update  $\phi$  with gradients computed by Eq.(13);   
end
Table 1: Summary statistics of datasets.
LastFM	LastFM*	Yelp*
#Users	1,801	1,801	27,675
#Items	7,432	7,432	70,311
#Attributes	33	8,438	590
#Interactions	76,693	76,693	1,368,606
Table 2: Main results. For SR@15 and hDCG, higher is better. For AT, lower is better.  $^\dagger$  represents the improvement over all baselines is statistically significant with  $p$ -value  $< 0.01$ .
LastFM	LastFM*	Yelp*
SR@15	AT	hDCG	SR@15	AT	hDCG	SR@15	AT	hDCG
Abs Greedy	0.222	13.48	0.073	0.635	8.66	0.267	0.189	13.43	0.089
Max Entropy	0.283	13.91	0.083	0.669	9.33	0.269	0.398	13.42	0.121
CRM	0.325	13.75	0.092	0.580	10.79	0.224	0.177	13.69	0.070
EAR	0.429	12.88	0.136	0.595	10.51	0.230	0.182	13.63	0.079
SCPR	0.465	12.86	0.139	0.709	8.43	0.317	0.489	12.62	0.159
FPAN	0.630	10.16	0.224	0.667	7.82	0.407	0.236	12.77	0.116
UNICORN	0.535	11.82	0.175	0.788	7.58	0.349	0.520	11.31	0.203
CRSIRL	0.772†	10.12†	0.231†	0.913†	6.79†	0.431†	0.622†	10.61†	0.228†
Table 3: Abalation study of different components of CRSIRL.
LastFM	LastFM*	Yelp*
SR@15	AT	hDCG	SR@15	AT	hDCG	SR@15	AT	hDCG
PG	0.724	10.42	0.217	0.882	7.41	0.401	0.598	10.95	0.186
¬HRS	0.732	10.58	0.219	0.898	7.16	0.401	0.602	11.21	0.196
¬RPM	0.754	10.14	0.227	0.904	6.89	0.415	0.606	10.81	0.213
MTL	0.768	10.09	0.224	0.908	6.92	0.426	0.613	10.73	0.207
CRSIRL	0.772	10.12	0.231	0.913	6.79	0.431	0.622	10.61	0.228
where  $T$  and  $K$  represent the number of conversation turns and recommended items in each turn,  $r(t,k)$  denotes the relevance of the results at the turn  $t$  and position  $k$ . Intuitively, successful sessions with fewer turns are preferable for CRS. Also, the target item is expected to be ranked higher on the recommendation list at the success turn. We report  $hDCG@(15,10)$  by default.
Training details. All datasets are split by 7:1.5:1.5 ratio for training, validation and testing. We used the Transformer-based state encoder proposed in [Chu et al., 2023]. We adopt TransE [Bordes et al., 2013] to pretrain the node embeddings on the training set, and use the user simulator described before for online policy learning on the validation set. We first pretrain the policy with only extrinsic reward using policy gradient and then apply CRSIRL to fine-tune the pretrained policy. The learning rates in the inner and outer loop are searched from  $\{1e^{-5}, 5e^{-5}, 1e^{-4}\}$  with Adam optimizer. The coefficient of intrinsic reward  $\lambda$  is searched from  $\{0.05, 0.1, 0.5, 1.0\}$ . The discount factor  $\gamma$  is set to 0.999. All experiments are run on an NVIDIA Geforce RTX 3080Ti GPU with 12 GB memory. RL-based baselines rely on handcrafted rewards, we follow Lei et al. [2020a] to set (1)  $r_{\mathrm{rec\_succ}} = 1$  for successful recommendation; (2)  $r_{\mathrm{rec\_fail}} = -0.1$  for failed recommendation; (3)  $r_{\mathrm{ask\_succ}} = 0.1$  when the inquired attribute is confirmed by the user; (4)  $r_{\mathrm{rec\_fail}} = -0.1$  when the inquired attribute is dismissed by the user; (5)  $r_{\mathrm{quit}} = -0.3$  when the user quits the conversation without a successful recommendation. We set the maximum turn  $T$  as 15 and the size  $K$  of the recommendation list as 10. We provide more implementation details in the supplementary material.
5.2 Results & Analysis
We present the main results in Table 2. We can clearly observe the CRSIRL outperformed all baselines with a large margin. Both FPAN and EAR are policy gradient based methods, but they pretrain their policies using conversation history generated by a rule-based strategy via supervised learning. This training approach biases policies towards pre-set rules, limiting the performance of policy learning on datasets with larger action spaces (like LastFM* and Yelp*), where more exploration is necessary. SCPR and UNICORN have relatively stable performance on all the datasets. Our CRSIRL outperforms all baselines significantly with its learned intrinsic rewards. Rather than arbitrarily assigning the reward values, we dynamically optimize them in CRSIRL. Any action that contributes to a final successful recommendation should receive credit and thus be promoted by policy learning, regardless of whether it involves a rejected attribute or a failed recommendation.
Contributions of each component in CRSIRL. We evaluate different variants of CRSIRL to study the contributions of each proposed component. Firstly, we disable the fine-tuning with CRSIRL and directly report the results after the policy gradient pretraining with only extrinsic rewards, denoted as PG. Secondly, we remove HRS and RPM to evaluate their individual effectiveness. In both variants, the outer loop degenerates to a single objective optimization problem. Finally, we conduct an experiment where instead of updating the objectives with MOO, we treat the two objectives as distinct tasks and assign them equal weights, a process referred to as Multi-Task Learning (MTL).
We present the results in Table 3. Interestingly, we observe that directly optimizing the extrinsic rewards with policy gradient already outperformed most of baselines. It is worth noting that PG uses sparser rewards than other baselines in Table 2 with manually-defined rewards for intermediate actions. However, the exploratory behavior of PG enables it to outperform these baselines. We can observe that without HRS, the AT metric degenerated on all three datasets. HRS prefers actions which can increase the rank of the target item, which is the most direct metric of action utilities in CRS. Even though the asked questions could still be helpful without HRS, HRS provides explicit hints about how to ask the most useful questions, leading to a smaller AT. Besides, the performance decreases after removing RPM, which finds actions leading to successful recommendations by comparing successful and failed trajectories. Lastly, MTL shows a significant improvement compared to PG, and it occasionally outperforms CRSIRL (e.g., AT on LastFM). However, MTL has difficulty balancing the two objectives, generally resulting in worse performance than CRSIRL.
Analysis of  $\lambda$ . We use the hyper-parameter  $\lambda$  to control the influence of intrinsic rewards. It is important to study how it affects the policy learning. In this experiment, we evaluate CRSIRL with  $\lambda = \{0.05, 0.1, 0.5, 1.0\}$  on the LastFM dataset. The result is shown in Figure 3. We can clearly observe that the performance peaks when an appropriate  $\lambda$  is chosen. With a small  $\lambda$ , the intrinsic reward is not strong enough to affect the final performance. However, a large value of  $\lambda$  can unfortunately impair performance. This is because the estimation errors in the intrinsic reward could overwhelm the extrinsic reward. Even
Figure 3: Performance with different  $\lambda$ .
though it is sparse, the extrinsic reward can help calibrate the intrinsic reward.
Additionally, we performed a qualitative study to analyze the learned intrinsic rewards of CRSIRL (shown in Figure 4) on the LastFM dataset. The natural language questions and user responses are generated by predefined templates. We observe that the intrinsic rewards depend not only on whether the user accepts or rejects the action, but also on how well the action contributes to the final recommendation. Even though the user accepts pop, the intrinsic reward for this action remains negative. This is because pop is a very general attribute and contributes little to modeling the user's preference. Conversely, although vocalist is rejected by the user, it still carries a small positive value as it aids in identifying the target artist. Finally, Indie and Punk are two attributes that are accepted and best describe the target artist, Franz Ferdinand $^2$  (a band known for indie rock and post-punk revival). Consequently, they carry relatively large positive intrinsic rewards. This case shows the CRSIRL can provide more fine-grained reward signals, leading to better final performance.
Figure 4: Conversations generated by CRSIRL. The values of the learned intrinsic rewards are marked in red.
$^{2}$ https://en.wikipedia.org/wiki/Franz_Ferdinand_(band)
In this paper, we study an important but largely under-explored problem in CRS, i.e., reward function design. We present a principled solution for reward learning for CRS and formulate an online algorithm to learn intrinsic rewards via bi-level multi-objective optimization. In the inner loop of CRSIRL, we optimize the policy with the learned intrinsic reward function. And in the outer loop, we optimize the intrinsic reward function to satisfy two criteria designed for CRS: maximizing success rate and minimizing number of turns in conversations. The results on three CRS benchmarks demonstrated the effectiveness of learned intrinsic rewards.
CRSIRL sheds light on learning reward functions to improve CRS. Currently, we consider two directly quantifiable objectives for CRS, i.e., success rate and conversation length. Other perspectives, such as user satisfaction [Liang et al., 2006] and fairness [Lin et al., 2022], are worth to be investigated and embedded into reward learning. Moreover, beyond the template-based conversation generation, it is important to integrate CRSIRL with advanced natural language-based conversational agents, such as [Touvron et al., 2023, Zhang et al., 2023], to learn reward functions that satisfy multiple objectives favored by humans during natural language driven interactions, such as conversational persuasiveness, cohesiveness and explainability [Moon et al., 2019].
We thank the anonymous reviewers for their insightful comments. This work was partially supported by NSF IIS-2007492, IIS-2128019 and NSF IIS-1838615.
Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3(Nov):397-422, 2002.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26, 2013.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
Qibin Chen, Junyang Lin, Yichang Zhang, Ming Ding, Yukuo Cen, Hongxia Yang, and Jie Tang. Towards knowledge-based recommender dialog system. arXiv preprint arXiv:1908.05391, 2019.
Konstantina Christakopoulou, Filip Radlinski, and Katja Hofmann. Towards conversational recommender systems. In Proceedings of the 22nd ACM SIGKDD, pages 815-824, 2016.
Konstantina Christakopoulou, Alex Beutel, Rui Li, Sagar Jain, and Ed H Chi. Q&R: A two-stage approach toward interactive recommendation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 139-148, 2018.
Zhendong Chu and Hongning Wang. Meta-reinforcement learning via exploratory task clustering. arXiv preprint arXiv:2302.07958, 2023.
Zhendong Chu, Hongning Wang, Yun Xiao, Bo Long, and Lingfei Wu. Meta policy learning for cold-start conversational recommendation. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, pages 222-230, 2023.
Kalyanmoy Deb and Kalyanmoy Deb. Multi-objective optimization. In Search methodologies: Introductory tutorials in optimization and decision support techniques, pages 403-449. Springer, 2013.
Yang Deng, Yaliang Li, Fei Sun, Bolin Ding, and Wai Lam. Unified conversational recommendation policy learning via graph-based reinforcement learning. arXiv preprint arXiv:2105.09710, 2021.
Jean-Antoine Désideri. Multiple-gradient descent algorithm (mgda) for multiobjective optimization. Comptes Rendus Mathematique, 350(5-6):313-318, 2012.