SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhendong Chu
University of Virginia  
zc9uy@virginia.edu  
Charlottesville, VA, USA
Nan Wang*
Netflix Inc.  
nanw@netflix.com  
Los Gatos, CA, USA
Hongning Wang
University of Virginia hw5x@virginia.edu Charlottesville, VA, USA
Conversational Recommender Systems (CRS) actively elicit user preferences to generate adaptive recommendations. Mainstream reinforcement learning-based CRS solutions heavily rely on handcrafted reward functions, which may not be aligned with user intent in CRS tasks. Therefore, the design of task-specific rewards is critical to facilitate CRS policy learning, which remains largely under-explored in the literature. In this work, we propose a novel approach to address this challenge by learning intrinsic rewards from interactions with users. Specifically, we formulate intrinsic reward learning as a multi-objective bi-level optimization problem. The inner level optimizes the CRS policy augmented by the learned intrinsic rewards, while the outer level drives the intrinsic rewards to optimize two CRS-specific objectives: maximizing the success rate and minimizing the number of turns to reach a successful recommendation in conversations. To evaluate the effectiveness of our approach, we conduct extensive experiments on three public CRS benchmarks. The results show that our algorithm significantly improves CRS performance by exploiting informative learned intrinsic rewards.
Conversational recommender systems (CRS) leverage interactive conversations to delineate a user's preferences [Zhang et al., 2018, Lei et al., 2020a, Chu et al., 2023]. The conversations revolve around questions aimed at discerning users' preferences on specific item attributes (e.g., music genres). Through an interactive process of questions and answers, a profile about a user's intended item can be depicted. Numerous CRS formulations have been proposed [Chen et al., 2019, Christakopoulou et al., 2018, 2016]. In this work, we investigate a prevalent CRS setting known as the multi-round conversational recommendation [Lei et al., 2020a, Deng et al., 2021], where a CRS agent can ask a question or recommend an item in consecutive rounds of conversations. The conversation continues until the user accepts the recommendation (indicating a successful conversation) or quits the conversation (considered as a failed conversation).
CRS fundamentally embodies a sequential decision making problem, for which numerous reinforcement learning (RL)-based solutions have been proposed [Lei et al., 2020a, Chu et al., 2023]. However, as the users only provide textual or binary responses (e.g., accepting or rejecting the inquired attributes), existing RL-based solutions heavily rely on heuristic reward functions that are manually defined to train CRS policies. These reward functions, such as promoting attributes accepted by a user and penalizing those rejected, may not accurately reflect user intent due to their heuristic nature. This becomes problematic since the effectiveness of CRS policy learning largely depends on the quality of pre-defined reward function – an inadequately designed reward function can lead to solutions that
*Work was done while at the University of Virginia.
significant deviates from optimality. Additionally, these arbitrary reward functions can inadvertently distort the modeling of conversation states, influencing the subsequent actions taken by the RL agent.
Arguably, an effective reward function should promote actions that lead to more precise modeling of users' preferences. As a result, different attributes or items, including those rejected, can each uniquely contribute to user preference modeling. As an example illustrated in Figure 1, even though Heavy metal rock is rejected by the user, it still, to certain extent, contributes to identifying the target item, Hey Jude. However, existing handcrafted heuristic reward functions fall short in delivering information at this granularity, as they assign uniform rewards to all accepted or rejected actions. This motivates us to learn a reward function that enables more fine-grained policy learning.
Instead of manually define reward functions, we introduce a principled approach to reward learning for CRS, where we learn an intrinsic reward for each action taken by the agent utilizing the optimal rewards framework [Singh et al., 2010]. This framework delineates the optimal intrinsic reward function as the one that, when employed by an RL agent, fosters behaviors that optimize the task-specific or extrinsic rewards – in the case of CRS, successful recommendations.
Two notable technical challenges stand out when learning intrinsic rewards for CRS. First, explicit extrinsic rewards in CRS are extremely sparse, which com
Figure 1: Motivating example of intrinsic reward learning.
plicates the intrinsic reward learning. Despite that the agent interacts with the user in each round, the only clear extrinsic reward signal, which is whether the overall conversation is successful or not, is only revealed from the user at the conclusion of the conversation. The significance of each accepted or rejected attribute/item prior to the conversation's ends remain ambiguous. For instance, an inquired attribute that is rejected by the user does not necessarily imply a negative reward for policy learning, as it can signify what the user is not looking for. Second, the assessment of CRS is multi-dimensional, entailing various factors that contribute to the overall effectiveness and user experience, such as recommendation quality and user effort. On the one hand, asking more questions may be necessary to accurately profile user preferences to facilitate a successful recommendation. On the other hand, reducing user effort in conversations (i.e., fewer conversation turns) is essential to ensure users' engagement and maintain their satisfaction. Balancing these factors is a delicate task.
To tackle the challenges for improving CRS from a reward learning perspective, we develop an online algorithm for learning intrinsic reward functions via multi-objective bi-level optimization. We name the proposed solution CRSIRL, meaning CRS with Intrinsic Reward Learning. In the inner loop of CRSIRL, the policy is optimized with the learned intrinsic reward function. In the outer loop, the intrinsic reward function is updated to satisfy two specific criteria designed for CRS. The first criterion aims to maximize the sparse extrinsic reward, augmented by a reward shaping strategy to encourage actions that promote the target item as quickly as possible. The second criterion involves tailoring the learnt reward function to promote successful trajectories over the failed ones. The results of our extensive experiments demonstrate that CRSIRL not only improves the success rate of CRS but also achieves it with shorter conversations.
Conversational Recommender Systems. Christakopoulou et al. [2016] pioneered the concept of Conversational Recommender Systems (CRS). Their approach primarily focused on determining which items to solicit feedback on and applied off-the-shelf metrics such as the upper confidence bound [Auer, 2002] for this purpose. This laid the groundwork for reinforcement learning (RL) based methods, which have recently become the prevalent solutions for CRS. For example, Sun and Zhang [2018] developed a policy network to decide whether to recommend an item or inquire about an item attribute at each conversation turn. However, these initial studies terminated the conversation upon
making a recommendation, regardless of user acceptance. Lei et al. [2020a] studied multi-round conversational recommendation, where CRS can ask a question or recommend an item in multiple rounds before the user accepts the recommendation or quits. This is also the setting of our work in this paper. To better address multi-round CRS, Lei et al. [2020b] leveraged knowledge graphs to select more relevant attributes to ask across turns. Xu et al. [2021] extended [Lei et al., 2020a] by revising user embeddings dynamically based on users' feedback on attributes and items. And Deng et al. [2021] unified the question selection module and the recommendation module in an RL-based CRS solution. However, all the aforementioned works depend on heuristically crafted reward functions, which may lead policies to deviate from the optimal solution. In this work, we propose to learn intrinsic rewards which can maximize the recommendation performance.
Intrinsic Reward Learning in Reinforcement Learning. Intrinsic reward learning has emerged as a promising approach to enhance the performance and efficiency of reinforcement learning algorithms. Singh et al. [2010] introduced the Optimal Reward Framework which aims to find a good reward function that allows agents to solve a distribution of tasks using exhaustive search. Pathak et al. [2017] introduced the concept of curiosity-driven intrinsic rewards, where the agent is rewarded for actions that lead to novel states, improving its ability to explore complex environments. Zheng et al. [2018] proposed a meta-gradient method named LIRPG to learn intrinsic rewards via a bi-level optimization framework. Zheng et al. [2020] extended LIRPG by learning intrinsic rewards on a distribution of tasks. Liu et al. [2022] developed another meta-gradient method to learn intrinsic rewards from trajectory preferences. In this work, we propose a novel intrinsic reward learning framework designed for CRS, where we learn intrinsic reward functions to satisfy multiple CRS-sepfcic objectives from users' extremely sparse explicit reward feedback.
In this section, we define the notations to be used in our technical discussions and some basic notions in multi-objective optimization.
Similar to traditional recommender systems, CRS serves a set of users  $\mathcal{U}$  with a set of items  $\nu$ ; and we denote a specific user as  $u$  and an item as  $v$ . Each item  $v$  is associated with a set of predefined attributes  $\mathcal{P}_v$ . Attributes describe basic properties of the items, such as genres in movie recommendations and cuisine type in restaurant recommendations.
We formulate the CRS problem using a Markov decision process (MDP) [Deng et al., 2021, Lei et al., 2020b, Chu and Wang, 2023], which can be fully described by a tuple  $(S, \mathcal{A}, \mathcal{T}, \mathcal{R})$ .  $S$  denotes the state space, which summarizes the conversation between the system and user so far.  $\mathcal{A}$  denotes the action space for the system, which includes recommending a particular item or asking for feedback on a specific attribute.  $\mathcal{T}: S \times \mathcal{A} \to S$  is the state transition function, and  $\mathcal{R}: S \times \mathcal{A} \to \mathbb{R}$  is a reward function.
With this formulation, a conversation in CRS can be represented as  $d = \{(a_1, r_1), \ldots, (a_T, r_T)\}$ , where  $T$  is the maximum number of allowed turns. A conversation (or an episode in the language of RL, which we will use exchangeable) terminates when: (1) the user accepts the recommended item; or (2) the CRS agent runs out of maximum number of allowed turns. At each time step  $t$ , the CRS agent, which follows a policy  $\pi_{\theta}(a_t | s_t)$  parameterized by  $\theta$ , selects an action  $a_t$  based on the current state  $s_t$ . The training objective of a CRS policy is to maximize the expected cumulative rewards over the set of observed episodes  $\mathcal{D}$ , i.e., minimizing the loss
$$
\mathcal {L} (\pi) = - \underset {d \sim P (\mathcal {D})} {\mathbb {E}} \left[ \sum_ {t = 0} ^ {T} R _ {t} \right], \tag {1}
$$
where  $R_{t} = \sum_{t^{\prime} = t}^{T}\gamma^{T - t^{\prime}}r(a_{t})$  is the accumulated reward from turn  $t$  to the final turn  $T$ , and  $\gamma \in [0,1]$  is a discount factor to emphasize rewards collected in a near term.
Instead of using handcrafted reward functions  $\mathcal{R}$  as in previous works [Lei et al., 2020a, Deng et al., 2021, Chu et al., 2023], we learn an intrinsic reward function defined as  $r_{\phi}^{in}(s,a)$  parameterized by  $\phi$  to enhance CRS policy learning. In this context, the original CRS-specific reward is referred to
Figure 2: Overview of CRSIRL, which consists of two modules, a policy parameterized by  $\theta$  and an intrinsic reward function parameterized by  $\phi$ . The optimization of CRSIRL has two levels. In the inner level, a policy is trained to maximize the return defined by both intrinsic and extrinsic rewards. In the outer level, the intrinsic reward function is trained to optimize two CRS-specific objectives realized by the learnt policy's behaviors.
as the extrinsic reward, denoted as  $r^{ex}(s,a)$ . The extrinsic reward is inherently sparse, as the only discernible and useful reward signal is the success or failure of an episode, with the intermediate actions' contributions remaining ambiguous. We assign a positive extrinsic reward at the conclusion of a successful episode and a negative reward otherwise. All intermediate actions are assigned a zero extrinsic reward.
3.2 Multi-Objective Optimization
We utilize multi-objective optimization (MOO) to achieve the multi-dimensional goal of CRS, i.e., maximizing the success rate and reducing the length of conversations. MOO aims to simultaneously optimize multiple objectives, possibly conflicting ones. This results in a trade-off among objectives, making the CRS problem more complex and challenging to solve. In these cases, the Pareto-optimal solutions represent different optimal trade-offs between the objectives [Deb and Deb, 2013].
Consider  $M$  objective functions  $\{\mathcal{L}^1,\dots,\mathcal{L}^M\}$ , a model parameterized by  $\theta$  is optimized towards them. We specify the multi-objective optimization using a vector valued loss  $\mathbb{L}$
$$
\min  _ {\theta} \mathbb {L} (\theta) = \min  _ {\theta} \left(\mathcal {L} ^ {1} (\theta), \dots , \mathcal {L} ^ {M} (\theta)\right) ^ {\top} \tag {2}
$$
The goal of multi-objective optimization is achieving Pareto optimality.
Definition 1 (Pareto optimality).
(a) A solution  $\theta$  dominates a solution  $\bar{\theta}$  if  $\mathcal{L}^t (\theta)\leq \mathcal{L}^t (\bar{\theta})$  under all objectives and  $\mathbb{L}(\theta)\neq \mathbb{L}(\bar{\theta})$
(b) A solution  $\theta^{*}$  is called Pareto optimal if there exists no solution  $\theta$  that dominates  $\theta^{*}$ .
The set of Pareto optimal solutions is called the Pareto front  $\mathcal{P}_\theta$
4 Multi-Objective Intrinsic Reward Learning for CRS
In this section, we elaborate on the structure of CRSIRL, as illustrated in Figure 2. CRSIRL operates on two tiers of optimization: the inner optimization, which improves the policy using both the intrinsic reward and the learned intrinsic reward function, and the outer optimization, which refines the intrinsic reward function based on the policy assessment derived from the inner optimization. Given the absence of supervision for the intrinsic reward function, we establish the relationship between it and the refined policy via gradient descent in the inner optimization. Specifically, we compute meta-gradient for the intrinsic reward function using chain rule in the outer optimization. In the outer optimization, we design a point-wise objective striving to enhance extrinsic rewards in the learnt intrinsic reward function, through the use of hindsight reward shaping (HRS). This objective aids in pinpointing pivotal actions that significantly improve the target item's ranking, thereby shortening the conversation length. In parallel, we introduce a pair-wise objective that favors successful trajectories over unsuccessful ones, which assists in identifying actions that result in
preferred conversations. This objective is named as recommendation preference matching (RPM). Finally, we introduce a holistic multi-objective bi-level optimization framework that optimizes intrinsic rewards to meet both objectives.
4.1 Hindsight Reward Shaping
As we discussed before, the extrinsic reward is extremely sparse in CRS. The only clear and informative signal from the extrinsic reward is whether the conversation is successful, making it hard to judge the progress of user preference elicitation during the conversation. Reward shaping, as proposed by Ng et al. [1999], serves as a valuable tool for incorporating task-specific knowledge to estimate the reward function. We leverage reward shaping within the outer loop of our model to imbue the process of intrinsic reward learning with more nuanced, task-specific guidance. We use the following hindsight reward shaping to augment the extrinsic reward,
$$
\tilde {r} ^ {e x} \left(s _ {t}, a _ {t}\right) = r ^ {e x} \left(s _ {t}, a _ {t}\right) + \gamma w \left(s _ {t + 1}, v\right) - w \left(s _ {t}, v\right), \tag {3}
$$
where  $w$  is a scoring function,  $v$  is the target item and  $\gamma$  is a discount factor.  $\tilde{r}^{ex}(s_t, a_t)$  encourages actions which promote the target item. In turn, it helps shorten the conversation length. In our experiments, we use  $w = \log(\rho(s_t, v) + 1)$ , where  $\rho(s_t, v)$  is the rank of target item  $v$  under state  $s_t$ .
Lemma 1. Consider any reward shaping function  $\mathcal{F}:S\times \mathcal{A}\times S\to \mathbb{R}$ , we say  $\mathcal{F}$  is a potential-based reward shaping function (PBRS) if there exists a real-valued function  $\Phi :S\rightarrow \mathbb{R}$  satisfying,
$$
\mathcal {F} (s, a, s ^ {\prime}) = \gamma \Phi \left(s ^ {\prime}\right) - \Phi (s), \tag {4}
$$
Then  $\mathcal{F}$  being PBRS is a necessary and sufficient condition for it to guarantee the consistency of the optimal policy, i.e., the optimal policy of  $(\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R} + \mathcal{F})$  is the same as  $(\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R})$ .
The proof is based on [Ng et al., 1999] and omitted in this paper. By matching the form of Eq.(4) and Eq.(3), we can conclude the hindsight reward shaping satisfies the PBRS condition, and thus the optimal policy is consistent. The resulted objective induced by HRS is
$$
\mathcal {L} ^ {e x} (\theta) = - \mathbb {E} \left[ \sum_ {t = 0} ^ {T} \tilde {R} _ {t} ^ {e x} \right], \tag {5}
$$
where  $\tilde{R}_t^{ex} = \sum_{t' = t}^T\gamma^{T - t'}\tilde{r}_{t'}^{ex}$ . Note that the information of target item is unknown beforehand, we can only use HRS after the target item is hit, which is why we call it *hindsight*. Otherwise HRS degenerates to the original extrinsic reward.
4.2 Recommendation Preference Matching
Even though the contributions of intermediate actions to a conversation are undefined in the extrinsic reward, it is still feasible to discern valuable intermediate actions that could potentially lead to a successful conversation, and the learned intrinsic reward should help us identify them. We realize this by contrasting successful and failed episodes by the learnt intrinsic reward: a preferred episode should have a higher likelihood under the optimal policy, comparing to a less preferred one; and this optimal policy should be achieved by the correct reward function. Given a policy  $\pi_{\theta}$ , the probability of conversation  $\tau^0$  is preferred over  $\tau^1$  is computed based on the likelihood of the trajectories,
$$
P _ {\theta} \left[ \tau^ {0} \succ \tau^ {1} \right] = \frac {\exp \sum_ {t \in \tau^ {0}} \log \pi_ {\theta} \left(a _ {t} \mid s _ {t}\right)}{\exp \sum_ {t \in \tau^ {0}} \log \pi_ {\theta} \left(a _ {t} \mid s _ {t}\right) + \exp \sum_ {t \in \tau^ {1}} \log \pi_ {\theta} \left(a _ {t} \mid s _ {t}\right)}, \tag {6}
$$
Assume  $\tau^0$  is preferred over  $\tau^1$ , the resulting loss function is given by,
$$
\mathcal {L} ^ {p} (\theta) = - \mathbb {E} \left[ \sum_ {\tau^ {0} \succ \tau^ {1}} \log P _ {\theta} \left[ \tau^ {0} \succ \tau^ {1} \right] \right], \tag {7}
$$
where  $\tau^0, \tau^1 \in \mathcal{B}$  are sampled from a buffer storing past trajectories. This follows the Bradley-Terry model [Bradley and Terry, 1952] for estimating score functions from pairwise preferences. In the context of CRS, the preference is naturally defined by whether the recommendation is successful or not; and among successful recommendations, we prefer the one shorter. We truncate the failed trajectory to match the length of the successful trajectory.
4.3 Multi-Objective Bi-Level Optimization
The intrinsic reward function is expected to lead to a policy satisfying the above two objectives. This translates to a bi-level optimization procedure for policy learning: first update the policy with learned intrinsic rewards, and then improve the intrinsic rewards to help the resulting policy better satisfy the above two objectives. More formally, we define,
$$
\min  _ {\phi} \mathbb {L} (\theta^ {\prime}),
$$
$$
\begin{array}{l} \phi \\ \text {s . t .} \theta^ {\prime} = \arg \min  _ {\theta} \mathcal {L} ^ {e x + i n} (\theta , \phi). \end{array} \tag {8}
$$
where  $\mathbb{L}(\theta') = \left(\mathcal{L}^{ex}(\theta'), \mathcal{L}^p(\theta')\right)$  and  $\mathcal{L}^{ex + in}(\theta, \phi)$  is the negative cumulative reward calculated with weighted sum  $r^{ex} + \lambda r_{\phi}^{in}$ ,  $\lambda$  is a hyper-parameter to balance two rewards. In the inner loop, we optimize the policy with both the extrinsic reward and the learned intrinsic reward function. In the outer loop, we optimize the intrinsic reward function to minimize the vector value loss. To derive the gradients for optimization, we first build the connection between  $\theta$  and  $\phi$  in the inner loop, and then derive the gradients on  $\phi$  in the outer loop.
Inner Loop: Optimizing  $\theta$ , building the connection between  $\theta$  and  $\phi$ . We update  $\theta$  as follows,
$$
\theta^ {\prime} = \theta - \eta \cdot \nabla_ {\theta} \mathcal {L} ^ {e x + i n} (\theta , \phi), \tag {9}
$$
where  $\nabla_{\theta}\mathcal{L}^{ex + in}(\theta ,\phi)$  can be calculated by the policy gradient theorem [Sutton et al., 1999] and  $\eta$  is the learning rate used in the inner loop. In this way, the updated parameter  $\theta^{\prime}$  becomes a function of  $\phi$ . With the built connection, we are able to compute the gradient of  $\phi$  by taking the gradient of gradient on  $\theta^{\prime}$ , i.e., the meta-gradient.
Outer Loop: Optimizing  $\phi$ . In the outer loop, we optimize the vector value loss  $\mathbb{L}(\theta')$  to satisfy aforementioned two CRS-specific objectives. Even though we do not have supervision on  $\phi$ , the gradient of  $\phi$  can still be derived using the chain rule,
$$
g (\phi) = \frac {\partial \mathbb {L} \left(\theta^ {\prime}\right)}{\partial \theta^ {\prime}} \cdot \frac {\partial \theta^ {\prime}}{\partial \phi} \tag {10}
$$
Different from single objective optimization, the first part of Eq.(13) is the derivative w.r.t. the multi-objective function  $\mathbb{L}(\theta')$ . Sener and Koltun [2018] adopt the multiple gradient descent algorithm (MDGA) [Désideri, 2012] to find a Pareto stationary point for a MOO problem. We follow their approach to solve the following optimization problem,
$$
\min  _ {\alpha \cdot \in [ 0, 1 ]} \left\{\left\| \alpha \nabla_ {\theta^ {\prime}} \mathcal {L} ^ {e x} \left(\theta^ {\prime}\right) + (1 - \alpha) \cdot \nabla_ {\theta^ {\prime}} \mathcal {L} ^ {p} \left(\theta^ {\prime}\right) \right\| _ {2} ^ {2} \right\}, \tag {11}
$$
where  $\alpha$  has the following analytical solution,
$$
\alpha = \left[ \frac {\nabla_ {\theta^ {\prime}} \mathcal {L} ^ {p} \left(\theta^ {\prime}\right) - \nabla_ {\theta^ {\prime}} \mathcal {L} ^ {e x} \left(\theta^ {\prime}\right) ^ {\top} \nabla_ {\theta^ {\prime}} \mathcal {L} ^ {p} \left(\theta^ {\prime}\right)}{\| \nabla_ {\theta^ {\prime}} \mathcal {L} ^ {e x} \left(\theta^ {\prime}\right) - \nabla_ {\theta^ {\prime}} \mathcal {L} ^ {p} \left(\theta^ {\prime}\right) \|} \right] _ {+, \frac {1}{\tau}}, \tag {12}
$$
where  $[\cdot ]_{+, \frac{1}{\top}}$  represents clipping to  $[0, 1]$  as  $[a]_{+, \frac{1}{\top}} = \max(\min(a, 1), 0)$ . The resulted meta-gradient of  $\phi$  becomes,