SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiarui Jin $^{1,\dagger}$ , Xianyu Chen $^{1}$ , Fanghua Ye $^{2}$ , Mengyue Yang $^{2}$ , Yue Feng $^{2}$ , Weinan Zhang $^{1,\ddagger}$ , Yong Yu $^{1}$ , Jun Wang $^{2,\ddagger}$
$^{1}$ Shanghai Jiao Tong University,  $^{2}$ University College London
{jinjiarui97,wnzhang,xianyujun,yyu}@sjtu.edu.cn
{fanghua.ye.19,mengyue.yang.20,yue.feng.20,jun.wang}@ucl.ac.uk
Recommender systems trained on offline historical user behaviors are embracing conversational techniques to online query user preference. Unlike prior conversational recommendation approaches that systemically combine conversational and recommender parts through a reinforcement learning framework, we propose CORE, a new offline-training and online-checking paradigm that bridges a COnversational agent and R Ecommender systems via a unified uncertainty minimization framework. It can benefit any recommendation platform in a plug-and-play style. Here, CORE treats a recommender system as an offline relevance score estimator to produce an estimated relevance score for each item; while a conversational agent is regarded as an online relevance score checker to check these estimated scores in each session. We define uncertainty as the summation of unchecked relevance scores. In this regard, the conversational agent acts to minimize uncertainty via querying either attributes or items. Based on the uncertainty minimization framework, we derive the expected certainty gain of querying each attribute and item, and develop a novel online decision tree algorithm to decide what to query at each turn. We reveal that CORE can be extended to query attribute values, and we establish a new Human-AI recommendation simulator supporting both open questions of querying attributes and closed questions of querying attribute values. Experimental results on 8 industrial datasets show that CORE could be seamlessly employed on 9 popular recommendation approaches, and can consistently bring significant improvements, compared against either recently proposed reinforcement learning-based or classical statistical methods, in both hot-start and cold-start recommendation settings. We further demonstrate that our conversational agent could communicate as a human if empowered by a pre-trained large language model, e.g., gpt-3.5-turbo.
Recommender systems are powerful tools to facilitate users' information seeking [28, 38, 4, 20, 22, 21, 52, 51]; however, most prior works solely leverage offline historical data to build a recommender system. The inherent limitation of these recommendation approaches lies in their offline focus on users' historical interests, which would not always align with users' present needs. As intelligent conversational assistants (a.k.a., chat-bots) such as ChatGPT and Amazon Alexa, have entered the daily life of users, these conversational techniques bring an unprecedented opportunity to online obtain users' current preferences via conversations. This possibility has been envisioned as conversational
†Work done during Jiarui Jin's visit at University College London.
Correspondence: Weinan Zhang, Jun Wang.
Figure 1: An illustrated example of CORE, an offline-training and online-checking framework, where a recommender system operates as an offline relevance score estimator (colored in green), while a conversational agent acts as an online relevance score checker (colored in blue). Concretely, given a matrix of candidate items, as shown in (a), the recommender system could offline assign an estimated relevance score to each item, and then the conversational agent would online check these scores by querying either items or attributes, depicted in (b).
recommender systems and has inspired a series of conversational recommendation methods [26, 30, 44]. Unfortunately, all of these approaches try to model the interactions between users and systems using a reinforcement learning-based framework, which inevitably suffers from data insufficiency and deployment difficulty, because most recommendation platforms are based on supervised learning.
In this paper, we propose CORE that can bridge a Conversational agent and REcommender systems in a plug-and-play style. In our setting, a conversational agent can choose either to query (a.k.a., to recommend) an item (e.g., Hotel A) or to query an attribute (e.g., Hotel Level), and the user should provide their corresponding preference. Here, the goal of the conversational agent is to find (a.k.a., to query) an item that satisfies the user, with a minimal number of interactions.
We formulate the cooperation between a conversational agent and a recommender system into a novel offline-training and online-checking framework. Specifically, CORE treats a recommender system as an offline relevance score estimator that offline assigns a relevance score to each item, while a conversational agent is regarded as an online relevance score checker that online checks whether these estimated relevance scores could reflect the relevance between items and the user's current needs. Here, "checked items" means those items, we can certainly say that they can not satisfy the user according to already queried items and attributes. We introduce a new uncertainty metric defined as the summation of estimated relevance scores of those unchecked items. Then, the goal of our conversational agent can be formulated as minimizing the uncertainty via querying items or attributes during the interactions. To this end, we derive expected certainty gain to measure the expectation of uncertainty reduction by querying each item and attribute. Then, during each interaction, our conversational agent selects an item or an attribute with the maximum certainty gain, resulting in an online decision tree algorithm. We exemplify the above process in Figure 1.
Notice that users usually do not hold a clear picture of their preferences on some attributes (i.e., attribute IDs), e.g., what Hotel Level they need, instead, they could have a clear preference on a specific value of an attribute (i.e., attribute value), e.g., Hotel Level=5 is too expensive for a student user. Also, asking an open question of querying attributes could result in an unexpected answer, e.g., a user answers 3.5 to Hotel Level. In this regard, querying attribute values leading to closed questions (i.e., Yes or No questions) could be a better choice. We reveal that CORE could be directly applied to the above querying strategies. We also develop a new Human-AI recommendation simulator that supports both querying attributes and attribute values.
In practice, we extend CORE to handle continuous attributes and to consider the dependence among attributes. Moreover, we demonstrate that our conversational agent could straightforwardly be empowered by a pre-trained language model, e.g., gpt-3.5-turbo, to communicate as a human. Note that CORE poses no constraint on recommender systems, only requiring the estimated relevance scores. Therefore, CORE can be seamlessly applied to any recommendation platform. We conduct experiments on 8 industrial datasets (including both tabular data, sequential behavioral data and graph-structured
data) with 9 popular recommendation approaches (e.g., DeepFM [20], DIN [52]). Experimental results show that CORE can bring significant improvements in both hot-start recommendation (i.e., the recommender system is offline trained) and cold-start recommendation (i.e., the recommender system is not trained) settings. We compare CORE against recently proposed reinforcement learning based methods and classical statistical methods, and CORE could consistently show better performance.
2 Bridging Conversational Agents and Recommender Systems
Let  $\mathcal{U}$  denote a set of users,  $\mathcal{V} = \{v_{1},\dots,v_{M}\}$  be a set of  $M$  items,  $\mathcal{X} = \{x_{1},\dots,x_{N}\}$  be a set of  $N$  attributes (a.k.a., features) of items. We consider a recommender system as a mapping function, denoted as  $\Psi_{\mathrm{RE}}:\mathcal{U}\times \mathcal{V}\to \mathbb{R}$  that assigns an estimated relevance score to each item regarding a user. Then, during each online session, a conversational agent also can be formulated as a mapping function, denoted as  $\Psi_{\mathrm{C0}}:\mathcal{U}\times \mathcal{A}\rightarrow \mathbb{R}$  that chooses either an item or an attribute to query user, and  $\mathcal{A} = \mathcal{V}\cup \mathcal{X}$  denotes the action space of the conversational agent. For convenience, we focus on the most favorite item of the given user in each session. In other words, our goal is to identify the item that will receive the first click when all items are equally presented. We call this item as target item.
For this purpose,  $\Psi_{\mathrm{RE}}(\cdot)$  acts as an offline estimator to produce an estimated relevance distribution of items regarding each user through offline training on previous behavioral data; while  $\Psi_{\mathrm{C0}}(\cdot)$  operates as an online checker to check whether these estimated scores fit the user's current needs through online interactions. Here, "checked items" denote those items that can not be the target item according to queried items and attributes. For example, as Figure 1 illustrates, after querying Breakfast Service, we have items Hotel C and Hotel D checked. We introduce uncertainty as the summation of estimated relevance scores of unchecked items. Formally, we have:
Definition 1 (Uncertainty and Certainty Gain). For the  $k$ -th turn, we define uncertainty, denoted as  $\mathrm{U}_k$ , to measure how many estimated relevance scores are still unchecked, i.e.,
$$
\mathrm {U} _ {k} := \operatorname {S U M} \left(\left\{\Psi_ {\mathrm {R E}} \left(v _ {m}\right) \mid v _ {m} \in \mathcal {V} _ {k} \right\}\right), \tag {1}
$$
where  $\Psi_{\mathrm{RE}}(v_m)^1$  outputs the estimated relevance score for item  $v_m$ ,  $\mathcal{V}_k$  is the set of all the unchecked items after  $k$  interactions, and  $\mathcal{V}_k$  is initialized as  $\mathcal{V}_0 = \mathcal{V}$ . Then, the certainty gain of  $k$ -th interaction is defined as  $\Delta \mathrm{U}_k \coloneqq \mathrm{U}_{k-1} - \mathrm{U}_k$ , i.e., how many relevance scores are checked at the  $k$ -th turn. Since our goal is to find the target item, if  $\Psi_{\mathbb{C0}}(\cdot)$  successfully finds one at the  $k$ -th turn, then we set all the items checked, namely  $\mathrm{U}_k = 0$ .
In this regard, our conversational agent is to minimize  $\mathsf{U}_k$  at each  $k$ -th turn via removing those checked items from  $\nu_{k}$ . Considering that online updating  $\Psi_{\mathrm{RE}}(\cdot)$  is infeasible in practice due to the high latency and computation costs, the objective of  $\Psi_{\mathbb{C0}}(\cdot)$  can be expressed as:
$$
\min  _ {\Psi_ {\mathrm {R E}} ^ {*}} K, \text {s . t .}, \mathrm {U} _ {K} = 0, \tag {2}
$$
where  $K$  is the number of turns, and  $\Psi_{\mathrm{RE}}^*$  means that the recommender system is frozen. To this end, the design of our conversational agent could be organized as an uncertainty minimization problem.
2.2 Comparisons to Previous Work
Bridging conversational techniques and recommender systems has become an appealing solution to model the dynamic preference and weak explainability problems in recommendation task [15, 26], where the core sub-task is to dynamically select attributes to query and make recommendations upon the corresponding answers. Along this line, the main branch of previous studies is to combine the conversational models and the recommendation models from a systematic perspective. Namely conversational and recommender models are treated and learned as two individual modules [30, 2, 44, 48] in the system. The system is developed from a single-turn conversational recommender system [5, 6, 47] to a multiple-turn one [49]. To decide when to query attributes and when to make recommendations (i.e., query items), recent papers [30, 31, 44, 30] develop reinforcement
<sup>1</sup>In this paper, as our conversational agent only faces one user  $u \in \mathcal{U}$  in each session, we omit the input  $u$  in mapping functions  $\Psi(\cdot)$ s for simplicity.
learning-based solutions, which are innately suffering from insufficient usage of labeled data and high complexity costs of deployment. However, reinforcement learning approaches often demand a substantial number of training samples, a condition referred to as data insufficiency, and require operating within a relatively compact action space. Instead, our conversational agent can be regarded as a generalist agent that can query either items or attributes. In addition, our querying strategy is derived based on the uncertainty minimization framework, which only requires estimated relevance scores from the recommender system. Hence, CORE can be straightforwardly applied to any supervised learning-based recommendation platform, in a plug-and-play way, free of meticulous reward function design in reinforcement learning methods. CORE is more friendly to open-world scenarios especially when training data is limited.
We present the connections to other previous work (e.g., decision tree algorithms) in Appendix A5.
3 Making the Conversational Agent a Good Uncertainty Optimizer
3.1 Building an Online Decision Tree
As described in Section 2.1, the aim of our conversational agent is to effectively reduce uncertainty via querying either items or attributes. The core challenge is how to decide which item or attribute to query. To this end, we begin by introducing expected certainty gain to measure the expectation of how much uncertainty could be eliminated by querying each item and each attribute. Then, we can choose an item or an attribute with the maximum expected certainty gain to query.
Formally, let  $\mathcal{X}_k$  denote the set of unchecked attributes after  $k$  interactions. Then, for each  $k$ -th turn, we define  $a_{\mathrm{query}}$  as an item or an attribute to query, which is computed following:
$$
a _ {\text {q u e r y}} = \underset {a \in \mathcal {V} _ {k - 1} \cup \mathcal {X} _ {k - 1}} {\arg \max } \Psi_ {\mathrm {C G}} (\text {q u e r y} (a)), \tag {3}
$$
where  $\Psi_{\mathbb{C}\mathbb{G}}(\cdot)$  denotes the expected certainty gain of querying  $a$ , and  $a$  can be either an unchecked item (from  $\mathcal{V}_{k-1}$ ) or an unchecked attribute (from  $\mathcal{X}_{k-1}$ ).
$\Psi_{\mathrm{CG}}(\cdot)$  for Querying an Item. We first consider the case where  $a \in \mathcal{V}_{k-1}$ . Let  $v^{*}$  denote the target item in the session. Since we only need to find one target item, therefore, if  $a \in \mathcal{V}_{k-1}$ , we can derive:
$$
\begin{array}{l} \Psi_ {\mathrm {C G}} (\text {q u e r y} (a)) = v ^ {*}) \cdot \Pr (a = v ^ {*}) + \Psi_ {\mathrm {C G}} (a \neq v ^ {*}) \cdot \Pr (a \neq v ^ {*}) \\ = \left(\sum_ {v _ {m} \in \mathcal {V} _ {k - 1}} \Psi_ {\mathrm {R E}} \left(v _ {m}\right)\right) \cdot \Pr (a = v ^ {*}) + \Psi_ {\mathrm {R E}} (a) \cdot \Pr (a \neq v ^ {*}), \tag {4} \\ \end{array}
$$
where  $a = v^{*}$  and  $a \neq v^{*}$  denote that queried  $a$  is the target item and not. If  $a = v^{*}$ , the session is done, and therefore, the certainty gain (i.e.,  $\Psi_{\mathbb{CG}}(a = v^{*})$ ) is the summation of all the relevance scores in  $\mathcal{V}_{k-1}$ . Otherwise, only  $a$  is checked, and the certainty gain (i.e.,  $\Psi_{\mathbb{CG}}(a \neq v^{*})$ ) is the relevance score of  $a$ , and we have  $\mathcal{V}_k = \mathcal{V}_{k-1} \backslash \{a\}$  and  $\mathcal{X}_k = \mathcal{X}_{k-1}$ .
Considering that  $a$  being the target item means  $a$  being the most relevant item, we leverage the user's previous behaviors to estimate the user's current preference. With relevance scores estimated by  $\Psi_{\mathrm{RE}}(\cdot)$ , we estimate  $\Pr(a = v^*)$  as:
$$
\Pr (a = v ^ {*}) = \frac {\Psi_ {\mathrm {R E}} (a)}{\operatorname {S U M} \left(\left\{\Psi_ {\mathrm {R E}} \left(v _ {m}\right) \mid v _ {m} \in \mathcal {V} _ {k - 1} \right\}\right)}, \tag {5}
$$
and  $\operatorname{Pr}(a \neq v^{*}) = 1 - \operatorname{Pr}(a \neq v^{*})$ .
$\Psi_{\mathrm{CG}}(\cdot)$  for Querying an Attribute. We then consider the case where  $a \in \mathcal{X}_{k-1}$ . For each queried attribute  $a$ , let  $\mathcal{W}_a$  denote the set of all the candidate attribute values, and let  $w_a^* \in \mathcal{W}_a$  denote the user preference on  $a$ , e.g.,  $a$  is Hotel Level,  $w_a^*$  is 3. Then, if  $a \in \mathcal{X}_{k-1}$ , we have:
$$
\Psi_ {\mathbb {C G}} (\operatorname {q u e r y} (a)) = \sum_ {w _ {a} \in \mathcal {W} _ {a}} \left(\Psi_ {\mathbb {C G}} \left(w _ {a} = w _ {a} ^ {*}\right) \cdot \Pr \left(w _ {a} = w _ {a} ^ {*}\right)\right), \tag {6}
$$
where  $w_{a} = w_{a}^{*}$  means that when querying  $a$ , the user's answer (represented by  $w_{a}^{*}$ ) is  $w_{a}$ ,  $\Psi_{\mathbb{C}\mathbb{G}}(w_{a} = w_{a}^{*})$  is the certainty gain when  $w_{a} = w_{a}^{*}$  happens, and  $\Pr(w_{a} = w_{a}^{*})$  is the probability of  $w_{a} = w_{a}^{*}$  occurring. If  $w_{a} = w_{a}^{*}$  holds, then all the unchecked items whose value of  $a$  is not equal to  $w_{a}$  should be removed from  $\mathcal{V}_{k-1}$ , as they are certainly not satisfying the user's needs.
Formally, let  $\mathcal{V}_{a_{\mathrm{value}}} = w_a$  denote the set of all the items whose value of  $a$  is equal to  $w_a$ , and let  $\mathcal{V}_{a_{\mathrm{value}} \neq w_a}$  denote the set of rest items. Then,  $\Psi_{\mathbb{C}\mathbb{G}}(w_a = w_a^*)$  can be computed as:
$$
\Psi_ {\mathrm {C G}} \left(w _ {a} = w _ {a} ^ {*}\right) = \operatorname {S U M} \left(\left\{\Psi_ {\mathrm {R E}} \left(v _ {m}\right) \mid v _ {m} \in \mathcal {V} _ {k - 1} \cap \mathcal {V} _ {a _ {\text {v a l u e}} \neq w _ {a}} \right\}\right), \tag {7}
$$
which indicates that the certainty gain, when  $w_{a}$  is the user's answer, is the summation of relevance scores of those items not matching the user preference.
To finish  $\Psi_{\mathrm{CG}}(\mathsf{query}(a))$ , we also need to estimate  $\operatorname*{Pr}(w_a = w_a^*)$ . To estimate the user preference on attribute  $a$ , we leverage the estimated relevance scores given by  $\Psi_{\mathrm{RE}}(\cdot)$  as:
$$
\Pr \left(w _ {a} = w _ {a} ^ {*}\right) = \frac {\operatorname {S U M} \left(\left\{\Psi_ {\mathrm {R E}} \left(v _ {m}\right) \mid v _ {m} \in \mathcal {V} _ {k - 1} \cap \mathcal {V} _ {\text {a v a l u e}} = w _ {a} \right\}\right)}{\operatorname {S U M} \left(\left\{\Psi_ {\mathrm {R E}} \left(v _ {m}\right) \mid v _ {m} \in \mathcal {V} _ {k - 1} \right\}\right)}. \tag {8}
$$
In this case, we remove  $\mathcal{V}_{k-1} \cap \mathcal{V}_{a_{\text{value}} \neq w_a^*}$  from  $\mathcal{V}_{k-1}$ , namely we have  $\mathcal{V}_k = \mathcal{V}_{k-1} \setminus \mathcal{V}_{a_{\text{value}} \neq w_a^*}$ . As attribute  $a$  is checked, we have  $\mathcal{X}_k = \mathcal{X}_{k-1} \setminus \{a\}$ . Here,  $w_a^*$  is provided by the user after querying  $a$ .
By combining Eqs. (4), (6), and (7), we can derive a completed form of  $\Psi_{\mathbb{CG}}(\text{query}(a))$  for  $a \in \mathcal{V}_{k-1} \cup \mathcal{X}_{k-1}$  (See Appendix A1.1 for details). Then, at each  $k$ -th turn, we can always follow Eq. (3) to obtain the next query  $a_{\text{query}}$ . As depicted in Figure 1(b), the above process results in an online decision tree, where the nodes in each layer are items and attributes to query, and the depth of the tree is the number of turns (see Appendix A4.3 for visualization of a real-world case).
3.2 From Querying Attributes to Querying Attribute Values
We note that the online decision tree introduced above is a general framework; while applying it to real-world scenarios, there should be some specific designs.
$\Psi_{\mathrm{CG}}(\cdot)$  for Querying an Attribute Value. One implicit assumption in the above online decision tree is that the user's preference on queried attribute  $a$  always falls into the set of attribute values, namely  $w_{a}^{*}\in \mathcal{W}_{a}$  holds. However, it can not always hold, due to (i) a user would not have a clear picture of an attribute, (ii) a user's answer would be different from all the candidate attribute values, e.g.,  $a$  is Hotel Level,  $w_{a}^{*} = 3.5$ , and  $\mathcal{W}_a = \{3,5\}$ , as shown in Figure 1(a). In these cases, querying attributes would not be a good choice. Hence, we propose to query attribute values instead of attribute IDs, because (i) a user is likely to hold a clear preference for a specific value of an attribute, e.g., a user would not know an actual Hotel Level of her favoring hotels, but she clearly knows she can not afford a hotel with Hotel Level=5, and (ii) since querying attribute values leads to closed questions instead of open questions, a user only needs to answer Yes or No, therefore, avoiding the user's answer to be out of the scope of all the candidate attribute values.
Formally, in this case,  $\mathcal{A} = \mathcal{W}_x\times \mathcal{X}_{k - 1}$  which indicates we need to choose a value  $w_{x}\in \mathcal{W}_{x}$  where  $x\in \mathcal{X}_{k - 1}$ . In light of this, we compute the expected certainty gain of querying attribute value  $w_{x}$  as:
$$
\Psi_ {\mathrm {C G}} (\text {q u e r y} (x) = w _ {x}) = \Psi_ {\mathrm {C G}} \left(w _ {x} = w _ {x} ^ {*}\right) \cdot \Pr \left(w _ {x} = w _ {x} ^ {*}\right) + \Psi_ {\mathrm {C G}} \left(w _ {x} \neq w _ {x} ^ {*}\right) \cdot \Pr \left(w _ {x} \neq w _ {x} ^ {*}\right), \tag {9}
$$
where  $w_{x}^{*}\in \mathcal{W}_{x}$  denotes the user preference on attribute  $x$ . Here, different from querying attributes, a user would only respond with Yes (i.e.,  $w_{x} = w_{x}^{*}$ ) or No (i.e.,  $w_{x}\neq w_{x}^{*}$ ). Therefore, we only need to estimate the certainty gain for the above two cases.  $\Psi_{\mathrm{CG}}(w_x = w_x^*)$  can be computed following Eq. (7) and  $\Psi_{\mathrm{CG}}(w_x\neq w_x^*)$  can be calculated by replacing  $\nu_{x_{\mathrm{value}}\neq w_x}$  with  $\nu_{x_{\mathrm{value}} = w_x}$ .  $\operatorname *{Pr}(w_x = w_x^*)$  is estimated in Eq. (8) and  $\operatorname *{Pr}(w_x\neq w_x^*) = 1 - \operatorname *{Pr}(w_x = w_x^*)$ . In this case, if all the values of  $x$  have been checked, we have  $\mathcal{X}_k = \mathcal{X}_{k - 1}\backslash \{x\}$ ; otherwise,  $\mathcal{X}_k = \mathcal{X}_{k - 1}$ ; and  $\mathcal{V}_k = \mathcal{V}_{k - 1}\setminus \mathcal{V}_{x_{\mathrm{value}}\neq w_x}$  if receiving Yes from the user,  $\mathcal{V}_k = \mathcal{V}_{k - 1}\setminus \mathcal{V}_{x_{\mathrm{value}} = w_x}$ , otherwise.
We reveal the connection between querying attributes (i.e., querying attribute IDs) and querying attribute values in the following proposition.
Proposition 1. For any attribute  $x \in \mathcal{X}_{k-1}$ ,  $\Psi_{\mathrm{CG}}(\mathrm{query}(x)) \geq \Psi_{\mathrm{CG}}(\mathrm{query}(x) = w_x)$  holds for all the possible  $w_x \in \mathcal{W}_x$ .
This proposition shows that if users could give a clear preference for the queried attribute and their preferred attribute value is one of the candidate attribute values, then querying attributes would be an equivalent or a better choice than querying attribute values. In other words, querying attributes and querying attribute values can not operate on the same attributes (otherwise,  $\Psi_{\mathsf{C0}}(\cdot)$  would always choose to query attributes). Therefore, we can combine querying items and querying attribute values