Title: Lending Interaction Wings to Recommender Systems with Conversational Agents
Venue: NeurIPS 2023
Accepted: yes
Keywords: Conversational Agent

Recommender System

Conversational Recommendation
Abstract: An intelligent conversational agent (a.k.a., chat-bot) could embrace conversational technologies to obtain user preferences online, to overcome inherent limitations of recommender systems trained over the offline historical user behaviors. In this paper, we propose CORE, a new offline-training and online-checking framework to plug a COnversational agent into REcommender systems. Unlike most prior conversational recommendation approaches that systemically combine conversational and recommender parts through a reinforcement learning framework, CORE bridges the conversational agent and recommender system through a unified uncertainty minimization framework, which can be easily applied to any existing recommendation approach. Concretely, CORE treats a recommender system as an offline estimator to produce an estimated relevance score for each item, while CORE regards a conversational agent as an online checker that checks these estimated scores in each online session. We define uncertainty as the sum of unchecked relevance scores. In this regard, the conversational agent acts to minimize uncertainty via querying either attributes or items. Towards uncertainty minimization, we derive the certainty gain of querying each attribute and item, and develop a novel online decision tree algorithm to decide what to query at each turn. Our theoretical analysis reveals the bound of the expected number of turns of CORE in a cold-start setting. Experimental results demonstrate that CORE can be seamlessly employed on a variety of recommendation approaches, and can consistently bring significant improvements in both hot-start and cold-start settings.

Lending Interaction Wings to Recommender Systems with Conversational Agents

Abstract

Recommender systems trained on offline historical user behaviors are embracing conversational techniques to online query user preference. Unlike prior conversational recommendation approaches that systemically combine conversational and recommender parts through a reinforcement learning framework, we propose CORE , a new offline-training and online-checking paradigm that bridges a CO nversational agent and RE commender systems via a unified uncertainty minimization framework. It can benefit any recommendation platform in a plug-and-play style. Here, CORE treats a recommender system as an offline relevance score estimator to produce an estimated relevance score for each item; while a conversational agent is regarded as an online relevance score checker to check these estimated scores in each session. We define uncertainty as the summation of unchecked relevance scores. In this regard, the conversational agent acts to minimize uncertainty via querying either attributes or items . Based on the uncertainty minimization framework, we derive the expected certainty gain of querying each attribute and item, and develop a novel online decision tree algorithm to decide what to query at each turn. We reveal that CORE can be extended to query attribute values, and we establish a new Human-AI recommendation simulator supporting both open questions of querying attributes and closed questions of querying attribute values. Experimental results on 8 industrial datasets show that CORE could be seamlessly employed on 9 popular recommendation approaches, and can consistently bring significant improvements, compared against either recently proposed reinforcement learning-based or classical statistical methods, in both hot-start and cold-start recommendation settings. We further demonstrate that our conversational agent could communicate as a human if empowered by a pre-trained large language model, e.g., gpt-3.5-turbo .

1 Introduction

Recommender systems are powerful tools to facilitate users' information seeking [28, 38, 4, 20, 22, 21, 52, 51]; however, most prior works solely leverage offline historical data to build a recommender system. The inherent limitation of these recommendation approaches lies in their offline focus on users' historical interests, which would not always align with users' present needs. As intelligent conversational assistants (a.k.a., chat-bots) such as ChatGPT and Amazon Alexa, have entered the daily life of users, these conversational techniques bring an unprecedented opportunity to online obtain users' current preferences via conversations. This possibility has been envisioned as conversational

Query Hotel A

Query Hotel C

5

3

5

3

Y

Y

N

Y

N

Y

N

N

B

C

D

A

(a) Matrix of C andidate Items with Offline

Estimated R elevance S cores

My offline estimated relevance

scores are: Hotel A is 0.1,  Hotel B

is 0.4, Hotel C  is 0.4, Hotel D is 0.1.

Do you require Breakfast S ervice?

Y es.

I recommend Hotel B for you.

Great. Thank you.

0.1

0.4

0.4

0.1

Query Breakfast S ervice

Query Hotel B

B

C

Y es

No

Y es

A

No

Query Hotel D

D

No

Cheers!

Y es

Cheers!

Y es

Cheers!

No Item

Found

No

Y es

Cheers!

No Item

Found

No

An E xample of C OR E : an Offline-training and

Online-checking F ramework

I build an online decision tree

according to your estimation.

I will offline update estimations for

Hotel A, Hotel B, Hotel C , Hotel D.

(b) Online Decision T ree to Decide What to

Query (E ither an Attribute or an Item)

recommender systems and has inspired a series of conversational recommendation methods [26, 30, 44]. Unfortunately, all of these approaches try to model the interactions between users and systems using a reinforcement learning-based framework, which inevitably suffers from data insufficiency and deployment difficulty, because most recommendation platforms are based on supervised learning.

In this paper, we propose CORE that can bridge a CO nversational agent and RE commender systems in a plug-and-play style. In our setting, a conversational agent can choose either to query (a.k.a., to recommend) an item (e.g., Hotel A ) or to query an attribute (e.g., Hotel Level ), and the user should provide their corresponding preference. Here, the goal of the conversational agent is to find (a.k.a., to query) an item that satisfies the user, with a minimal number of interactions.

We formulate the cooperation between a conversational agent and a recommender system into a novel offline-training and online-checking framework. Specifically, CORE treats a recommender system as an offline relevance score estimator that offline assigns a relevance score to each item, while a conversational agent is regarded as an online relevance score checker that online checks whether these estimated relevance scores could reflect the relevance between items and the user's current needs. Here, 'checked items' means those items, we can certainly say that they can not satisfy the user according to already queried items and attributes. We introduce a new uncertainty metric defined as the summation of estimated relevance scores of those unchecked items. Then, the goal of our conversational agent can be formulated as minimizing the uncertainty via querying items or attributes during the interactions. To this end, we derive expected certainty gain to measure the expectation of uncertainty reduction by querying each item and attribute. Then, during each interaction, our conversational agent selects an item or an attribute with the maximum certainty gain, resulting in an online decision tree algorithm. We exemplify the above process in Figure 1.

Notice that users usually do not hold a clear picture of their preferences on some attributes (i.e., attribute IDs), e.g., what Hotel Level they need, instead, they could have a clear preference on a specific value of an attribute (i.e., attribute value), e.g., Hotel Level = 5 is too expensive for a student user. Also, asking an open question of querying attributes could result in an unexpected answer, e.g., a user answers 3.5 to Hotel Level . In this regard, querying attribute values leading to closed questions (i.e., Yes or No questions) could be a better choice. We reveal that CORE could be directly applied to the above querying strategies. We also develop a new Human-AI recommendation simulator that supports both querying attributes and attribute values.

In practice, we extend CORE to handle continuous attributes and to consider the dependence among attributes. Moreover, we demonstrate that our conversational agent could straightforwardly be empowered by a pre-trained language model, e.g., gpt-3.5-turbo , to communicate as a human. Note that CORE poses no constraint on recommender systems, only requiring the estimated relevance scores. Therefore, CORE can be seamlessly applied to any recommendation platform. We conduct experiments on 8 industrial datasets (including both tabular data, sequential behavioral data and graph-structured

data) with 9 popular recommendation approaches (e.g., DeepFM [20], DIN [52]). Experimental results show that CORE can bring significant improvements in both hot-start recommendation (i.e., the recommender system is offline trained) and cold-start recommendation (i.e., the recommender system is not trained) settings. We compare CORE against recently proposed reinforcement learning based methods and classical statistical methods, and CORE could consistently show better performance.

2 Bridging Conversational Agents and Recommender Systems

2.1 Problem Formulation

Let U denote a set of users, V = { v 1 , . . . , v M } be a set of M items, X = { x 1 , . . . , x N } be a set of N attributes (a.k.a., features) of items. We consider a recommender system as a mapping function, denoted as Ψ RE : U × V → R that assigns an estimated relevance score to each item regarding a user. Then, during each online session, a conversational agent also can be formulated as a mapping function, denoted as Ψ CO : U × A → R that chooses either an item or an attribute to query user, and A = V ∪ X denotes the action space of the conversational agent. For convenience, we focus on the most favorite item of the given user in each session. In other words, our goal is to identify the item that will receive the first click when all items are equally presented. We call this item as target item .

For this purpose, Ψ RE ( · ) acts as an offline estimator to produce an estimated relevance distribution of items regarding each user through offline training on previous behavioral data; while Ψ CO ( · ) operates as an online checker to check whether these estimated scores fit the user's current needs through online interactions. Here, 'checked items' denote those items that can not be the target item according to queried items and attributes. For example, as Figure 1 illustrates, after querying Breakfast Service , we have items Hotel C and Hotel D checked. We introduce uncertainty as the summation of estimated relevance scores of unchecked items. Formally, we have:

Definition 1 ( Uncertainty and Certainty Gain ) . For the k -th turn, we define uncertainty, denoted as U k , to measure how many estimated relevance scores are still unchecked, i.e.,

where Ψ RE ( v m ) 1 outputs the estimated relevance score for item v m , V k is the set of all the unchecked items after k interactions, and V k is initialized as V 0 = V . Then, the certainty gain of k -th interaction is defined as ∆ U k := U k -1 -U k , i.e., how many relevance scores are checked at the k -th turn. Since our goal is to find the target item, if Ψ CO ( · ) successfully finds one at the k -th turn, then we set all the items checked, namely U k = 0 .

In this regard, our conversational agent is to minimize U k at each k -th turn via removing those checked items from V k . Considering that online updating Ψ RE ( · ) is infeasible in practice due to the high latency and computation costs, the objective of Ψ CO ( · ) can be expressed as:

where K is the number of turns, and Ψ ∗ RE means that the recommender system is frozen. To this end, the design of our conversational agent could be organized as an uncertainty minimization problem.

2.2 Comparisons to Previous Work

Bridging conversational techniques and recommender systems has become an appealing solution to model the dynamic preference and weak explainability problems in recommendation task [15, 26], where the core sub-task is to dynamically select attributes to query and make recommendations upon the corresponding answers. Along this line, the main branch of previous studies is to combine the conversational models and the recommendation models from a systematic perspective. Namely conversational and recommender models are treated and learned as two individual modules [30, 2, 44, 48] in the system. The system is developed from a single-turn conversational recommender system [5, 6, 47] to a multiple-turn one [49]. To decide when to query attributes and when to make recommendations (i.e., query items), recent papers [30, 31, 44, 30] develop reinforcement

learning-based solutions, which are innately suffering from insufficient usage of labeled data and high complexity costs of deployment. However, reinforcement learning approaches often demand a substantial number of training samples, a condition referred to as data insufficiency, and require operating within a relatively compact action space. Instead, our conversational agent can be regarded as a generalist agent that can query either items or attributes. In addition, our querying strategy is derived based on the uncertainty minimization framework, which only requires estimated relevance scores from the recommender system. Hence, CORE can be straightforwardly applied to any supervised learning-based recommendation platform, in a plug-and-play way, free of meticulous reward function design in reinforcement learning methods. CORE is more friendly to open-world scenarios especially when training data is limited.

We present the connections to other previous work (e.g., decision tree algorithms) in Appendix A5.

3 Making the Conversational Agent a Good Uncertainty Optimizer

3.1 Building an Online Decision Tree

As described in Section 2.1, the aim of our conversational agent is to effectively reduce uncertainty via querying either items or attributes. The core challenge is how to decide which item or attribute to query. To this end, we begin by introducing expected certainty gain to measure the expectation of how much uncertainty could be eliminated by querying each item and each attribute. Then, we can choose an item or an attribute with the maximum expected certainty gain to query.

Formally, let X k denote the set of unchecked attributes after k interactions. Then, for each k -th turn, we define a query as an item or an attribute to query, which is computed following:

where Ψ CG ( · ) denotes the expected certainty gain of querying a , and a can be either an unchecked item (from V k -1 ) or an unchecked attribute (from X k -1 ).

Ψ CG ( · ) for Querying an Item. We first consider the case where a ∈ V k -1 . Let v ∗ denote the target item in the session. Since we only need to find one target item, therefore, if a ∈ V k -1 , we can derive:

̸

̸

̸

̸

where a = v ∗ and a = v ∗ denote that queried a is the target item and not. If a = v ∗ , the session is done, and therefore, the certainty gain (i.e., Ψ CG ( a = v ∗ ) ) is the summation of all the relevance scores in V k -1 . Otherwise, only a is checked, and the certainty gain (i.e., Ψ CG ( a = v ∗ ) ) is the relevance score of a , and we have V k = V k -1 \{ a } and X k = X k -1 .

̸

Considering that a being the target item means a being the most relevant item, we leverage the user's previous behaviors to estimate the user's current preference. With relevance scores estimated by Ψ RE ( · ) , we estimate Pr ( a = v ∗ ) as:

̸

and Pr ( a = v ∗ ) = 1 -Pr ( a = v ∗ ) .

̸

Ψ CG ( · ) for Querying an Attribute. We then consider the case where a ∈ X k -1 . For each queried attribute a , let W a denote the set of all the candidate attribute values, and let w ∗ a ∈ W a denote the user preference on a , e.g., a is Hotel Level , w ∗ a is 3 . Then, if a ∈ X k -1 , we have:

where w a = w ∗ a means that when querying a , the user's answer (represented by w ∗ a ) is w a , Ψ CG ( w a = w ∗ a ) is the certainty gain when w a = w ∗ a happens, and Pr ( w a = w ∗ a ) is the probability of w a = w ∗ a occurring. If w a = w ∗ a holds, then all the unchecked items whose value of a is not equal to w a should be removed from V k -1 , as they are certainly not satisfying the user's needs.

̸

Formally, let V a value = w a denote the set of all the items whose value of a is equal to w a , and let V a value = w a denote the set of rest items. Then, Ψ CG ( w a = w ∗ a ) can be computed as:

̸

which indicates that the certainty gain, when w a is the user's answer, is the summation of relevance scores of those items not matching the user preference.

To finish Ψ CG ( query ( a )) , we also need to estimate Pr ( w a = w ∗ a ) . To estimate the user preference on attribute a , we leverage the estimated relevance scores given by Ψ RE ( · ) as:

̸

In this case, we remove V k -1 ∩ V a value = w ∗ a from V k -1 , namely we have V k = V k -1 \ V a value = w ∗ a . As attribute a is checked, we have X k = X k -1 \{ a } . Here, w ∗ a is provided by the user after querying a .

̸

By combining Eqs. (4), (6), and (7), we can derive a completed form of Ψ CG ( query ( a )) for a ∈ V k -1 ∪ X k -1 (See Appendix A1.1 for details). Then, at each k -th turn, we can always follow Eq. (3) to obtain the next query a query . As depicted in Figure 1(b), the above process results in an online decision tree, where the nodes in each layer are items and attributes to query, and the depth of the tree is the number of turns (see Appendix A4.3 for visualization of a real-world case).

3.2 From Querying Attributes to Querying Attribute Values

We note that the online decision tree introduced above is a general framework; while applying it to real-world scenarios, there should be some specific designs.

Ψ CG ( · ) for Querying an Attribute Value. One implicit assumption in the above online decision tree is that the user's preference on queried attribute a always falls into the set of attribute values, namely w ∗ a ∈ W a holds. However, it can not always hold, due to (i) a user would not have a clear picture of an attribute, (ii) a user's answer would be different from all the candidate attribute values, e.g., a is Hotel Level , w ∗ a = 3 . 5 , and W a = { 3 , 5 } , as shown in Figure 1(a). In these cases, querying attributes would not be a good choice. Hence, we propose to query attribute values instead of attribute IDs, because (i) a user is likely to hold a clear preference for a specific value of an attribute, e.g., a user would not know an actual Hotel Level of her favoring hotels, but she clearly knows she can not afford a hotel with Hotel Level = 5 , and (ii) since querying attribute values leads to closed questions instead of open questions, a user only needs to answer Yes or No , therefore, avoiding the user's answer to be out of the scope of all the candidate attribute values.

Formally, in this case, A = W x ×X k -1 which indicates we need to choose a value w x ∈ W x where x ∈ X k -1 . In light of this, we compute the expected certainty gain of querying attribute value w x as:

̸

̸

̸

where w ∗ x ∈ W x denotes the user preference on attribute x . Here, different from querying attributes, a user would only respond with Yes (i.e., w x = w ∗ x ) or No (i.e., w x = w ∗ x ). Therefore, we only need to estimate the certainty gain for the above two cases. Ψ CG ( w x = w ∗ x ) can be computed following Eq. (7) and Ψ CG ( w x = w ∗ x ) can be calculated by replacing V x value = w x with V x value = w x . Pr ( w x = w ∗ x ) is estimated in Eq. (8) and Pr ( w x = w ∗ x ) = 1 -Pr ( w x = w ∗ x ) . In this case, if all the values of x have been checked, we have X k = X k -1 \{ x } ; otherwise, X k = X k -1 ; and V k = V k -1 \ V x value = w x if receiving Yes from the user, V k = V k -1 \ V x value = w x , otherwise.

̸

We reveal the connection between querying attributes (i.e., querying attribute IDs) and querying attribute values in the following proposition.

Proposition 1. For any attribute x ∈ X k -1 , Ψ CG ( query ( x )) ≥ Ψ CG ( query ( x ) = w x ) holds for all the possible w x ∈ W x .

This proposition shows that if users could give a clear preference for the queried attribute and their preferred attribute value is one of the candidate attribute values, then querying attributes would be an equivalent or a better choice than querying attribute values. In other words, querying attributes and querying attribute values can not operate on the same attributes (otherwise, Ψ CO ( · ) would always choose to query attributes). Therefore, we can combine querying items and querying attribute values

̸

̸

̸

Algorithm 1 CORE for Querying Items and Attributes

by setting the action space to A = W x ×X k -1 ∪ V k -1 . Then, we can re-formulate Eq. (3) as:

In the context of querying attribute values, we further reveal what kind of attribute value is an ideal one in the following theorem.

Proposition 2. In the context of querying attribute values, an ideal choice is always the one that can partition all the unchecked relevance scores into two equal parts (i.e., the ideal w x ∈ W x , x ∈ X k -1 is the one that makes Ψ CG ( w x = w ∗ x ) = SUM ( { Ψ RE ( v m ) | v m ∈ V k -1 } ) / 2 hold), if it is achievable. And the certainty gain in this case is Ψ CG ( query ( x ) = w x ) = SUM ( { Ψ RE ( v m ) | v m ∈ V k -1 } ) / 2 .

Then, we consider the bound of the expected number of turns. To get rid of the impact of Ψ RE ( · ) , we introduce a cold-start setting [43], where Ψ RE ( · ) knows nothing about the user, and equally assigns relevance scores to all M items, resulting in Ψ RE ( v m ) = 1 /M holds for any v m ∈ V .

Lemma 1. In the context of querying attribute values, suppose that Ψ RE ( v m ) = 1 /M holds for any v m ∈ V , then the expected number of turns (denoted as ̂ K ) is bounded by log M +1 2 ≤ ̂ K ≤ ( M +1) / 2 .

Here, the good case lies in that our conversational agent is capable of finding an attribute value to form an ideal partition at each turn, while the bad case appears when we can only check one item at each turn. We provide detailed proofs of Propositions 1 and 2, and Lemma 1 in Appendix A1.2.

Ψ CG ( · ) for Querying Attributes in Large Discrete or Continuous Space. All the above querying strategies are designed in the context that for each attribute, the range of its candidate values is a 'small' discrete space, namely |W x | ≪|V k -1 | where x ∈ X k -1 . When it comes to cases where W x is a large discrete space or a continuous space, then either querying attribute x or any attribute value w x ∈ W x would not be a good choice. For example, let x be Hotel Price , then when querying x , the user would not respond with an accurate value, and querying x =one possible value could be ineffective. To address this issue, we propose to generate a new attribute value w x and query whether the user's preference is not smaller than it or not. Formally, we have:

where x ∈ X k -1 and w x can be either in or out of W x . Compared to querying attribute values (i.e., Eq. (9)), the new action space is A = R ×X k -1 . Notice that Proposition 2 is also suitable for this case (see detailed description in Appendix A2.1), where the best partition is to divide the estimated relevance scores into two equal parts. Therefore, we produce w x by averaging all the candidate attribute values weighted by the corresponding relevance scores. Formally, for each x ∈ X k -1 , we compute w x as:

where w v m is the value of attribute x in item v m , e.g., in Figure 1(a), let a be Hotel Level , and v m be Hotel A , then w v m = 3 .

Do you require Breakfast S ervice?

Service

Conversational

Agent

Question

Generator

Answer

Extractor

User

I do not care about Breakfast

Service, and I really want a hotel

with a S hower.

Breakfast S ervice: Not

Care

Shower: Y es

Hotel Level: Not Know

Chat-bot E mpowered C onversational Agent

Empowered

Conversational

Agent

(a)

(b)

(c)

In this case, X k = X k -1 , and V k = V k -1 \ V x value <w x if receiving Yes from the user when querying whether user preference is not smaller than w x , V k = V k -1 \ V x value ≥ w x otherwise. V x value <w x is the set of all the items whose value of x is smaller than w x and V x value ≥ w x is the set of the rest items.

3.3 Plugging the Conversational Agent into Recommender Systems

Overall Algorithm. We begin by summarizing CORE for querying items and attributes or querying items and attribute values in Algorithm 1. From the algorithm, we can clearly see that our Ψ CO ( · ) puts no constraints on Ψ RE ( · ) and only requires the estimated relevance scores from Ψ RE ( · ) , therefore, CORE can be seamlessly integrated into any recommendation platform. We note that in a conversational agent, querying attributes and querying attribute values can be compatible, but can not simultaneously operate on the same attribute, due to Proposition 1 . See Appendix A2.3 for a detailed discussion.

Making Ψ CG ( · ) Consider Dependence among Attributes. We notice that the above formulations of either querying attributes or querying attribute values, does not consider the dependence among attributes (e.g., as Figure 1(a) shows, attribute Hotel Level can largely determine attribute Shower Service ). To address this issue, we take Ψ CG ( · ) in Eq. (6) as an example (see detailed descriptions of the other Ψ CG ( · ) s in Appendix A2.2), and re-formulate it as:

where a ∈ X k -1 , and Pr ( query ( a ′ ) | query ( a )) measures the probability of the user preference on a determining the user preference on a ′ . Compared to Ψ CG ( query ( a )) , Ψ D CG ( query ( a )) further considers the impact of querying attribute a on other attributes. To estimate Pr ( query ( a ′ ) | query ( a )) , we develop two solutions. We notice that many widely adopted recommendation approaches are developed on factorization machine (FM) [38], e.g., DeepFM [20]. Therefore, when applying these FM-based recommendation approaches, one approach is to directly adopt their learned weight for each pair of attributes ( a, a ′ ) as the estimation of Pr ( query ( a ′ ) | query ( a )) . When applying CORE to any other recommendation method (e.g., DIN [52]), we develop a statistics-based approach that does estimations by computing this conditional probability Ψ D CG ( query ( a )) based on the given candidate items. We leave the detailed computations of Ψ D CG ( query ( a )) in both ways in Appendix A2.2.

Empowering Ψ CO ( · ) to Communicate with Humans. When applying CORE into real-world scenarios, users may provide a Not Care attitude regarding the queried attributes or queried attribute values. In these cases, we generate V k and X k by V k = V k -1 and X k = X k -1 \{ a } , because querying a is non-informative. To capture the user's different attitudes on queried items and attributes or attribute values, we can incorporate a pre-trained large language model (LLM) (e.g., gpt-3.5-turbo ) in Ψ CO ( · ) . As our online-checking part does not require training, simply plugging an LLM would not cause the non-differentiable issue. In light of this, we exemplify some task-specific prompts to enable the conversational agent to (i) produce questions by prompting queried items and attributes, and (ii) extract the key messages from the user's answers. As shown in Figure 2, a conversational agent consisting of an LLM-chat-bot and our online decision tree algorithm would communicate like humans. We also provide some case studies of the conversational agent in the context of a question generator and an answer extractor. See Appendix A4.2 for a detailed description.

4 Experiments

4.1 Experimental Configurations

We summarize different experimental settings as follows. (i) We design two different quering strategies regarding attributes (shown in line 5 in Algorithm 1). One is querying attributes (i.e., attribute IDs); and the other is querying attribute values. (ii) We introduce two different recommender system settings. One is the hotstart setting (shown in line 1 in Algorithm 1) that initializes the estimated relevance scores of items by a given pre-trained recommender system; and the other is the cold-start setting where those estimated relevance scores are uniformly generated (corresponding to the case where the recommender system knows nothing about the given user). Because the conversational agent Ψ CO ( · ) operates in a dynamic process, we de-

velop a new simulator to simulate the Human-AI recommendation interactions, which consists of a conversational agent and a user agent. Specifically, for each user, we use her browsing log as session data, and treat all the items receiving positive feedback (e.g., chick) as target items. Then, for each k -th turn, when the conversational agent queries an attribute x ∈ X k -1 , the user agent returns a specific attribute value if all the target items hold the same value for x ; otherwise, the user agent returns Not Care . When the conversational agent queries an attribute value w x ∈ W x , the user agent returns Yes if at least one target item holds w x as the value of attribute x ; otherwise, returns No .

For each experimental setting, we first set K MAX , and then evaluate the performance in terms of the average turns needed to end the sessions, denoted as T@ K MAX (where for each session, if Ψ CO ( · ) successfully queries the target item within K MAX turns, then return the success turn; otherwise, we enforce Ψ CO ( · ) to query an item at ( K MAX +1) -th turn, if succeeds, return K MAX +1 , otherwise return K MAX +3 ); and the average success rate, denoted as S@ K MAX (where for each session, if Ψ CO ( · ) does not successfully query the target item within K MAX turns, then we enforce Ψ CO ( · ) to query an item after K MAX turns, if succeeds, return 1 , otherwise return 0 ).

To verify CORE can be applied to a variety of recommendation platforms, we conduct evaluations on three tubular datasets: Amazon [8, 33], LastFM [9] and Yelp [12], three sequential datasets: Taobao [10], Tmall [11] and Alipay [7], two graph-structured datasets: Douban Movie [35, 53] and Douban Book [35, 53]. The recommendation approaches used in this paper, i.e., Ψ RE ( · ) s, include FM [38], DEEP FM [20], PNN [37], DIN [52], GRU [23], LSTM [18], MMOE [32], GCN [27] and GAT [46]. We also use COLD START to denote the cold-start recommendation setting. The conversational methods used in this paper, i.e., Ψ CO ( · ) s, include (i) Abs Greedy (AG) always queries an item with the highest relevance score at each turn; (ii) Max Entropy (ME) always queries the attribute with the maximum entropy in the context of querying attributes, or queries the attribute value of the chosen attribute, with the highest frequency in the context of querying attribute values; (iii) CRM [44], (iv) EAR [30], (v) CRIF [25], (vi) UNICORN [13]. Here, AG can be regarded as a strategy of solely applying Ψ RE ( · ) . Both CRM and EAR are reinforcement learning based approaches, originally proposed on the basis of FM recommender system. Thus, we also evaluate their performance with hot-start FM-based recommendation methods, because when applying them to a cold-start recommendation platform, their strategies would reduce to a random strategy. Consider that ME is a Ψ CO ( · ) , independent of Ψ RE ( · ) (namely, the performance of hot-start and cold-start recommendation settings are the same); and therefore, we only report their results in the cold-start recommendation setting. We further introduce a variant of CORE , denoted as CORE + D where we compute and use Ψ D CG ( · ) s instead of Ψ CG ( · ) s in line 5 in Algorithm 1.

We provide detailed descriptions of datasets and data pre-processing, simulation design, baselines, and implementations in Appendix A3.1, A3.2, A3.3, and A3.4.

4.2 Experimental Results

We report our results of querying attributes and items in Table 1, and the results of querying attribute values and items in Tables 2, and 3, 4, and summarize our findings as follows.

Reinforcement learning based methods work well in querying items and attributes but perform poorly in querying items and attribute values. By comparing Table 1 to Table 2, we can see a huge performance reduction of CRM and EAR. One possible explanation is that compared to attribute IDs, the action space of querying attribute values is much larger. Thus, it usually requires much more collected data to train a well-performed policy.

T@ K MAX could not align with S@ K MAX . A higher success rate might not lead to fewer turns, and ME gains a worse performance than AG in some cases in the cold-start recommendation setting. The main reason is that although querying an attribute value can obtain an equivalent or more certainty gain than querying an item at most times, however, only querying (a.k.a., recommending) an item could end a session. Therefore, sometimes, querying an attribute value is too conservative. It explains why CORE outperforms AG in terms of S@3 but gets a lower score of T@3 in the Amazon dataset and FM recommendation base.

Our conversational agent can consistently improve the recommendation performance in terms of success rate. CORE can consistently outperform AG, in terms of success rate, especially for the cold-start recommendation setting. As AG means solely using recommender systems, it indicates that Ψ CO ( · ) can consistently help Ψ RE ( · ) . One possible reason is that our uncertainty minimization

framework unifies querying attribute values and items. In other words, AG is a special case of CORE , where only querying items is allowed.

Considering Dependence among attributes is helpful. Comparisons between CORE and CORE + D reveal that considering the dependence among attributes could improve the performance of CORE in most cases.

We further investigate the impact of K MAX by assigning K MAX = 1 , 3 , 5 , 7 , 9 and reporting the results of CORE and AG on Amazon dataset in the context of the cold-start and hot-start recommendation settings in Figure 3. The results further verify the superiority of CORE , especially with a cold-start Ψ RE ( · ) .

Our conversational agent is more stable when the training data is limited. For further assessment of CORE 's stability in comparison to the baseline methods, we conducted an evaluation by randomly selecting a subset of the training set comprising only 50% of the samples. The results of this evaluation are presented in Table 5. The table clearly demonstrates that CORE exhibits a higher level of stability in its performance when compared to existing reinforcement learning-based frameworks.

1.0

0.8

SuccessRate

0.6

0.4

0.2

CORE+COLD

AG+COLD

CORE+HOT

AG+HOT

0.0

1

2

3

4

5

6

7

8

9

KMAX

We also provide a case study of incorporating an LLM into CORE to handle free-text inputs and output human language Appendix A4.2, where we provide detailed prompts. We further offer a visualization of an online decision tree in A4.3.

5 Conclusions and Future Work

In this paper, we propose CORE that can incorporate a conversational agent into any recommendation platform in a plug-and-play fashion. Empirical results verify that CORE outperforms existing reinforcement learning-based and statistics-based approaches in both the setting of querying items and attributes, and the setting of querying items and attribute values. In the future, it would be interesting to evaluate CORE in some online real-world recommendation platforms.

Acknowledgement. The Shanghai Jiao Tong University team is supported by National Key R&D Program of China (2022ZD0114804), Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102) and National Natural Science Foundation of China (62076161, 62177033). Jiarui Jin would like to thank Wu Wen Jun Honorary Doctoral Scholarship from AI Institute, Shanghai Jiao Tong University.

References

Deepak Agarwal and Bee-Chung Chen. Regression-based latent factor models. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 19-28, 2009.

Keping Bi, Qingyao Ai, Yongfeng Zhang, and W Bruce Croft. Conversational product search based on negative feedback. In Proceedings of the 28th acm international conference on information and knowledge management , pages 359-368, 2019.

Haibin Chen, Qianli Ma, Zhenxi Lin, and Jiangyue Yan. Hierarchy-aware label semantics matching network for hierarchical text classification. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 4370-4379, 2021.

Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems , pages 7-10, 2016.

Konstantina Christakopoulou, Alex Beutel, Rui Li, Sagar Jain, and Ed H Chi. Q&r: A twostage approach toward interactive recommendation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 139-148, 2018.

Konstantina Christakopoulou, Filip Radlinski, and Katja Hofmann. Towards conversational recommender systems. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining , pages 815-824, 2016.

Alipay Dataset. https://tianchi.aliyun.com/dataset/datadetail?dataid=53.

Amazaon Dataset. https://jmcauley.ucsd.edu/data/amazon/.

LastFM Dataset. https://grouplens.org/datasets/hetrec-2011/.

Taobao Dataset. https://tianchi.aliyun.com/dataset/datadetail?dataid=649.

Tmall Dataset. https://tianchi.aliyun.com/dataset/datadetail?dataid=42.

Yelp Dataset. https://www.yelp.com/dataset/.

Yang Deng, Yaliang Li, Fei Sun, Bolin Ding, and Wai Lam. Unified conversational recommendation policy learning via graph-based reinforcement learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 1431-1441, 2021.

Mehdi Elahi, Francesco Ricci, and Neil Rubens. A survey of active learning in collaborative filtering recommender systems. Computer Science Review , 20:29-50, 2016.

Chongming Gao, Wenqiang Lei, Xiangnan He, Maarten de Rijke, and Tat-Seng Chua. Advances and challenges in conversational recommender systems: A survey. AI Open , 2:100-126, 2021.

Nadav Golbandi, Yehuda Koren, and Ronny Lempel. On bootstrapping recommender systems. In Proceedings of the 19th ACM international conference on Information and knowledge management , pages 1805-1808, 2010.

Nadav Golbandi, Yehuda Koren, and Ronny Lempel. Adaptive bootstrapping of recommender systems using decision trees. In Proceedings of the fourth ACM international conference on Web search and data mining , pages 595-604, 2011.

Alex Graves and Alex Graves. Long short-term memory. Supervised sequence labelling with recurrent neural networks , pages 37-45, 2012.

Asela Gunawardana and Christopher Meek. Tied boltzmann machines for cold start recommendations. In Proceedings of the 2008 ACM conference on Recommender systems , pages 19-26, 2008.

Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: a factorization-machine based neural network for ctr prediction. arXiv preprint arXiv:1703.04247 , 2017.

Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. Lightgcn: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval , pages 639-648, 2020.

Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In Proceedings of the 26th international conference on world wide web , pages 173-182, 2017.

Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session-based recommendations with recurrent neural networks. arXiv preprint arXiv:1511.06939 , 2015.

Badr Hssina, Abdelkarim Merbouha, Hanane Ezzikouri, and Mohammed Erritali. A comparative study of decision tree id3 and c4. 5. International Journal of Advanced Computer Science and Applications , 4(2):13-19, 2014.

Chenhao Hu, Shuhua Huang, Yansen Zhang, and Yubao Liu. Learning to infer user implicit preference in conversational recommendation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 256-266, 2022.

Dietmar Jannach, Ahtsham Manzoor, Wanling Cai, and Li Chen. A survey on conversational recommender systems. ACM Computing Surveys (CSUR) , 54(5):1-36, 2021.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer , 42(8):30-37, 2009.

Xuan Nhat Lam, Thuc Vu, Trong Duc Le, and Anh Duc Duong. Addressing cold-start problem in recommendation systems. In Proceedings of the 2nd international conference on Ubiquitous information management and communication , pages 208-211, 2008.

Wenqiang Lei, Xiangnan He, Yisong Miao, Qingyun Wu, Richang Hong, Min-Yen Kan, and Tat-Seng Chua. Estimation-action-reflection: Towards deep interaction between conversational and recommender systems. In Proceedings of the 13th International Conference on Web Search and Data Mining , pages 304-312, 2020.

Raymond Li, Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. Towards deep conversational recommendations. Advances in neural information processing systems , 31, 2018.

Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pages 1930-1939, 2018.

Julian McAuley and Alex Yang. Addressing complex and subjective product-related queries with customer reviews. In Proceedings of the 25th International Conference on World Wide Web , pages 625-635, 2016.

Nima Mirbakhsh and Charles X Ling. Improving top-n recommendation for cold-start users via cross-domain information. ACM Transactions on Knowledge Discovery from Data (TKDD) , 9(4):1-19, 2015.

Douban Movie and Douban Book Datasets. https://www.kaggle.com/datasets/fengzhujoey/doubandatasetratingreviewside-information.

Seung-Taek Park and Wei Chu. Pairwise preference regression for cold-start recommendation. In Proceedings of the third ACM conference on Recommender systems , pages 21-28, 2009.

Yanru Qu, Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, and Jun Wang. Productbased neural networks for user response prediction. In 2016 IEEE 16th international conference on data mining (ICDM) , pages 1149-1154. IEEE, 2016.

Steffen Rendle. Factorization machines. In 2010 IEEE International conference on data mining , pages 995-1000. IEEE, 2010.

Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. Bpr: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:1205.2618 , 2012.

Neil Rubens, Mehdi Elahi, Masashi Sugiyama, and Dain Kaplan. Active learning in recommender systems. Recommender systems handbook , pages 809-846, 2015.

Neil Rubens and Masashi Sugiyama. Influence-based collaborative active learning. In Proceedings of the 2007 ACM conference on Recommender systems , pages 145-148, 2007.

Martin Saveski and Amin Mantrach. Item cold-start recommendations: learning local collective embeddings. In Proceedings of the 8th ACM Conference on Recommender systems , pages 89-96, 2014.

Andrew I Schein, Alexandrin Popescul, Lyle H Ungar, and David M Pennock. Methods and metrics for cold-start recommendations. In Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval , pages 253-260, 2002.

Yueming Sun and Yi Zhang. Conversational recommender system. In The 41st international acm sigir conference on research & development in information retrieval , pages 235-244, 2018.

Alex Tamkin, Kunal Handa, Avash Shrestha, and Noah Goodman. Task ambiguity in humans and language models. arXiv preprint arXiv:2212.10711 , 2022.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 , 2017.

Tong Yu, Yilin Shen, and Hongxia Jin. A visual dialog augmented interactive recommender system. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , pages 157-165, 2019.

Xiaoying Zhang, Hong Xie, Hang Li, and John CS Lui. Conversational contextual bandit: Algorithm and application. In Proceedings of the web conference 2020 , pages 662-672, 2020.

Yongfeng Zhang, Xu Chen, Qingyao Ai, Liu Yang, and W Bruce Croft. Towards conversational search and recommendation: System ask, user respond. In Proceedings of the 27th acm international conference on information and knowledge management , pages 177-186, 2018.

Cheng Zhao, Chenliang Li, Rong Xiao, Hongbo Deng, and Aixin Sun. Catn: Cross-domain recommendation for cold-start users via aspect transfer network. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 229-238, 2020.

Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. Deep interest evolution network for click-through rate prediction. In Proceedings of the AAAI conference on artificial intelligence , volume 33, pages 5941-5948, 2019.

Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pages 1059-1068, 2018.

Feng Zhu, Chaochao Chen, Yan Wang, Guanfeng Liu, and Xiaolin Zheng. Dtcdr: A framework for dual-target cross-domain recommendation. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management , pages 1533-1542, 2019.

A1 Conversational Agent Design on Uncertainty Minimization Framework

A1.1 Detailed Deviations

This paper introduces a conversational agent built upon the recommender system to interact with a human user. We begin by summarizing the interaction principles, taking Figure 1 as an example.

Definition A1 ( Conversational Agent and Human User Interactions ) . Our conversational agent is designed to act in the following four ways.

(i) Query an item v ∈ V k -1 , where V k -1 is the set of unchecked items after k -1 interactions (e.g., recommend Hotel A to the user).

(ii) Query an attribute x ∈ X k -1 , where X k -1 is the set of unchecked attributes after k -1 interactions (e.g., query what Hotel Level does the user want).

(iii) Query whether the user's preference on an attribute x ∈ X k -1 is equal to a specific attribute value w x ∈ W x where W x is the set of values of attribute x (e.g., query whether the user likes a hotel with Hotel Level = 5).

(iv) Query whether the user's preference on an attribute is not smaller than a specific value w x ∈ R (e.g., query whether the user likes a hotel with Hotel Level ≥ 3.5).

The human user is supposed to respond in the following ways.

(i) For queried item v , the user should answer Yes (if v satisfies the user) or No (otherwise) (e.g., answer Yes , if the user likes Hotel A ).

(ii) For queried attribute x , the user should answer her preferred attribute value, denoted as w ∗ x ∈ W x (e.g., answer 3 for queried attribute Hotel Level ), or answer Not Care to represent that any attribute value works.

(iii) For queried attribute value w x , the user should answer Yes (if w x matches the user preference) or No (otherwise) (e.g., answer Yes , if the user wants a hotel with Hotel Level = 5), or answer Not Care to represent that any attribute value works.

(iv) For queried attribute value w x , the user should answer Yes (if the user wants an item whose value of attribute x is not smaller than w x ) or No (otherwise) (e.g., answer Yes , if the user wants a hotel with Hotel Level = 5), or answer Not Care to represent that any attribute value works.

Then, we separately describe the key concepts, including uncertainty, certainty gain, and expected certainty gain, introduced in this paper.

Definition A2 ( Uncertainty ) . For the k -th turn, we define uncertainty, denoted as U k , to measure how many estimated relevance scores are still unchecked, which can be formulated as:

where Ψ RE ( v m ) outputs the estimated relevance score for item v m . The above equation tells us that the uncertainty of each turn is decided by the unchecked items.

It is straightforward to derive the certainty gain, as the uncertainty reduction at each turn.

Definition A3 ( Certainty Gain ) . For the k -th turn, we define certainty gain of k -th interaction as:

where ∆ V k = V k -1 \ V k . For simplicity, we use a to denote the k -th action of the conversational agent. According to the Human-AI interactions introduced in Definition A1, we can derive:

̸

(16)

̸

̸

̸

where V a value = w ∗ a is the set of unchecked items whose value of attribute a is not equal to the user answer w ∗ a , V x value = w x is the set of unchecked items whose value of attribute x is not equal to the queried attribute value w x , V x value = w x , a subset of V k -1 , is the set of unchecked items whose value of attribute x is equal to the queried attribute value w x , V x value <w x is the set of unchecked items whose value of attribute x is smaller than the queried attribute value w x , V x value ≥ w x is the set of unchecked items whose value of attribute x is not smaller than the queried attribute value w x .

To estimate the certainty gain from taking each possible action, we introduce the expected certainty gain as follows.

Definition A4 ( Expected Certainty Gain ) . For the k -th turn, we define expected certainty gain to estimate ∆ U k on Ψ CO ( · ) taking a different action.

Then, at each turn, we can compute the candidate action, getting the maximum expected certainty gain, as the action to take, denoted as a query . In practice, as shown in Proposition 1, for each attribute, querying attribute IDs, i.e., (ii), and querying attribute values, i.e., (iii), is not compatible. And, (iv) is particularly designed for a large discrete or continuous value space, which can be regarded as a specific attribute value generation engineering for (iii) (i.e., using Eq. (12) to directly compute the queried value for each attribute), and thus, we treat (iv) as a part of (iii). Therefore, we organize two querying strategies. One is querying (i) and (ii), whose objective can be formulated as Eq. (3). The other one is querying (i) and (iii), and the objective can be written as Eq. (10).

Besides V k , we further summarize the update of X k as follows. Similarly, we can define ∆ X k := X k -1 \ X k , then ∆ X k can be written as:

Based on the above, CORE runs as Algorithm 1 shows.

Remark. One of the advantages of querying attribute values, compared to querying attributes, is that the user's answer to queried attribute would be out of the candidate attribute values (i.e., W x for queried attribute x ). We are also aware that one possible solution is that the conversational agent would list all the candidate attribute values in the query. However, we argue that this approach would work only when the number of candidate values is small (namely, |W x | is small) such as attributes Color and Hotel Level , but can not work when there are many candidate values, e.g., attribute Brand , since listing all of them would significantly reduce the user satisfaction.

A1.2 Proofs

Proposition A1. For any attribute x ∈ X k -1 , Ψ CG ( query ( x )) ≥ Ψ CG ( query ( x ) = w x ) holds for all the possible value w x ∈ W x .

Proof. For consistency, we re-formulate Eq. (6) as:

where x is the queried attribute, and w ∗ x represents the user preference on x (corresponding to the notations a and w ∗ a respectively). We can also re-write Ψ CG ( w x = w ∗ x ) as:

̸

̸

̸

where w x is an arbitrary attribute value in W x \{ w ′ x } . The above equation is built upon the simple fact that after an attribute x , the answer of the user preferring w ′ x is equivalent to the answer of the user not preferring all the other w ′′ x s, which can remove all the unchecked items whose value is equal to any w ′′ x . Thus, the expected certainty gain of knowing the user preferring w ′ x is not smaller than knowing the user not preferring any one w x ∈ W x \{ w ′ x } , and the equality holds only in the case where W x = { w x , w ′ x } , namely there are only two candidate attribute values.

Based on the above equations, we can derive:

̸

̸

Since we put no constraint on x ∈ X k -1 , thus it proves the proposition.

Proposition A2. In the context of querying attribute values, an ideal choice is always the one that can partition all the unchecked relevance scores into two equal parts (i.e., the ideal w x ∈ W x , x ∈ X k -1 is the one that makes Ψ CG ( w x = w ∗ x ) = SUM ( { Ψ RE ( v m ) | v m ∈ V k -1 } ) / 2 hold), if it is achievable. And the certainty gain in this case is Ψ CG ( query ( x ) = w x ) = SUM ( { Ψ RE ( v m ) | v m ∈ V k -1 } ) / 2 .

Proof. Without loss of generalizability, in the context of querying attribute values, we recap the formulation of Ψ CG ( query ( x ) = w x ) , shown in Eq. (9) as:

̸

̸

̸

Therefore, we have R Yes = R/ 2 , and in this case, Ψ = R/ 2 . Then, we can reach the conclusion that the ideal partition is the one dividing all the unchecked relevance scores, i.e., R , into two equal parts; and in this case, Ψ CG ( query ( x ) = w x ) = R/ 2 = SUM ( { Ψ RE ( v m ) | v m ∈ V k -1 } ) / 2 , which indicates that querying w x can check half of the relevance scores in expectation.

Lemma A1. In the context of querying attribute values, suppose that Ψ RE ( v m ) = 1 /M holds for any v m ∈ V , then the expected number of turns (denoted as ̂ K ) is bounded by log M +1 2 ≤ ̂ K ≤ ( M +1) / 2 .

Proof. We begin by considering the best case. According to Proposition 2, if we can find an attribute value w x , where querying w x can partition the unchecked relevance scores into two equal parts, then

̸

where we use R YES to denote SUM ( { Ψ RE ( v m ) | v m ∈ V k -1 ∩V a value = w a } ) , the expected certainty gain of the event w x = w ∗ x happening (i.e., the user answers Yes to querying w x ), and use R to denote the summation of relevance scores of all the unchecked items, i.e., SUM ( { Ψ RE ( v m ) | v m ∈ V k -1 } ) . For convenience, we use Ψ to denote Ψ CG ( query ( x ) = w x ) . Then, Ψ can be regarded as a function of R YES . Namely, R YES is the independent variable and Ψ is the dependent variable.

To maximize Ψ , we have:

̸

̸

̸

we can build a binary tree, where we can check M/ 2 k at the k -th turn. Therefore, we have:

which derives ̂ K = log M +1 2 . In the worst case, we only can query one item during one query. Then, the expected number of turns is:

Combining Eqs. (24) and (25) together, we can draw log M +1 2 ≤ ̂ K ≤ ( M +1) / 2 .

A2 Plugging the Conversational Agent into Recommender Systems

A2.1 Ψ CG ( · ) for Querying Attributes in Large Discrete or Continuous Space

The main idea of our conversational agent is to recursively query the user to reduce uncertainty. The core challenge is that there exist some cases where querying any attribute values or items can not effectively reduce uncertainty. Most of these cases occur when some key attributes have a large discrete space or a continuous space, leading to a broad decision tree. Formally, for a key attribute x ∈ X k -1 , a 'small' discrete space usually means |W x | ≪|V k -1 | . For example, for attribute Hotel Price , then querying x , the user would not respond with an accurate value, and querying x =one possible value could be ineffective.

To address this issue, we propose to find a w x ∈ R instead of w x ∈ W x , and then we can query whether the user's preference is not smaller than it or not, i.e., query ( x ) ≥ w x instead of whether the user's preference is equal to w x or not, i.e., query ( x ) = w x . Then, the expected certainty gain, in this case, can be written as:

where

where V x ≥ w x is the set of items whose value of attribute x is not smaller than w x , and V x<w x is the set of the rest items, namely V x ≥ w x ∪ V x<w x = V k -1 ; and

Therefore, the same as query ( x ) = w x , query ( x ) ≥ w x also divides the unchecked items into two parts, and the user is supposed to answer Yes or No , corresponding to either one of the two parts. Then, Proposition 2 also works here. Namely, for each attribute x ∈ X k -1 , the oracle w x , denoted as w O x , is the one that can partition the relevance scores into two equal parts. Formally, we have:

Since it is infeasible to find an exact oracle one, we approximate w O x as:

where w v m is the value of attribute x in item v m . It indicates that our estimation is the average of the attribute values for the items in V k -1 weighted by their relevance scores.

A2.2 Making Ψ CG ( · ) Consider Dependence among Attributes

The following techniques allow CORE to take the dependence among attributes into account. We provide two ways, where one requires an FM-based recommender system, while the other one poses no constraint.

Taking Ψ CG ( · ) in Eq. (6) as an example, we re-formulate Eq. (6) as, when a ∈ X k -1 , and then we can compute Ψ CG ( query ( a )) as:

where we use Ψ D CG ( · ) to denote this variant of Ψ CG ( · ) .

Estimation from a Pre-trained FM-based Recommender System. If our recommender system applies a factorization machine (FM) based recommendation approach, then we can directly adopt the learned weights as the estimation of Pr ( query ( a ′ ) | query ( a )) in Eq. (31). Taking DeepFM [20] as an example, we begin by recapping its FM component:

where the model parameters should be estimated in the recommender system (in line 1 in Algorithm 1), including w 0 ∈ R , w ∈ R N , V ∈ R N × D and D is the dimension of embedding. And, ⟨· , ·⟩ is the dot product of two vectors of size D , defined as ⟨ v i , v j ⟩ = ∑ D d =1 v id · v jd . In this regard, for each pair of attributes (e.g., ( a, a ′ ) in Eq. (31)), we can find the corresponding normalized ⟨ v n , v n ′ ⟩ / | v n | · | v n ′ | as the estimation of Pr ( query ( a ′ ) | query ( a )) .

Estimation in a Statistical Way. If applying any other recommendation approach to the recommender system, we design a statistical way. We first decompose Ψ D CG ( query ( a )) according to Eq. (6):

where we define Ψ D CG ( w a = w ∗ a ) as:

where Pr ( w a ′ = w ∗ a ′ | w a = w ∗ a ) measures the probability of how likely getting the user's preference on attribute a (i.e., w a = w ∗ a ) determinates the user's preference on other attributes (i.e., w a ′ = w ∗ a ′ ). For example, in Figure 1, if the user's preference on attribute Hotel Level is 5 (i.e., a is Hotel Level , w a is 5 and the user's answer is Yes ), then we could be confident to say that the user preference on attribute Shower Service is Yes (i.e., a ′ is Shower Service , w a ′ is Yes , and the user's answer is Yes ), i.e., Pr ( w a ′ = w ∗ a ′ | w a = w ∗ a ) is close to 1.

We estimate Pr ( w a ′ = w ∗ a ′ | w a = w ∗ a ) by using the definition of the conditional probability:

where V a value = w a is the set of items whose value of a equals w a , and V ( a value = w a ) ∧ ( a ′ value = w a ′ ) is the set of items whose value of a equals w a and value of a ′ equals w a ′ . By incorporating Eqs. (34) and (35) into Eq. (33), we can compute Ψ D CG ( query ( a )) for any a ∈ X k -1 .

Extensions to Other Cases. Besides querying attributes, we also introduce another querying strategy to query attribute values. Formally, we can have:

̸

̸

̸

where Ψ D CG ( w x = w ∗ x ) can be computed by Eq. (34), and the formulation of Ψ D CG ( w x = w ∗ x ) could be directly extended from Eq. (34) by replacing Ψ CG ( w x = w ∗ x ) with Ψ CG ( w x = w ∗ x ) , and replacing Pr ( w a ′ = w ∗ a ′ | w a = w ∗ a ) with Pr ( w a ′ = w ∗ a ′ | w a = w ∗ a ) . Pr ( w a ′ = w ∗ a ′ | w a = w ∗ a ) could be computed by replacing V a value = w a with V a value = w a , and replacing V ( a value = w a ) ∧ ( a ′ value = w a ′ ) with

̸

̸

̸

̸

̸

̸

̸

̸

V ( a value = w a ) ∧ ( a ′ value = w a ′ ) . V a value = w a is the set of items whose value of a does not equal w a , and V ( a value = w a ) ∧ ( a ′ value = w a ′ ) is the set of items whose value of a does not equal w a and value of a ′ does not equal w a ′ .

̸

Then, we have made our conversational agent consider the dependence among attributes for cases (ii) and (iii), summarized in Definition A1. There is no need to consider the dependence in case (i), and, as concluded in Appendix A1.1, (iv) can be regarded as a special engineering technique in (iii), and thus, one just needs to follow the same way to handle case (iv).

A2.3 Overall Algorithm

We summarize the overall algorithm in Algorithm 1. CORE follows an offline-training-and-onlinechecking paradigm, where offline-training is represented in lines 1 and 12, and online-checking is represented in lines 5, 6 and 7.

As shown in line 5, there are two querying settings, i.e., querying items and attributes, and querying items and attribute values. We note that querying attributes and querying attribute values can be compatible, but can not simultaneously operate on the same attribute. We recap that Proposition 1 says that for each attribute, assuming users could give a clear answer showing their preference, querying an attribute can always obtain certainty gain not smaller than querying any attribute value of the attribute.

Therefore, in practice, we would select those attributes that are likely to receive a clear preference from users (e.g., attributes Category , Brand ) in the setting of querying items and attributes, and use the rest of attributes (e.g., attribute Price ) in the setting of querying items and attribute values. Also, as stated at the end of Appendix A1.1, we can further select several attributes with a small space of attribute values, use them in the setting of querying items and attributes, and list all the candidate attribute values in the queries. In this regard, for any attribute, since the space of attribute values is changing in the context of querying attribute values, then we may transfer from the setting of querying attribute values to querying attributes, when there are few unchecked candidate attribute values.

All the above operations need careful feature engineering, which should be task-specific and datasetspecific. We argue that this is out of the scope of this paper, and we leave it for future work.

A3 Experimental Configuration

A3.1 Dataset Descriptions and Data Pre-processing

We summarize the datasets used in this paper as follows.

Amazon dataset [8, 33] is a dataset collected by Amazon from May 1996 to July 2014. There are 1,114,563 reviews of 133,960 users and 431,827 items and 6 attributes.

LastFM dataset [9] is a dataset collected from Lastfm, a music artist recommendation platform. There are 76,693 interactions of 1,801 users and 7,432 items and 33 attributes.

Yelp dataset [12] is a dataset collected from Yelp, a business recommendation platform. There are 1,368,606 interactions of 27,675 users and 70,311 items and 590 attributes. We follow [30] to create 29 (parents) attributes upon 590 original attributes, and we use the newly created ones in our experiments.

Taobao dataset [10] is a dataset collected by Taobao from November 2007 to December 2007. It consists of 100,150,807 interactions of 987,994 users and 4,162,024 items with an average sequence length of 101 and 4 attributes.

Tmall dataset [11] is a dataset collected by Tmall from May 2015 to November 2015. It consists of 54,925,331 interactions of 424,170 users and 1,090,390 items with an average length of 129 and 9 attributes.

Alipay dataset [7] is a dataset collected by Alipay, from July 2015 to November 2015. There are 35,179,371 interactions of 498,308 users and 2,200,191 items with an average sequence length of 70 and 6 attributes.

Douban Movie dataset [35, 53] is a dataset collected from Douban Movie, a movie recommendation platform. There are 1,278,401 interactions of 2,712 users and 34,893 items with 4 attributes.

̸

̸

Douban Book dataset [35, 53] is a dataset collected from Douban Book, a book recommendation platform. There are 96,041 interactions of 2,110 users and 6,777 items with 5 attributes.

In summary, our paper includes three tubular datasets (i.e., Amazon, LastFM, Yelp), three sequential datasets (i.e., Taobao, Tmall, Alipay), and two graph-structured datasets (i.e., Douban Book, Douban Movie). First, we follow the common setting of recommendation evaluation [22, 39] that reduces the data sparsity by pruning the users that have less than 10 historical interactions and the users that have at least 1 positive feedback (e.g., clicks in Taobao). We construct each session by sampling one user and 30 items from her browsing log (if less than 30 items, we randomly sample some items that are not browsed, as the items receive negative feedback, into the session). During sampling, we manage the ratio of the number of items receiving positive feedback and the number of negative feedback falls into the range from 1:10 to 1:30. We use a one-to-one mapping function to map all the attribute values into a discrete space to operate. From those attributes with continuous spaces, we directly apply our proposed method introduced in Section 3.2.

A3.2 Simulator Design

As summarized in Definition A1, there are two main agents in our simulator, namely a conversational agent and a user agent. The conversational agent is given the set of candidate items (i.e., V ), and the set of candidate attributes (i.e., X ) (together with their candidate values, i.e., W x for every x ∈ X ). Then, at k -th turn, the conversational agent is supposed to provide an action of querying, either one from (i), (ii), (iii) and (iv) shown in Definition A1, and the user agent is supposed to generate the corresponding answer and derive the set of unchecked items (i.e., V k ), and the set of unchecked attributes (i.e., X k ) (together with the unchecked values of each attribute x ). Let W k x be the set of the unchecked values of x , then its update function is simple. Firstly, we assign W 0 x = W x , and we can further define ∆ W k x = W k -1 x \ W k x , then ∆ W k x can be written as:

For simplicity, we omit the above update in the main text.

From the above description, we know that the conversational agent and the user agent are communicating through exchanging the set of unchecked items and unchecked attributes (and unchecked attribute values). We also develop a port function in the conversational agent that leverages a pre-trained large language model to generate the human text for each action. See Appendix A4.2 for detailed descriptions and examples.

A3.3 Baseline Descriptions

We first summarize the recommendation approaches, denoted as Ψ RE ( · ) , used in this paper as follows.

COLD START denotes the cold-start setting, where all the relevance scores of items are uniformly generated. In other words, for the item set V = { v m } M m =1 , we set the relevance score for each item v m ∈ V by Ψ RE ( v m ) = 1 /M .

FM [38] is a factorization machine-based recommendation method working on tabular data, which considers the second-order interactions among attributes (i.e., feature fields).

DEEP FM [20] combines an FM component and a neural network component together to produce the final prediction.

PNN [37] includes an embedding layer to learn a representation of the categorical data and a product layer to capture interactive patterns among categories.

DIN [52] designs a deep interest network that uses a local activation unit to adaptively learn the representation of user interests from historical behaviors.

GRU [23] applies a gated recurrent unit (GRU) to encode the long browsing histories of users.

LSTM [18] applies a long short term memory unit (LSTM) to encode the historical browsing logs of users.

MMOE [32] develops a multi-gate mixture-of-experts that can model the user's multiple behaviors by sharing the expert sub-models across all the behaviors.

GCN [27] designs a graph convolutional network that learns representations of nodes (either users or items) by passing and aggregating their neighborhood information.

GAT [46] designs a graph attention network that adopts an attention mechanism to consider the different contributions from the neighbor nodes in representing the central nodes (either users or items).

We then summarize the conversational techniques, denoted as Ψ CO ( · ) , used in this paper as follows.

AG (Abs Greedy) always queries an item with the highest relevance score at each turn, which is equivalent to solely using the recommender system as a conversational agent.

ME (Max Entropy) always generates a query in the attribute level. In the setting of querying items and attributes, it queries the attribute with the maximum entropy, which can be formulated as:

In the setting of querying items and attribute values, we first apply Eq. (38) to obtain the chosen attribute and then we select the attribute value with the highest frequency of the chose attribute as:

where xx is computed following Eq. (38). To evaluate the success rate, during the evaluation turn, we apply AG after employing ME.

CRM [44] integrates the conversational component and the recommender component by feeding the belief tracker results to an FM-based recommendation method. It is originally designed for the single-round setting, and we follow [30] to extend it to the multiple-round setting.

EAR [30] consists of three stages, i.e., the estimation stage to build predictive models to estimate user preference on both items and attributes based on an FM-based recommendation approach, the action stage to determine whether to query attributes or recommend items, the reflection stage to update the recommendation method.

CRIF [25] formulates the conversational recommendation scheme as a four-phase process consisting of offline representation learning, tracking, decision, and inference. In the inference module, by fully utilizing the relation between users' attribute-level and item-level feedback, CRIF can explicitly deduce users' implicit preferences.

UNICORN [13] formulate the conversational recommendation problem as a unified policy learning task. UNICORN uses a dynamic weighted graph-based enforcement learning method to learn a policy to select the action at each conversation turn, either asking an attribute or recommending items.

The proposed methods are listed as follows.

CORE is our proposed method calculating Ψ CG ( · ) s in line 5 in Algorithm 1.

CORE + D is a variant of CORE that computes Ψ D CG ( · ) s instead of Ψ CG ( · ) s, making Ψ D CG ( · ) s consider the dependence among attributes.

A3.4 Implementation Details

For each recommendation approach, we directly follow their official implementations with the following hyper-parameter settings. The learning rate is decreased from the initial value 1 × 10 -2 to 1 × 10 -6 during the training process. The batch size is set as 100. The weight for the L2 regularization term is 4 × 10 -5 . The dropout rate is set as 0.5. The dimension of embedding vectors is set as 64. For those FM-based methods (i.e., FM, DEEP FM), we build a representation vector for each attribute. We treat it as the static part of each attribute embedding, while the dynamic part is the representation of attribute values stored in the recommendation parameters. In practice, we feed the static and dynamic parts together as a whole into the model. After the training process, we store the static part and use it to estimate the dependence among attributes, as introduced in Appendix A2.2. All the models are trained under the same hardware settings with 16-Core AMD Ryzen 9 5950X (2.194GHZ), 62.78GB RAM, NVIDIA GeForce RTX 3080 cards.

A4 Additional Experimental Results

A4.1 Performance Comparisons

We conduct the experiment in two different experimental settings. One is the setting of querying items and attributes, and the other is the setting of querying items and attribute values. We report the results of the former setting on tabular datasets (i.e., Amazon, LastFM, Yelp) in Table A1, and also report the results of the latter setting on these tabular datasets in Table A2. We also evaluate the performance of CORE in sequential datasets and graph-structured datasets, and report their results in Table A3 and Table A4 respectively.

By combining these tables, our major findings are consistent with those shown in Section 4.2. Moreover, we also note that the performance of CORE in querying items and attributes is close to the oracle, and thus considering the dependence among attributes in CORE + D does not bring much improvement.

A4.2 Incorporating Our Conversational Agent with a Frozen Chat-bot

With the development of pre-trained large language models (LLMs), chat-bots built based on these LLMsare capable of communicating like humans, which is a powerful tool to allow our conversational agent to extract the key information from the user's free text feedback and generate free text for querying attributes and items. Concretely, chat-bot can act as either a question generator or a answer extractor . As shown in Figure 2, if our conversational agent decides to query attribute breakfast service , then the command passes to the question generator to generate a free text question 'Do you require breakfast service?' The user answers the question by free text 'I do not care about breakfast service, and I really want a hotel with shower', and then the answer extractor extracts the user preference on the given answer, namely the user does not care about attribute breakfast service and gives positive feedback on attribute shower .

The following is the corresponding output provided by the LLM.

Great choice! Hotel A is a wonderful option. Would you like me to provide more information about the hotel or help you book a room?

Also, we evaluate using an LLM as an answer extractor by an example of extracting the user preference on attributes, e.g., breakfast service , hotel level , and shower .

Excuse me, may I ask for your preference on hotel level? Would you prefer 5-star hotel or are you open to other options?

According to Definition A1, we have exemplified querying (i), (ii), and (iii) in the above examples. We further evaluate querying (iv). Namely, we evaluate using an LLM as a question generator by an example of generating a question to query whether the user preference is not smaller than an attribute value, e.g., Hotel Level not smaller than 3.

Query Attribute Value:

whether C hildren's Books or not

whether 0001061100 or not

Query Attribute Value:

whether Price is not smaller

than $5.5 or not

whether 0001062395 or not

Y es

No

Y es

Y es

Y es

No

No

No

We note that if there are too many attribute IDs (or too many attribute values) in the use case, then it might need to further incorporate some hierarchical designs [3] and ambiguous matching [45] into the above system, which is out of the scope of this paper, and we leave it for future work.

A4.3 Visualization and Case Study

We investigate a case in Amazon dataset in the setting of querying items and attribute values, where the target item is 0001062395 . We depict the online decision tree in Figure A1. From the figure, we can see that the conversational agent first queries a value of attribute Category , then queries (i.e., recommends) an item 000161100 ; and after that, it queries another attribute, i.e., Price , and finally queries an item 0001062395 .

Compared to Figure 1(b), this example seems like a chain. The main reason is that in practice, the user would give the corresponding answer to the query at each turn. Therefore, the binary tree (in the setting of querying items and attribute values) would reduce to a chain.

From this case, we also can observe that our conversational agent is capable of jointly considering the items and attributes to search for the target items in the session.

A5 Connections to Existing Approaches

A5.1 Connections to Conversational Recommender Systems

Bridging recently emerged conversational techniques and recommender systems becomes an appealing solution to model the dynamic preference and weak explainability problems in recommendation task [15, 26], where the core sub-task is to dynamically select attributes to query and make recommendations upon the corresponding answers. Along this line, one popular direction is to build a conversational recommender system, which combines the conversational models and the recommendation models from a systematic perspective. In other words, these models are treated and learned as two individual modules [30, 2, 44, 48]. For example, compared to previous literature [5, 6, 47], recent work [49] builds the systems upon the multiple turn scenarios; unfortunately, it does not investigate when to query attributes and when to make recommendations (i.e., query items). To solve this issue, prior works [30, 31] develop reinforcement learning-based solutions. However, all these previous methods based on a reinforcement learning framework are innately suffering from insufficient usage of labeled data and high complexity costs of deployment.

Instead, CORE can be seamlessly adopted to any recommendation method (especially those widely adopted supervised learning-based recommendation methods), and is easy to implement due to our conversational strategy based on the uncertainty minimization theory.

A5.2 Connections to (Offline) Decision Tree Algorithms

Decision tree algorithms [24] such as ID3 and C4.5 were proposed based on information theory, which measures the uncertainty in each status by calculating its entropy. If we want to directly adopt the entropy measurement for the conversational agent, then one possible definition of entropy is

where H k is the empirical entropy for k -th turn, Y is the set of all the labels, and V y value = y is the set of items whose label is y . For convenience, we call this traditional decision tree as offline decision tree .

The main difference between the previous offline decision tree and our online decision tree lies in that our online decision tree algorithm does not have labels to measure the 'uncertainty', instead, we have access to the estimated relevance scores given by recommender systems. We also note that directly using the user's previous behaviors as the labels would lead to a sub-optimal solution, because (i) offline labels in collected data are often biased and can only cover a small number of candidate items, and (ii) offline labels only can reflect the user's previous interests, but the user's preferences are always shifting.

To this end, we measure the uncertainty in terms of the summation of the estimated relevance scores of all the unchecked items after previous ( k -1) interactions. Formally, we define our uncertainty as:

where Ψ RE ( · ) denotes the recommender system. Similar to the information gain in the offline decision tree, we then derive the definition of certainty gain (as described in Definition 1), and formulate the conversational agent into an uncertainty minimization framework.

A5.3 Connections to Recommendation Approaches to Addressing Cold-start Issue

Cold-start issues are situations where no previous events, e.g., ratings, are known for certain users or items [29, 43]. Commonly, previous investigations have revealed that the more (side) information, the better the recommendation results. In light of this, roughly speaking, there are two main branches to address the cold-start problem. One direction is to combine the content information into collaborative filtering to perform a hybrid recommendation [1, 19, 36, 42], and a recent advance [34, 50] proposes to further combine the cross-domain information to the recommender system. The other direction

is to incorporate an active learning strategy into the recommender system [14, 40], whose target is to select items for the newly-signed users to rate. For this purpose, representative methods include the popularity strategy [17], the coverage strategy [16], and the uncertainty reduction strategy [41], where the first one selects items that have been frequently rated by users, the second one selects items that have been highly co-rated with other items, and the third one selects items that can help the recommender system to better learn the user preference.

Integrating the conversational agent with the recommender system offers a promising solution to address the cold start issue by effectively leveraging user interactions. This approach allows for querying additional information from users, thereby enhancing the system's understanding of their preferences and requirements.

A distinctive aspect of our conversational agent's strategy compared to existing active learning strategies, lies in its primary objective. While active learning strategies aim to improve the recommender system's performance, our conversational agent prioritizes meeting the user's immediate needs during the ongoing session. Consequently, the offline-training and online-checking paradigm employed in our approach assigns higher relevance scores to items that are considered "uncertain." This diverges from conventional settings where uncertainty estimation is typically conducted independently of relevance estimation.

By incorporating the conversational agent into recommender system, we leverage user interactions to gather more comprehensive information, which can effectively address the challenges associated with cold-start scenarios. Our primary focus is to provide an optimal user experience in real-time rather than solely optimizing the recommender system's accuracy. This novel paradigm emphasizes the importance of uncertainty estimation in relevance estimation, setting our approach apart from traditional methods.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method that combines offline recommendation learning with online decision tree learning to enable recommendation in a conversational format with a few rounds of interaction. The proposed approach is evaluated on multiple datasets and in various validation settings to assess its effectiveness.\n\nSTRENGTHS:\nThis paper proposes a novel framework called CORE for uncertainty reduction. CORE improves the performance of recommender systems by revealing user preferences through querying attributes and attribute values.\n\nThe paper evaluates the performance of CORE using multiple datasets and different recommendation methods. The experimental results demonstrate that CORE improves the success rate of recommendations.\n\nWEAKNESSES:\n(1) This paper heavily relies on decision trees, and there is a significant design cost involved in creating those decision trees. Decision trees are domain-specific and lack clear design guidelines, and the paper does not discuss this aspect or provide any discussion on it in relation to each experimental dataset. Therefore, the weakness lies in the inability to evaluate this aspect.\n\n\n(2) The reliance on metadata such as price, location, and hotel rank in decision trees makes the approach simplistic. In conversational recommender systems, it is important to consider finer item characteristics that can be obtained from diverse evaluations in user reviews while engaging in conversational interactions with AI. Furthermore, in conversational recommender systems, it is crucial to incorporate language models (LM) to capture linguistic variations and users' intuitive expressions in order to reflect them in the recommendations. However, this aspect is not well addressed in the paper (though it states a little in Section 4).\n\n\n(3) The evaluation setup of this paper raises some concerns. While the main evaluation metric is the number of turns, the criteria used to determine correctness seem to be based solely on whether an item was checked or not. In the case of hotels, for example, it is important to consider whether a reservation was actually made, and relying solely on item checking may lead to an inflated number of correct predictions.\nAnd, while evaluations are performed on different datasets, there is a lack of detailed explanations about the specifics of the conversations and the actual recommendations made in each dataset. Without this information, it becomes difficult to fully understand the effectiveness and characteristics of the proposed method.\nFurthermore, the experimental results of this paper do not include comparisons with other methods or existing approaches. It is important to compare the proposed method with other recommendation techniques in order to clearly demonstrate its advantages and improvements.\nMoreover, it appears that there is a lack of qualitative evaluation. While the paper utilizes multiple experimental datasets for validation, it is difficult to grasp the nature of the interactions and how satisfactory item recommendations are achieved in each dataset. Given that the conversation is a crucial aspect, the absence of information on the nature of the conversations makes it challenging to form a clear picture of the system's performance.\n\nQUESTIONS:\nIt seems that the paper does not address the importance of considering finer item characteristics derived from diverse evaluations in user reviews during conversational interactions with AI. Could you please provide some insights into this aspect and its potential impact on the proposed approach?\n\nAre there any existing conversational recommender systems that could serve as comparative methods to the proposed approach in this paper?\n\nLIMITATIONS:\nThe paper lacks consideration for finer item characteristics obtained from user reviews with language models (LM) to capture linguistic variations and users' intuitive expressions. This limitation restricts the ability to provide more nuanced and personalized recommendations in conversational recommender systems.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel framework called CORE that bridges conversational agents and recommender systems via an uncertainty minimization principle. The framework treats a recommender system as an offline relevance score estimator and a conversational agent as an online relevance score checker. The conversational agent can query either items or attributes (or attribute values) to reduce the uncertainty of the user’s preference and find a target item. The paper shows that CORE can be applied to various recommendation platforms and datasets, and can outperform existing reinforcement learning-based or statistical methods in both hot-start and cold-start settings. The paper also demonstrates how to empower the conversational agent with a pre-trained language model to communicate more naturally with the user.\n\nSTRENGTHS:\n1.\tThe paper presents a comprehensive study of the core task of conversational recommendation by considering the complex situations that arise in conversational contexts.\n2.\tThe authors conduct an extensive experimental analysis and exploration.\n3.\tThe proposed method achieves lower complexity and latency compared to existing state-of-the-art approaches.\n\nWEAKNESSES:\n1.\tThe innovations and core contributions of the paper are questionable. Most conversational recommendation papers consist of offline and online components, but CORE does not appear to have any notable novel features despite considering more complex interaction settings. The improvements in these scenarios do not seem to be sufficient to warrant acceptance of this paper.\n2.\tThe formatting and organization of the paper could be improved to facilitate readability, with inconsistencies in notation and an excessive number of annotations. The description in Section 3 is too verbose, with the core method not being highlighted adequately. A concise explanation of Algorithm 1 would suffice to clarify the scenario.\n3.\tThe core of CORE still relies on an offline recommendation system, and even the online decision making depends on scores from the offline recommendation system. The key challenge of conversational recommendation is determining how to make dynamic decisions based on user interests and conversation context. Existing reinforcement learning-based methods evidently consider more factors, whereas the proposed heuristic method in this paper depends on recommendation scores that themselves have uncertainty in dynamic interactions. The methodological foundations of CORE appear weak and unconvincing.\n\nQUESTIONS:\n1.\tThe modeling approach preferred by CORE users appears to be more of a heuristic estimate compared to some dynamic decision-making methods in reinforcement learning. How does your method ensure that decision-making performance is better than RL-based methods, given that your method seems more like an improved version of Max-entropy?\n2.\tAs mentioned in the Weaknesses Section, all of your decisions rely on the estimated scores from the offline recommendation model. Using this score as a benchmark for uncertainty measurement is dubious. How can you ensure that the recommendation model's estimates are accurate and reliable?\n\nLIMITATIONS:\n1.\tPlease refer to the Weaknesses Section.\n2.\tThe lack of comparison with the latest CRS methods, such as CPR[1] and UNICORN[2]. These new methods share some similarities with the core settings of the paper, such as UNICORN's decision space, which also includes item space and attribute space.\n\n[1] Wenqiang Lei, Gangyi Zhang, Xiangnan He, Yisong Miao, Xiang Wang, Liang Chen, and Tat-Seng Chua. 2020. Interactive Path Reasoning on Graph for Conversational Recommendation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’20). 2073–2083.\n[2] Yang Deng, Yaliang Li, Fei Sun, Bolin Ding, and Wai Lam. 2021. Unified conversational recommendation policy learning via graph-based reinforcement learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1431–1441.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is about conversational recommender systems (CRS), which are systems that can interact with users through natural language and provide personalized recommendations. The paper addresses the challenge of incorporating a conversational agent into any existing recommender system in a plug-and-play fashion, without requiring reinforcement learning or data collection. The paper proposes CORE, a novel offline-training and online-checking framework that bridges a conversational agent and a recommender system via a unified uncertainty minimization objective. The paper claims that CORE can benefit any recommendation platform and can handle different types of data, attributes, and queries. The paper reports that CORE outperforms existing methods in both hot-start and cold-start recommendation settings, and can communicate as a human if empowered by a pre-trained language model.\n\nSTRENGTHS:\n1) Originality: The paper proposes a novel offline-training and online-checking framework, CORE, that bridges a conversational agent and a recommender system via a unified uncertainty minimization objective. This approach is original in its ability to incorporate a conversational agent into any existing recommender system in a plug-and-play fashion, without requiring reinforcement learning or data collection.\n2) Quality: The paper develops a new human-AI recommendation simulator and conducts extensive experiments on eight industrial datasets with nine popular recommendation approaches. The results show that CORE outperforms existing methods in both hot-start and cold-start recommendation settings.\n3) Significance: The proposed CORE framework has the potential to benefit any recommendation platform and can handle different types of data, attributes, and queries. This makes it a significant contribution to the field of conversational recommender systems.\n\nWEAKNESSES:\n1) The introduction does not provide enough background and motivation for the problem of conversational recommender systems. It should explain why this problem is important and challenging, and what are the existing gaps and limitations in the literature. A possible way to improve it is to cite more relevant works and compare them with the proposed approach.\n2) The proposed approach in Section 3 is not clearly explained and justified. It does not provide enough details and intuition for how the uncertainty minimization framework works, how the expected certainty gain is derived, how the online decision tree algorithm is implemented, and how the dependence among attributes is considered. It also does not discuss the advantages and disadvantages of the proposed approach compared to other methods. A possible way to improve it is to provide more examples, figures, pseudocode, and analysis to illustrate the proposed approach.\n3) The experimental setup in Section 4 is not comprehensive and fair. It does not describe how the datasets are preprocessed, how the hyperparameters are tuned, how the baselines are implemented, and how the evaluation metrics are calculated. \n4) The experimental results in Section 4 are not convincing and insightful. They do not show the statistical significance of the performance differences, the impact of different factors/hyperparameters, or the qualitative analysis of the generated conversations. They also do not discuss the limitations and challenges of the proposed approach, such as scalability, robustness, diversity, etc. \n5) The references in Section 6 are incomplete and inconsistent. Some references are missing important information such as authors, titles, venues, pages, etc., or have different formats or styles.\n\nQUESTIONS:\n1) How does CORE handle the situation when the user’s answer is not clear or consistent with their previous answers?\n2) How does CORE compare with other conversational recommender systems that use natural language generation or understanding techniques?\n3) How does CORE deal with the trade-off between exploration and exploitation in querying items or attributes?\n4) How does CORE adapt to different domains or scenarios of recommendation, such as books, movies, etc.?\n5) How does CORE cope with the noise or bias in the offline estimated relevance scores or the online user responses?\n6) How does CORE handle the scalability and efficiency issues when dealing with large-scale datasets or action spaces?\n7) How does CORE incorporate user preferences on multiple attributes or items simultaneously?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, a conversational part of a recommender system is proposed. It is assumed that a recommendation model is available that assigns scores to user-item pairs (which estimate the probabilities of acceptance of the corresponding recommendations), items have a number of important attributes (numerical and categorical), and each user has preferences over the values of these attributes. A conversational model should sequentially decide (based on the recommendation scores) whether to ask the current user about the preferenced values of one attribute, or to try recommending an item, which can be accepted or rejected by the user. The goals are to maximize the rate of successful dialogs which accepted recommendations and to minimize the average number of rounds until the user accepts a recommendation.\n\nIn the current paper, authors propose a simple greedy algorithm that chooses an action that minimizes the expected sum of estimated recommendation scores of the unchecked items, that is, items that are still can be chosen by the user in the light of the obtained information so far.\n\nAuthors compare their algorithm with two previous baselines based on RL (CRM and EAR) and conclude that the proposed method outperforms RL-based approaches.\n\nSTRENGTHS:\n-\tPaper is mostly well-written, the algorithmic ideas and propositions are clear.\n-\tThe idea of expected uncertainty minimization is reasonable\n\nWEAKNESSES:\n1.\tThe main questionable point is the contribution of the paper. I have the following doubts:\n\n-\tPapers on conversational RS referenced in related work are very old, I do not see any works dated by 21-23 years. Expected gain maximization resembles expected improvement criterion, one of the most widely-used Bayesian optimization algorithms (see \"Efficient global optimization of expensive black-box functions.\" by Jones et al.) Both algorithms are based on the estimation of the expected gain using the posterior distribution. I think, corresponding references are needed here.\n\n-\tMotivation behind the proposed method in comparison with RL-based approaches is not convincing for me. Problem setting does inherently lead to RL approaches: it includes actions, states, rewards, and requires a policy for an optimal trajectory. Why a simple definite empirical greedy algorithm should outperform RL-based models?\n\n2.\tTheoretical part is not perfect. \n\nIn problem setting, it would be useful to state explicitly what assumptions underly the proposed method. For example, it in implicitly assumed that there is a unique item the user needs (therefore the sum of probabilities over items equals 1, see Eq. 5). Another example is that information on user preferences on an attribute carries binary information on user preferences on items, what underlies their division on “cheched” and “unchecked”.  The motivation behind the approach is based on these assumptions, which are rather controversial in practical cases considered in experiments.\n\nTheoretical results are very simple. Claimed propositions are self-evident, but do not carry much understanding on the specific properties, novelty, and contribution of the proposed EG maximization technique.\n\nThere are some inaccuracies in equations. For example:\n\n-\tEquation for the action space at line 175 looks formally incorrect. As $W_x$ is different for different $x$, we cannot obtain $A$ as a set product.\n-\tEq. 10 Is not formally accurate: the argmax operator should be applied to a function.\n\n\n3.\tExperiments and practice.\n\nThe baselines are rather old (CRM and EAR). What about CRIF from “Learning to Infer User Implicit Preference in Conversational Recommendation”?\n\nThere is no discussion on the possibility of efficient implementation of the proposed algorithm. How the calculation of uncertainty gains can be implemented in practice? Naïve summation over all items with the given attribute (see, e.g., eqs. 7-8) for each possible attribute values (see eq. 7) looks impractical in real-time conversational recommender systems.\n\nQUESTIONS:\n-\tCould you summaries the proposed learning method?\n-\tWhat is the theoretical foundation behind the proposed greedy algorithm of expected gain maximization? Is it suboptimal in some assumptions?\n\nLIMITATIONS:\nI do not see any limitations of the proposed work", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-21"}

comments: SUMMARY:
In this paper,  the authors propose a learning framework called CORE that can incorporate a conversational agent into any recommendation platform, to complementarily check estimated offline relevance scores in each online user session. Experiments results on comprehensive benchmark datasets show that CORE outperforms existing reinforcement learning-based and statistics-based approaches in querying items and attributes/attribute values.

STRENGTHS:
1. The proposed offline-training and online-checking paradigm bridges a conversational agent and recommender systems via a unified uncertainty minimization framework. It's effective and efficient, and flexible to handle recommendations in both cold-start and hot-start settings. 

2. Various large-scale benchmark datasets are used in experiments, and the results are convincing.

WEAKNESSES:
1. Are the comparison results in Tables 1, 2, 3 and 4 statistically significant? It'd be great if t-test results could be provided. 

2. The central idea in this paper is to bring a conversational agent for recommendation systems. If possible, it'd be great to conduct an experiment on a real online platform to have user sessions to show the superiority of the proposed algorithm.

QUESTIONS:
1. What kind of data/features in each dataset are used to estimate the offline relevance scores in the first place?  Are the accuracy of the offline relevance scores critical to the performance of recommendations after online-checking framework via conversational agent?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-22

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors assert that there is a gap between traditional recommender systems trained on offline (historical) preference data and conversational assistants that aim to elicit user feedback about their *current* preferences. The authors assert that conversational assistant training relies on reinforcement learning-based frameworks and suffer from data insufficiency. They propose the CORE method for learning a conversational recommender system based on the idea of maximizing \"certainty gain\" by using a pre-trained recommender system as a scoring model for an online decision tree process. Experiments across a wide range of datasets demonstrates consistent improvements over reinforcement learning methods for learning conversational recommenders.\n\nSTRENGTHS:\n- The problem formulation makes sense and the authors organize the methodology in 3.1 and 3.2 in an intuitive manner. However, as mentioned in the weaknesses section there could be improvements in clarity.\n- It is important that the authors extend attribute certainty gain computation to continuous attributes (Lemma 1) as this is a realistic setting for many attributes of interest in real-world situations (see prices, interest rates, or other continuous values).\n- As a general extension of the above, the authors carefully consider multiple cases that bring the problem setting closer to real-world user interactions and preference considerations (neutral responses, attribute dependence). They extend the derivation of certainty gain across these situations.\n- Experiments across a variety of datasets and base recommender systems demonstrate that CORE performs better in most cases compared to CRM and EAR baselines.\n\nWEAKNESSES:\n- The authors should clarify the reasoning for the statement on lines (135-136) that \"if [the queried item is in the target set], the sesion is done and therefore the certainty gain is the summation of all relevance scores in V_{k-1}\". Why is this the certainty gain? It is rooted in the definition in (1) that the certainty gain of a target item sets all items to \"checked for certainty\" but it is unclear what the basis is for this or whether it's a choice made to simplify the modeling.\n- The authors omit discussion of a large area of conversational recommendation focusing on eliciting user feedback via critiques of surfaced attributes [including 1-4]. This area is directly related to the \"attribute query\" in the online-checking framework of CORE, and merits discussion in the literature review.\n- The reference to ChatGPT-3.5-turbo seems gratuitous, as there is little mention of it in the main paper other than a statement on (254) that a pre-trained language model can be used as the Psi_{CO} component.\n- The experimental analysis seems relatively sparse. It would be good to see a deeper analysis (case studies, for example, or human evaluations) in the main paper body. It would also help to see a discussion of the limitations, edge cases, or remaining challenges in the space and what other challenging experimental settings were considered.\n\n\nReferences:\n[1] Antognini, D. \"Interacting with Explanations through Critiquing\" (2021)\n[2] Antognini, D. \"Positive and Negative Critiquing for VAE-based Recommenders\" (2022)\n[3] Li, S. \"Self-Supervised Bot Play for Conversational Recommendation with Rationales\" (2022)\n[4] Wu, G. \"Deep language-based critiquing for recommender system\" (2019)\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nThe authors do not really discuss limitations of the work in a substantive way. I would like to see a greater discussion of challenges and limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-28"}