Title: CONTEXTUAL MEMORY BANDIT FOR PRO-ACTIVE DIALOG ENGAGEMENT

Abstract: An objective of pro-activity in dialog systems is to enhance the usability of conversational agents by enabling them to initiate conversation on their own. While dialog systems have become increasingly popular during the last couple of years, current task oriented dialog systems are still mainly reactive and users tend to initiate conversations. In this paper, we propose to introduce the paradigm of contextual bandits as framework for pro-active dialog systems. Contextual bandits have been the model of choice for the problem of reward maximization with partial feedback since they fit well to the task description. As a second contribution, we introduce and explore the notion of memory into this paradigm. We propose two differentiable memory models that act as parts of the parametric reward estimation function. The first one, Convolutional Selective Memory Networks, uses a selection of past interactions as part of the decision support. The second model, called Contextual Attentive Memory Network, implements a differentiable attention mechanism over the past interactions of the agent. The goal is to generalize the classic model of contextual bandits to settings where temporal information needs to be incorporated and leveraged in a learnable manner. Finally, we illustrate the usability and performance of our model for building a pro-active mobile assistant through an extensive set of experiments.

1. INTRODUCTION:

Dialog agents are about to become the interface of choice for eyeless interactions. The recent progress in the different building blocks of conversational agents have started to make such interfaces usable by the general public in large variety of situations. Furthermore, a large set of APIs has currently been introduced to design task-specific agents through graphical and textual development interfaces. Such interfaces allow even a non-expert to design dialog interfaces which is another reason for the current expansion of such a technology.

The majority of agents developed today assumes a reactive type of interaction (Williams and Young, 2007) . Indeed, in a task-oriented setting, the conversation is designed to be initiated by the user. We believe that the seamless nature of conversational interfaces in larger spectrum of contexts makes them particularly suitable for proactive service development. More precisely, the agent will have to infer, regarding a set of observed variables, the pertinence of a given suggestion or conversation engagement on its own and learn from user feedback. A large set of examples can be envisioned. For example, a driving assistant could use a voice interface to pro-actively warn about a potential traffic jam ahead. Similarly, a personal assistant agent could suggest a venue regarding the current location of its user and his interests.

A challenge for building such an agent is the absence of full feedback; the agent needs to infer the quality of its pro-active decisions from the partial feedback given by the user. Contextual bandit (Kakade et al., 2008; Chu et al., 2011) is a particularly appropriate framework to formalize and develop effective solutions with possible theoretical guarantees in such a setting. While in a conventional supervised learning paradigm, the learner has full access to a dataset where true labels are provided, in many domains, including many web-based applications, the agent needs to learn from partial feedback. Online advertising placement and personalized news (Li et al., 2010; Tang et al., 2014) are important examples of such domains. The typical setting can be summarized in the following way. First, the agent observes a current context. Then, using this context and other source of knowledge (past activities of the user, etc.), the system suggests an action. Finally, the user chooses to react (or not) to the suggestion. Crucially, the system is only able to learn from the user reaction to the suggested action. In the context of a pro-active dialog systems, the setting is comparable. An agent will have to choose the timing and the content of the dialog acts based on (a) the current perceived context (b) its knowledge about the user (c) the reward function to optimize. More formally, the purpose of a contextual bandit is to infer a function that maps the observation and available prior knowledge to the expected reward of a candidate action. Beyond the reward function learning, the partiality of feedback also leads to an exploration-exploitation dilemma.

A limitation of classic framework of contextual bandit is the hypothesis that the current context is sufficient support for decision (Lu et al., 2010; Zhou, 2015) . As a second contribution, we propose to incorporate memory to the reward estimation function. Two models are proposed; one with a selective memory and another with a differentiable attention mechanism as proposed in Perez and Liu (2016) computed over a, potentially long, history of contexts. As illustration, we show that such models are particularly meaningful for personal assistance use-cases.

Roadmap: Section 2 summarizes the state of the art for incrementality and pro-activity in dialog systems. Section 3 recalls the formalization of contextual bandit and some of its current challenges. Section 4 details our proposed models of contextual bandits with differentiable memorization capabilities. Section 5 describes our exploration policies and their usage in the setting of pro-active dialogs. Finally, Section 6 presents experiments conducted in the domain of personal assistance for mobility and situated suggestions.

2. PRO-ACTIVE DIALOG ENGAGEMENT:

In this section, we address pro-activity and its links to incrementality in dialog systems. After defining the terms we give an overview of both tasks. In the last part, we address evaluation in such a setting of partial feedback.

2.1. DEFINITIONS:

The problem of pro-active conversation is often studied from two different angles. On a first hand, the question of incrementality studies the usage of spontaneous dialog act emission as part of an active comprehension mechanism (Jonsdottir et al., 2008) . Indeed, while maintaining a distribution over latent variables composing the state of the current dialog, the dialog system faces three basic choices a) let the user continue to speak b) repeat a term said by the user for implicit confirmation c) ask the user to repeat for disambiguation or explicit confirmation. These problems are currently solved using rule-based systems as in the work of Baumann et al. (2011) or have been formalized as a delayed reward control tasks that can be solved using reinforcement learning as in the work of Khouzaimi et al. (2016) .

On the other hand, pro-active interaction is defined as the faculty of a conversational agent to spontaneously address the user. Using a variety of observed and inferred variables from the current perceived context or prior knowledge available, the conversational agent needs to choose either to continue to observe silently or to initiate a conversation regarding a certain subject (like suggesting a task to complete, display a reminder or propose a recommendation). In the context of an eyeless interaction, such capability is crucial yet under-studied. To our knowledge, no work of that kind has been carried out specificly in the context of dialog systems. In the next section, we address the questions of partial feedback and exploitation-exploration which are central to this task.

2.2. CHALLENGES:

The first challenge of pro-active conversation systems is to develop the capability to rely on partial feedback. Assuming a large panel of available subjects to address and variety of possible recommendations to make, supported by a potentially large panel of decision support variables, the system needs to leverage on a partial feedback that will be provided by the user. Indeed, given a proposition made by the agent, the user will only provide a feedback for the given proposition. In such a setting, a proactive conversational agent needs to solve a so-called exploration/exploitation dilemma. On one hand, the agent needs to leverage on the already gathered feedback to choose propositions that maximize the current expected reward. On the other hand, the agent needs to explore, i.e. to choose under-investigated propositions to gather meaningful information about the user preferences and profile.

Another difficulty is the necessity of the learnt decision model to incorporate potentially long term information from the history of client interactions, activities and their potential relationship to available prior knowledge. In such situations, using a learnable model with fixed-length memory like a Long Short Term Memory (Hochreiter and Schmidhuber, 1997) is challenging. To cope with this difficulty, attention-based models (Bahdanau et al., 2014) have been suggested in the context of Natural Language Understanding. Then, this proposition has been extended to more general cases of Markovian control. In such settings, memory enhanced controllers has been developed in a full reinforcement learning setting that features (delayed) rewards for chosen actions.

In the next section, we propose to formalize the problem of pro-activity in conversational system with a memory-enhanced contextual bandit model. The purpose is to respond to the dual problem of (1) long term dependencies and (2) partial feedback without the necessity of credit assignment learning.

3. CONTEXTUAL BANDIT:

Formally, a contextual bandit algorithm A proceeds in discrete trials t = 1, 2, 3, . . . , T . In trial t:

1. The algorithm observes the current user u t and a set A t of arms or actions together with their feature vectors x t,a for a ∈ A t . The vector x t,a summarizes information of both the user u t and arm a, and will be referred to as the context.

2.:

Based on observed payoffs in previous trials, A chooses an arm a t ∈ A t , and receives payoff r t,a t whose expectation depends on the context x t,a t .

3. The algorithm improves its arm-selection strategy with the new observation, (x t,a t , a t , r t,a t ).

It is important to emphasize here that no feedback (i.e., no payoff r t,a ) is observed for unchosen arms a = a t .

In the process above, the total T -trial payoff of A is defined as P A (T ) = ∑ T t=1 r t,a t . Similarly, we define the T -trial payoff of an oracle that always chooses an arm with the best expected payoff as P A * (T ) = ∑ T t=1 r t,a * t , where a * t is the arm with maximum expected payoff at trial t. The goal is to design A so that the expected total payoff E[P A (T )] is maximized. Equivalently, we may find an algorithm A whose expected regret with respect to the optimal arm-selection strategy A * is minimized. Here, the expected T -trial regret R A (T ) of algorithm A is defined formally by

R A (T ) = E [P A * (T ) -P A (T )] .

(1)

An important simplification of the general contextual bandit problem is the classic K-armed bandit in which (a) the arm set A t remains unchanged and contains K arms for all t, and (b) the user u t (or equivalently, the context (x t,1 , . . . , x t,K ) ) is the same for all t. Since both the arm set and contexts are constant at every trial, they make no difference to a bandit algorithm, and so we will also refer to this type of bandit as a context-free bandit.

Two limitations can be mentioned to the classic framework of contextual bandit. On a first hand, we might need to model potentially long history of interactions as part of the decision support. For example, in the context of a personal assistant a conversation engagement can be triggered regarding a very recent event like the time-matching of a calendar event or the current GPS location of the user. However, longer patterns from series of events occurred through time can motive a suggestion.

In such situation, it is difficult to define a priori the necessary time windows of observations to consider. On the other hand, we might explicitly incorporate previous success into an episodic memory and used it as decision support to the reward estimation function. In the next section, we address such problem by proposing learnable parametric reward function that incorporates a differentiable memorization mechanism. 

4. MEMORY ENHANCEMENT:

Memory-enhanced neural networks have recently regained popularity in general reinforcement learning (Hausknecht and Stone, 2015; Sorokin et al., 2015; Oh et al., 2016) . More precisely, recurrent neural networks have been used to automatically accumulate observations through time and learn to use a memory to support decision making in a differentiable manner. To our knowledge, such a development is yet to be pursued in the specific domain of contextual bandits. In this section, we propose two reward function models that incorporate such a memorization capability.

4.1. CONVOLUTIONAL SELECTIVE MEMORY NETWORK:

Our first model is depicted in Figure 1 . The Convolutional Selective Memory Network, CSMN, merges three information sources. First, X t and U t correspond respectively to the features of the context X t and the user U t observed both at time t. In addition, a record of the last K successful engagements of the agent with the corresponding user are stored into a list of memory blocks M u t = {m 1 , . . . , m K }. Such memory blocks corresponds to (at most) K latest contexts X s , (s < t) for which the rewards r s for U t have been greater than a threshold γ ∈ R.

A 1-dimensional convolutional layer followed by an average pooling layer is used to aggregate meaningful higher level features from these past successful contexts. The model architecture is inspired by Kim (2014) . Let m i ∈ R k , k be the k-dimensional vectors describing the individual contexts. The memory is represented as

EQUATION

where ⊕ is the concatenation operator. Let m i:i+ j refer to the concatenation of observations m i , m i+1 , . . . , m i+ j . A convolution operation involves a bank of filters. Each filter θ ∈ R hk is applied to a window of h observations to produce a new feature. For example, a feature c i is generated from a window of stored observations m i:i+h-1 by

EQUATION

with b ∈ R is a bias term and f is a non-linear function such as the hyperbolic tangent. This filter is applied to each possible window of observations stored into the memory {m 1:h , m 2:h+1 , ..., m n-h+1:n } to produce a feature map

EQUATION

with c ∈ R n-h+1 . We then apply an average pooling operation over the feature map and take the average value ĉ = 1 n-h+1 ∑ n-h+1 i=1 c i as the feature corresponding to this particular filter. The model can be interpreted as a late fusion of old contexts with a current candidate context and user features. The drawback of late fusion is that the memories are not specifically selected regarding the current context. To this purpose, we propose an adaptation of a recently proposed attention-based model presented in the next section.

4.2. CONTEXTUAL ATTENTIVE MEMORY NETWORK:

Our second model is strongly inspired by the Gated End-to-End Memory Network architecture, introduced by Perez and Liu (2016) as an extension to the work of Sukhbaatar et al. (2015a) . It Similarly, the current context x concatenated with the information regarding the user are encoded using another embedding matrix B ∈ R d×|V | , resulting in a context embedding u = BΦ(q). The input memories {m i }, together with the embedding of the context u, are utilized to determine the relevance of each of the history embedded into the memory blocks in the context, yielding in a vector of attention weights

EQUATION

where softmax(a i ) = e a i ∑ i e a i

. Subsequently, the response o from the output memory is constructed by the weighted sum:

EQUATION

For more difficult tasks requiring multiple supporting memories, the model can be extended to include more than one set of input/output memories by stacking K memory layers, often called hops, so that the (k + 1) th hop takes as input the output of the k th hop:

EQUATION

EQUATION

where denotes the Hadamard-product, σ (x) = (1 + e -x ) -1 , W k T and b k are the hop-specific parameter matrix and the bias term for the k th hop, and T k (x) is the transform gate for the k th hop. As the final step, the prediction of the expected reward r for the input context, is computed by

r t = u K W ,

where W ∈ R d is part of the parameters for the model to learn. Such enhancement allows to deal with the non-Markovian hypothesis in the context definition. In Section 6 the proposed model is used to implement a goal-oriented proactive dialog agent. 

5. EXPLORATION POLICIES:

A key challenge in bandit problems is the optimal balancing of exploration and exploitation. To minimize the regret in Eq. 1, an algorithm A exploits its past experience to select the arm that appears the best. On the other hand, this seemingly optimal arm may in fact be suboptimal, due to incompleteness of A's knowledge. In order to make informed decisions, A has to also explore by sometimes choosing seemingly suboptimal arms in order to gather more information about them (c.f., step 3 in the bandit process defined in the previous subsection). This exploration can increase short-term regret since some suboptimal arms may be chosen, but on the other hand, obtaining information about the arms' average payoffs will later enhance A's capability to reduce long term regret. Clearly, neither a purely exploring nor a purely exploiting algorithm works best in general but a good trade-off is needed.

In this work, we use Thompson sampling which is a popular Bayesian heuristic to achieve a successful exploration/exploitation trade-off Chapelle and Li (2011) ; Agrawal and Goyal (2013) . Let D be the set of past observations (x t , a t , r t ), where x t is the context in which the arm a t was pulled yielding the reward r t . Thompson sampling assumes a likelihood function P(r|a, x, Θ) paramerized by Θ for the reward. If we denote the true parameters by θ * , we would ideally choose an arm that maximizes the expected reward argmax a E[r|a, x, θ * ], but since we dont know the true parameters we describe our belief about it by a prior distribution P(Θ), and then based on the observed data D, update this belief by the Bayes rule P(Θ|D) ∝ P(Θ) ∏ T t=1 P(r t |x t , a t , Θ). Now if we just wanted to maximize the immediate reward, we would choose an arm that maximizes E[r|a, x] = E(r|a, x, Θ)P(Θ|D)dΘ, but in an exploration/exploitation setting, Thompson sampling advises to select an arm a according to its probability of being optimal, i.e., with probability

EQUATION

In practice, we do not need to calculate the integral but it suffices to draw a random parameter θ from posterior distribution P(Θ|D) and then select the arm with the highest expected reward E[r|a, x, θ ].

The general framework of Thompson sampling is described in Algorithm 1.

5.1. GOAL-ORIENTED PROACTIVE DIALOG POLICY:

In a pro-active dialog system, an agent has to periodically choose whether and what to interact to the user at time t. Its choices are picked among elibigible actions over the current context x t,a t ∈ X and a history of context variables x 1 , . . . , x t-1 ∈ H. To do so, the agent needs to learn a function f θ : (H, X A ) → R which estimates the expected reward associated to a candidate contextualized action and memorized history.

Following the paradigm of contextual bandits, the agent will estimate the expected reward of an action through the feedback of the user. The goal is to accumulate positive rewards for the chosen conversational engagements. Feedback can be explicit like a positive or a negative response, implicit or even no response at all. An example of an implicit positive feedback might be the observed change in travel trajectory in the case of the suggestion of place during a travel. In the next section, we give a detailed description of the experimental setting and evaluation of our models. live music , beers tap , gluten free , little italy , small cozy , cheap drinks , main course , drinks bar , cooked perfectly , group friends saturday night , dining experience , prix fixe , selection drinks , stay away , small portions , beca friend , huge selection , portions small , sat bar french toast , date night , hip hop , don come , outdoor seating , cozy little , second floor , date spot , ve tried , marinara sauce east village , dining area , beer bar , old school , financial district , statue liberty , long wait , craft beer , goat cheese , hole wall selection beers , small plates , cool spot , grab meal , drinks music , healthy lunch , fresh air , soup dumplings , lobster roll , breath fresh fried chicken , hang friends , tuna melt , peanut butter , grab drink , cocktail bar , free wifi , week lunch , eggs benedict , cheese plate sitting bar , cup coffee , bloody mary , took care , little slow , fish dishes , makes feel , cold brew , course tasting , wide variety avocado toast , low key , chocolate cake , ve seen , happy hour , fried egg , large group , new people , meet new , ho ketchup coffee shop , free lyft , bar bartender , pasta dish , fish chips , cool atmosphere , cute little , atmosphere music , bartender drinks , brooklyn bridge happy hour , wine list , went lunch , friday night , hour specials , happy hour specials , people watching , bars city , bread pudding , people don 10 minutes , bar tenders , dining room , bar area , fun atmosphere , minutes later , atmosphere drinks , soup salad , birthday party , steak sandwich ice cream , just right , make feel , feel welcome , little cafe , wonderful experience , did job , mac cheese , 15 20 , bar bit beer selection , dive bar , wine bar , wine selection , free lyft , neighborhood bar , irish pub , irish irish , little expensive , walked away

6. EXPERIMENTS:

In this section, we describe the simulation platform and the experiments we developed to study the performance of the proposed models.

6.1. MOBILITY AGENT PLATFORM:

The overall context of our simulations is situated recommandations. The purpose of the platform is to simulate the spontaneous engagement of a personal assistant with its user regarding situated recommandations near its geographic location. The platform simulates a city where a user follows a series of trajectories composed of waypoints. Situated contextual observations are used to simulate places of interest that will be suggested to user when they arrive at their vicinity. In summary, the goal of the agent is to propose actions regarding the inferred latent variables of the user representing its interests.

In order to enrich the simulation platform with realistic elements of information, we extracted from Google Map Place API 6000 reviews from the city of New York corresponding to four types of places: restaurant, bar, pub, cafe, for a total of 5000 places. A topic model based on Latent Dirichlet Allocation (Blei et al., 2003) was computed from these reviews. The topic distributions were used (1) as latent variable representing the user interests and (2) as latent variable representing the places. In order to leverage on reviews to extract meaningfull topics to characterize different types of places, we suppress to the reviews a lexicon of emotion related words and consider only 2-grams as vectorization of the reviews. Table 1 gives a sample of the latent topics extracted.

The topic model is used to sample keywords that act as observations of places. Table 2 gives examples of places with their corresponding latent topics and observed keywords. A simulated user is defined by a distribution over the latent topics which correspond to its centre of interest. For the places, a distribution of latent topics is drawn from the topic model and a series of observable keywords are sampled from the LDA topic model accordingly. In such a way, the personal assistant has access to a set of topic related keywords of each place of the simulated city. Its goal is to infer the latent centres of interest of the user by recommanding corresponding places as the user passes nearby. The reward r associated to a user latent centre of interest vector u and the latent topic of a given place x is computed by: r = u T x + ε with ε ∼ N (0, σ 2 ) and σ ∈ R is the predefined variance parameter of a Gaussian distributed noise.

Algorithm 2 describes the course of a simulation. The users performs a series of travels composed with waypoints into the city. At each waypoint of the trajectory, the agent has to decide whether to make a recommandation regarding a place nearby the current location of the user. If a recommandation is decided, the corresponding r is computed and presented as instantaneous reward to the personal assistant. Finally, the agent learns from its experience the expected reward of such context. In practice, such reward could be formulated in a real environment as the explicit acceptance by the user or refusal. The addition of the noise to the reward function helps to simulate the challenge of interpretation of a user feedback in a realistic setting. In summary, the dialog act of the agent consists of proposing a place near the current user position. For the experiments, 500 places are sampled over a squared area composed with 300 waypoints. User's trajectories are computed on-the-fly over sampled departure and arrival locations using a shortest path algorithm. The nearby candidate locations at each waypoint are determined using a fixed radius around the user location. A total of 50 users have been sampled for each experiment. Finally, each experiment consists in a total of 1000 trajectories which correspond to the number of epochs. Each experiment is relaunched 5 times for variance estimation. Regarding the learning algorithms, hyper-parameters have been determined by cross-validation. First, the CSMN model uses 64 filters of width 3 in its convolution layer and 4 hidden layers in its fully connected part with 100 hidden units each. Then, the CAMN model uses 3 hops with an embedding size of 30. The hop-specific weight tying schema has been adopted. The last part of the section presents a series of experimental results of the proposed models against a state of the art contextual neural network and an random policy.

6.2. RESULTS:

Figure 3 plots the cumulative sum of rewards for the four considered methods. Regarding the baselines, Fully Connected Neural Network, FFCN, is a multi-layer perceptron which merges the context variable x t and the user features u into the input layer of the model and computes the expected reward of this corresponding context. Two main observations can be provided.

Selective memory mechanism improves performance over neural contextual bandit. Indeed, having a fixed sized memory over the past successfully selected places is a useful addition as decision support. The convolutional memorization mechanism seems effective to such purpose. Attention-based models of the contextual memory bandit is beneficial. The capability to have an unbounded memory with a differentiable attention mechanism improves the observed cumulative rewards.

7. CONCLUSION AND FUTURE WORK:

In this paper we present a novel model for designing pro-active dialog agents. The model is in charge of deciding the timing and the nature of an interaction. We propose to formalize this problem using the framework of contextual memory bandit which is an extension of the classic contextual bandit formalism with the capability of handling variable length history through the usage of a differentiable attention model. We experiment our models in the context of a simulated platform of personal agent which is in charge of emitting spacio-temporally motivated recommendations. We believe this framework constitutes a strong baseline for future research in the domain of pro-active dialog agents as conversational interfaces become popular. Finally, we plan two research topics for future work. On the first hand, we think the hypothesis of contextual bandit can be questioned. Indeed by allowing exploration, the agent might end up changing the preferences of its user. In fact, this stationary hypothesis over the latent variables describing the user, but also the places, are limitation that we plan to address. On the other hand, we want to integrate external knowledge base as part of the memory of our decision models.

:



Figures and Tables:

Figure 1: Convolutional Selective Memory Network.

Figure 2: Contextual Attentive Memory Network with 3 hops.

Two types of weight tying schemes Sukhbaatar et al. (2015b) have been investigated for W k T and b k T : 1. Global: all the weight matrices W k T and bias terms b k T are shared across different hops, i.e., ∀k, l ∈ {1, . . . , K} W k T = W l T and b k T = b l T . 2. Hop-specific: each hop k has its specific weight matrix W k T and bias term b k T , and they are optimized independently.

Figure 3: Mean cumulative sum of rewards.

Algorithm 1 Thompson Sampling Define D = {} for t : 1, . . . , T max do receive context x t Draw θ from posterior distribution P(θ |D) Select arm a t = argmax a E(r|x t , a, θ t ) Receive reward r t D = D ∪ ({x t , a t , r t }) end for

Table content: <table/>

Top representative bi-grams for a set of 15-topics extracted from reviews. Each topic corresponds to a latent centre of interest of the user and observed description of the places. dim sum , late night , 20 minutes , took long , large groups , wednesday night , sweet potato fries , potato fries , sweet potato , beer pong happy hour , russ daughters , tables available , drinks happy hour , drinks happy , group people , lyft credit , open table , portions huge , lamb shank

Table content: <table/>

Algorithm 2 Simulation Dynamics H = {} for e : 1, . . . , TotalNumberO f E pochs do sample user u e ∈ U sample trajectory p 1 , . . . , p T for t : 1, . . . , T do extract the n places (a 1 , . . . , a n ) closest to p t Draw θ from Thompson sampling Choose a * = argmax a i f θ (u e , a i )

Table content: <table><tr><td>Observe reward r t H = H {(u t , a  *  , r t )}</td></tr><tr><td>end for</td></tr><tr><td>end for</td></tr></table>

Four places with main latent topics and observed sampled keywords.

Table content: <table><tr><td>Topics</td><td>Keywords</td><td/><td/><td/></tr><tr><td>8</td><td colspan="5">bloody mary canned tuna cheap beer chips typical nyc</td></tr><tr><td/><td colspan="5">come friends cool spot east village french toast gi-</td></tr><tr><td/><td>gantic portion</td><td/><td colspan="2">gluten free</td><td>happy hour</td><td>house wine</td></tr><tr><td/><td colspan="5">kind hospitality little expensive olive oil ordered burger out-</td></tr><tr><td/><td colspan="5">door seating sandwich steak salad second floor spot brunch</td></tr><tr><td/><td colspan="5">standard bar strawberry shake usual bar vegetarian options</td></tr><tr><td/><td>veggie burger</td><td/><td/><td/></tr><tr><td>16</td><td>attentive drinks</td><td/><td colspan="3">awesome spot</td><td>bar bit</td><td>cafe mocha</td></tr><tr><td/><td>coffee shop</td><td colspan="2">cool bar</td><td colspan="2">fish dishes</td><td>french toast</td><td>li-</td></tr><tr><td/><td colspan="5">brary hotel vacation long wait pretty expensive relaxed bar</td></tr><tr><td/><td colspan="5">scrambled eggs seating inside selection wine spot city</td></tr><tr><td/><td colspan="5">sunday night water refills went sunday</td></tr><tr><td>11</td><td colspan="2">appetizer entree</td><td colspan="2">bars city</td><td>beer selection</td><td>beet relish</td></tr><tr><td/><td colspan="5">cozy spot dance floor downtown tavern fresh squeezed</td></tr><tr><td/><td colspan="5">fried egg jazz band large group red wine small beer</td></tr><tr><td/><td colspan="5">small space steak tartare took forever usually crowded</td></tr><tr><td/><td colspan="5">ve nyc vegan options week wine bars</td></tr><tr><td>6 &amp; 8</td><td colspan="5">atmosphere pleasant authentic japanese bar big bath-</td></tr><tr><td/><td colspan="5">rooms clean cheese steak craft beer dim sum drinks decent</td></tr><tr><td/><td colspan="5">free wifi hang friends happy hour little gem lot fun</td></tr><tr><td/><td colspan="5">pint guinness poached egg salad pattern did seat bar</td></tr><tr><td/><td colspan="5">seating available soup dumplings sunday night</td></tr></table>

Peer review (iclr2018):
 This paper attempts to use contextual bandits for a dialog system. The paper is not clear about how exactly the problem is being mapped to the contextual bandit framework. Similarly, the Thompson sampling algorithm is used, but there is no mention of a posterior or how to sample. Furthermore, the lack of systematic experiments and ablation studies casts doubts on the entire framework. Below is a detailed review and questions for the authors: 1. Please motivate clearly the need for having a bandit framework. One would imagine that dialog systems have a huge amount of data that can be leveraged for a pro-active service. 2. In the sentence "the agent needs to leverage on the already gathered feedback to choose propositions that maximize the current expected reward". The expected reward or the agent is undefined at this point in the paper. 3. After introducing the contextual bandit problem, please give the explicit mapping of your system to this framework. What do the arms correspond to, how are they related, how is the expected reward computed at each round? Another thing which is not clear is what is the environment? It seems that a recommendation by the dialog system will cause the environment to change, in which case it*s not a bandit problem? Isn*t it more natural to model this problem as a reinforcement learing problem? 4. In the sentence, "a record of the last K successful engagements of the agent". It is not clear what constitutes a successful engagement. Also, please justify why you are not keeping negative examples in order to learn. 5. Section 5 describes the Thompson sampling algorithm. Again, there is no mapping between the problem at hand and the TS framework. For instance, what is the posterior in this case. How are you sampling it? Are we being Bayesian, if so what is the prior? 6. In the sentence "longer patterns from series of events occurred through time can motive a suggestion", it seems that the problem you are trying to solve involves delayed feedback. Can you use the strategies in [1] over here? 7. For equation 8, please give some intuition. Why is it necessary? 8. In the sentence, "Regarding the learning algorithms, hyper-parameters have been determined by cross-validation.". Isn*t the data is being collected on the fly in a bandit framework? What is the cross-validation being done on? 9.One experiment on one dataset does not imply that the method is effective. Similarly, the lack of ablation studies for the various components of the system is worrying. [1] Guha, Sudipto, Kamesh Munagala, and Martin Pal. "Multiarmed bandit problems with delayed feedback." arXiv preprint arXiv:1011.1161 (2010).