Venue: ICLR 2018 conference

Reviews:

final_score: rating: 2: Strong rejection
review_content: This paper proposes to use RL (Q-learning and A3C) to optimize the interaction strategy of a search assistant. The method is trained against a simulated user to bootstrap the learning process. The algorithm is tested on some search base of assets such as images or videos. 

My first concern is about the proposed reward function which is composed of different terms. These are very engineered and cannot easily transfer to other tasks. Then the different algorithms are assessed according to their performance w.r.t. to these rewards. They of course improve with training since this is the purpose of RL to optimize these numbers. Assessment of a dialogue system should be done according to metrics obtained through actual interactions with users, not according to auxiliary tasks etc. 

But above all, this paper incredibly lacks of context in both RL and dialogue systems. The authors cite a 2014 paper when it comes to refer to Q-learning (Q-learning was first published in 1989 by Watkins). The first time dialogue has been casted into a RL problem is in 1997 by E. Levin and R. Pieraccini (although it has been suggested before by M. Walker). User simulation has been proposed at the same time and further developed in the early 2000 by Schatzmann, Young, Pietquin etc. Using LSTMs to build user models has been proposed in 2016 (Interspeech) by El Asri et al. Buiding efficient reward functions for RL-based conversational systems has also been studied for more than 20 years with early work by M. Walker on PARADISE (@ACL 1997) but also via inverse RL by Chandramohan et al (2011). A2C (which is a single-agent version of A3C) has been used by Strub et al (@ IJCAI 2017) to optimize visually grounded dialogue systems. RL-based recommender systems have also been studied before (e.g. Shani in JMLR 2005).   

I think the authors should first read the state of the art in the domain before they suggest new solutions. 
review_title: Lack of context

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper "IMPROVING SEARCH THROUGH A3C REINFORCEMENT LEARNING BASED CONVERSATIONAL AGENT" proposes to define an agent to guide users in information retrieval tasks. By proposing refinements of the query, categorizations of the results or some other bookmarking actions, the agent is supposed to help the user in achieving his search. The proposed agent is learned via reinforcement learning. 

My concern with this paper is about the experiments that are only based on simulated agents, as it is the case for learning. While it can be questionable for learning (but we understand why it is difficult to overcome), it is very problematic for the experiments to not have anything that demonstrates the usability of the approach in a real-world scenario. I have serious doubts about the performances of such an artificially learned approach for achieving real-world search tasks. Also, for me the experimental section is not sufficiently detailed, which lead to not reproducible results. Moreover, authors should have considered baselines (only the two proposed agents are compared which is clearly not sufficient). 

Also, both models have some issues from my point of view. First, the Q-learning methods looks very complex: how could we expect to get an accurate model with 10^7 states ? No generalization about the situations is done here, examples of trajectories have to be collected for each individual considered state, which looks very huge (especially if we think about the number of possible trajectories in such an MDP). The second model is able to generalize from similar situations thanks to the neural architecture that is proposed. However, I have some concerns about it: why keeping the history of actions in the inputs since it is captured by the LSTM cell ? It is a redondant information that might disturb the process. Secondly, the proposed loss looks very heuristic for me, it is difficult to understand what is really optimized here. Particularly, the loss entropy function looks strange to me. Is it classical ? Are there some references of such a method to maintain some exploration ability. I understand the need of exploration, but including it in the loss function reduces the interpretability of the objective (wouldn't it be preferable to use a more classical loss but with an epsilon greedy policy?).


Other remarks: 
   - In the begining of "varying memory capacity" section, what is "100, 150 and 250" ? Time steps ? What is the unit ? Seconds ?   
   - I did not understand the "Capturing seach context at local and global level" at all
   - In the loss entropy formula, the two negation signs could be removed
  

review_title: An interesting problem but a not convincing experimental protocol

final_score: rating: 3: Clear rejection
review_content: The paper describes reinforcement learning techniques for digital asset search.  The RL techniques consist of A3C and DQN.  This is an application paper since the techniques described already exist.  Unfortunately, there is a lack of detail throughout the paper and therefore it is not possible for someone to reproduce the results if desired.  Since there is no corpus of message response pairs to train the model, the paper trains a simulator from logs to emulate user behaviours.  Unfortunately, there is no description of the algorithm used to obtain the simulator.  The paper explains that the simulator is obtained from log data, but this is not sufficient.  The RL problem is described at a very high level in the sense that abstract states and actions are listed, but there is no explanation about how those abstract states are recognized from the raw text and there is no explanation about how the actions are turned into text.  There seems to be some confusion in the notion of state.  After describing the abstract states, it is explained that actions are selected based on a history of states.  This suggests that the abstract states are really abstract observations.   In fact, this becomes obvious when the paper introduces the RNN where a hidden belief is computed by combining the observations.  The rewards are also described at a hiogh level, but it is not clear how exactly they are computed.  The digital search application is interesting, however a detailed description with comprehensive experiments are needed for the publication of an application paper.
review_title: lack of details

Meta-review: meta score: 4

This paper is primarily an application paper applying known RL techniques to dialogue.    Very little reference to the extensive literature in this area.

Pros:
 - interesting application (digital search)
 - revised version contains subjective evaluation of experiments

Cons:
 - limited technical novelty
 - very weak links to the state-of-the-art, missing many key aspects of the research domain

Decision: Reject