Abstract: Seeking information about products and services is an important activity of online consumers before making a purchase decision. Inspired by recent research on conversational reading comprehension (CRC) on formal documents, this paper studies the task of leveraging knowledge from a huge amount of reviews to answer multi-turn questions from consumers or users. Questions spanning multiple turns in a dialogue enables users to ask more specific questions that are hard to ask within a single question as in traditional machine reading comprehension (MRC). In this paper, we first build a dataset and then propose a novel task-adaptation approach to encoding the formulation of CRC task into a pre-trained language model. This task-adaptation approach is unsupervised and can greatly enhance the performance of the end CRC task that has only limited supervision. Experimental results show that the proposed approach is highly effective and has competitive performance as supervised approach. We plan to release the datasets and the code in May 2019.

Introduction: Seeking information to assess whether some products or services suit one's needs is a vital activity for consumer decision making. In online businesses, one major hindrance is that customers have limited access to answers to their specific questions or concerns about products and user experiences. Given the ever-changing environment of products and services, it is very hard, if not impossible, to pre-compile an up-to-date knowledge base to answer user questions as in KB-QA BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . As a compromise, community question-answering (CQA) BIBREF4 is leveraged to enable existing customers or sellers to answer customer questions. However, one obvious drawback of this approach is that many questions are not answered, and even if they are answered, the answers and the following up questions are delayed, which is not suitable for interactive QA. Although existing studies have used information retrieval (IR) techniques BIBREF4 , BIBREF5 to identify a whole review as an answer to a question, it is time-consuming to read a whole review and the approach has difficulty to answer questions in multiple turns. Inspired by recent research in Conversational Reading Comprehension (CRC) BIBREF6 , BIBREF7 , we explore the possibility of turning reviews as a source of valuable knowledge of experiences and to provide a natural way of answering customers' multiple-turn questions in a dialogue setting. The conversational setting of machine reading comprehension (MRC) enables more specific questions and allow customers to either omit or co-reference information in context. As an example in a laptop domain shown in Table 1 , a customer may have 5 turns of questions based on the context. The customer first has an opinion question targeting an aspect “retina display” of a to-be-purchased laptop. Then the customer carries (and omit) the question type opinion from the first question to the second and continually asking the second aspect “boot-up speed”. For the third question, the customer carries the aspect of the second question but change the question type to opinion explanation. Later, the customer can co-reference the aspect “SSD” from the previous answer and ask for the capacity (a sub-aspect) of “SSD”. Unfortunately, there is no answer in this review for the fourth question so the review may say “I don't know”. But the customer can still ask other aspects as in the fifth question. We formally define this problem as follows and call it review conversational reading comprehension (RCRC). Problem Definition: Given a review that consists of a sequence of $n$ tokens $d=(d_1, \dots , d_n)$ , a history of past $k-1$ questions and answers as the context $C=(q_1, a_1, q_2, a_2, \dots , q_{k-1}, a_{k-1})$ and the current question $q_k$ , find a sequence of tokens (a textual span) $a=(d_s, \dots , d_e)$ in $d$ that answers $q_k$ based on $C$ , where $1 \le s \le n$ , $d=(d_1, \dots , d_n)$0 , and $d=(d_1, \dots , d_n)$1 , or return NO ANSWER ( $d=(d_1, \dots , d_n)$2 ) if the review does not contain any answer for $d=(d_1, \dots , d_n)$3 . RCRC is a novel QA task that requires the understanding of both the current question $q_k$ and dialogue context $C$ . Compared to the traditional single-turn MRC, the key challenge is how to understand the context $C$ and the current question $q_k$ given it may have a co-reference resolution or context carryover. To the best of our knowledge, there are no existing review datasets for RCRC. We first build a dataset called $(\text{RC})_2$ based on laptop and restaurant reviews from SemEval 2016 Task 5. We choose this dataset to better align with existing research on review-based tasks in sentiment analysis. Each review is annotated with a few dialogues focusing on some topics. Note that although one dialogue is annotated on a single review, a trained RCRC model can potentially be deployed among an open set of reviews BIBREF8 where the context may potentially contain answers from different reviews. Given the wide spectrum of domains in online business (e.g., thousands of categories on Amazon.com) and the prohibitive cost of annotation, $(\text{RC})_2$ is designed to have limited supervision as in other tasks of sentiment analysis. We adopt BERT (Bidirectional Encoder Representation from Transformers BIBREF9 ) as our base model since its variants achieve dominant performance on MRC BIBREF10 , BIBREF11 and CRC BIBREF6 tasks. However, BERT is designed to learn features for a wide spectrum of NLP tasks with a large amount of training examples. The task-awareness of BERT can be hindered by the weak supervision of the $(\text{RC})_2$ dataset. To resolve this challenge, we introduce a novel pre-tuning stage between pre-training and end-task fine-tuning for BERT. The pre-tuning stage is formulated in a similar fashion as the RCRC task but requires no annotated RCRC data and just domain QA pairs (from CQA) and reviews, which are readily available online BIBREF4 . We bring certain characteristics of the RCRC task (inputs/outputs) to pre-tuning to encourage BERT's weight to be prepared for understanding the current question and locate the answer if there exists one. The proposed pre-tuning step is general and can potentially be used in MRC or CRC tasks in other domains. The main contributions of this paper are as follows. (1) It proposes a practical new task on reviews that allows multi-turn conversational QA. (2) To address this problem, an annotated dataset is first created. (3) It then proposes a pre-tuning stage to learn task-aware representation. Experimental results show that the proposed approach achieves competitive performance even compared with the supervised approach on a large-scale training data.

Question: What is the baseline model used?
Answer: The baseline models used are DrQA modified to support answering no answer questions, DrQA+CoQA which is pre-tuned on CoQA dataset, vanilla BERT, BERT+review tuned on domain reviews, BERT+CoQA tuned on the supervised CoQA data