Summary: The paper proposes a large-scale dataset for reading comprehension, with the final goal of releasing 1 million questions and answers. The authors have currently released 100,000 queries and their answers. The dataset differs from existing reading comprehension datasets mainly w.r.t queries being sampled from user queries rather than being generated by crowd-workers and answers being generated by crowd-workers rather than being spans of text from the provided passage. The paper presents some analysis of the dataset such as distribution of answer types. The paper also presents the results of some generative and some cloze-style models on the MS MARCO dataset.  Strengths:  1. The paper provides useful insights about the limitations of the existing reading comprehension datasets – questions asked by crowd-workers have different distribution compared to that of questions asked by actual users of intelligent agents, answers being restricted to span from the reading text rather than requiring reasoning across multiple pieces of text/passages.  2. MS MARCO dataset has novel useful characteristics compared to existing reading comprehension datasets – questions are sampled from user queries, answers are generated by humans.  3. The experimental evaluation of the existing baseline models on the MS MARCO dataset is satisfactory.  Weaknesses/Suggestions:  1. The paper does not report human performance on the dataset. Human performance should be reported to estimate the difficulty of the dataset. The degree of inter-human agreement will also reflect how well the metric (being used to compute inter-human agreement and accuracies of the baseline models) can deal with variance in the sentence structure with similar semantics.  2. I would like to see the comparison between the answer type distribution in the MS MARCO dataset and that in existing reading comprehension datasets such as SQuAD. This would ground the claim made in the paper the distributions of questions asked by crowd-workers is different from that of user queries.  3. The paper uses automatic metrics such as ROUGE, BLEU for evaluating natural language answers. However, it is known that such metrics poorly correlate with human judgement for tasks such as image caption evaluation (Chen et al., Microsoft COCO Captions: Data Collection and Evaluation Server, CoRR abs/1504.00325 (2015)). So, I wonder how authors justify using such metrics for evaluating open-ended natural language answers.  4. The paper mentions that a classifier was used to filter answer seeking queries from all Bing queries. It would be good to mention the accuracy of this classifier. This will provide insights into what percentage of the MS MARCO questions are answer seeking queries. Similarly, what is the accuracy of the information retrieval based system used to retrieve passages for filtered queries?  5. Please include the description of the best passage baseline in the paper.    6. Fix opening quotes, i.e. ” -> “ (for instance, on page 5, ”what” -> “what”).  Review Summary: The paper is well motivated, the use of user queries and human generated answers makes the dataset different from existing datasets. However, I would like to see the human performance on the dataset and quantitative comparison between the distribution of questions obtained from user queries and that of crowd-sourced questions. I would also like the authors to comment on the use of automatic metrics (such as ROUGE, BLEU) in the light of the fact that such metrics do not correlate well with human judgements for tasks such as image caption evaluation.
Output: en