Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors propose an approach where they aggregate, for each candidate answer, text from supporting passages. They make use of two ranking components. A strength-based re-ranker captures how often a candidate answer would be selected while a coverage-based re-ranker aims to estimate the coverage of the question by the supporting passages. Potential answers are extracted using a machine comprehension model. A bi-LSTM model is used to estimate the coverage of the question. A weighted combination of the outputs of both components generates the final ranking (using softmax). 
This article is really well written and clearly describes the proposed scheme. Their experiments clearly indicate that the combination of the two re-ranking components outperforms raw machine comprehension approaches. The paper also provides an interesting analysis of various design issues. Finally they situate the contribution with respect to some related work pertaining to open domain QA. This paper seems to me like an interesting and significant contribution.

review_title: Very good contribution on multi-sentences answer reranking with significant experimental results.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper is clear, although there are many English mistakes (that should be corrected).
The proposed method aggregates answers from multiple passages in the context of QA. The new method is motivated well and departs from prior work. Experiments on three datasets show the proposed method to be notably better than several baselines (although two of the baselines, GA and BiDAF, appear tremendously weak). The analysis of the results is interesting and largely convincing, although a more dedicated error analysis or discussion of the limitation of the proposed approach would be welcome.

Minor point: in the description of Quasar-T, the IR model is described as lucene index. An index is not an IR model. Lucene is an IR system that implements various IR models. The terminology should be corrected here. 

review_title: This is a neural-based approach for improving QA systems by aggregating answers from multiple passages.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Traditional open-domain QA systems typically have two steps: passage retrieval and aggregating answers extracted from the retrieved passages.  This paper essentially follows the same paradigm, but leverages the state-of-the-art reading comprehension models for answer extraction, and develops the neural network models for the aggregating component.  Although the idea seems incremental, the experimental results do seem solid.  The paper is generally easy to follow, but in several places the presentation can be further improved.

Detailed comments/questions:
  1. In Sec. 2.2, the justification for adding H^{aq} and \bar{H}^{aq} is to downweigh the impact of stop word matching.  I feel this is a somewhat indirect and less effective design, if avoiding stop words is really the reason.  A standard preprocessing step may be better.
  2. In Sec. 2.3, it seems that the final score is just the sum of three individual normalized scores. It's not truly a "weighted" combination, where the weights are typically assumed to be tuned.
  3. Figure 3: Connecting the dots in the two subfigures on the right does not make sense.  Bar charts should be used instead.
  4. The end of Sec. 4.2: I feel it's a bad example, as the passage does not really support the answer. The fact that "Sesame Street" got picked is probably just because it's more famous.
  5. It'd be interesting to see how traditional IR answer aggregation methods perform, such as simple classifiers or heuristics by word matching (or weighted by TFIDF) and counting.  This will demonstrates the true advantages of leveraging modern NN models.

Pros:
  1. Updating a traditional open-domain QA approach with neural models
  2. Experiments demonstrate solid positive results

Cons:
  1. The idea seems incremental
  2. Presentation could be improved

review_title: Incremental idea but with solid results

Meta-review: The pros and cons of this paper cited by the reviewers can be summarized below:

Pros:
* Solid experimental results against strong baselines on a task of great interest
* Method presented is appropriate for the task
* Paper is presented relatively clearly, especially after revision

Cons:
* The paper is somewhat incremental. The basic idea of aggregating across multiple examples was presented in Kadlec et al. 2016, but the methodology here is different.

Decision: Accept (Poster)