Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The authors present a scalable model for questioning answering that is able to train on long documents. On the TriviaQA dataset, the proposed model achieves state of the art results on both domains (wikipedia and web). The formulation of the model is straight-forward, however I am skeptical about whether the results prove the premise of the paper (e.g. multi-mention reasoning is necessary). Furthermore, I am slightly unconvinced about the authors' claim of efficiency. Nevertheless, I think this work is important given its performance on the task.

1. Why is this model successful? Multi-mention reasoning or more document context?
I am not convinced of the necessity of multi-mention reasoning, which the authors use as motivation, as shown in the examples in the paper. For example, in Figure 1, the answer is solely obtained using the second last passage. The other mentions provide signal, but does not provide conclusive evidence. Perhaps I am mistaken, but it seems to me that the proposed model cannot seem to handle negation, can the authors confirm/deny this? I am also skeptical about the computation efficiency of a model that scores all spans in a document (which is O(N^2), where N is the document length). Can you show some analysis of your model results that confirm/deny this hypothesis?

2. Why is the computational complexity not a function of the number of spans?
It seems like the derivations presents several equations that score a given span. Perhaps I am mistaken, but there seems to be n^2 spans in the document that one has to score. Shouldn't the computational complexity then be at least O(n^2), which makes it actually much slower than, say, SQuAD models that do greedy decoding O(2n + nm)?

Some minor notes
- 3.3.1 seems like an attention computation in which the attention context over the question and span is computed using the question. Explicitly mentioning this may help the reading grasp the formulation.
- Same for 3.4, which seems like the biattention (Seo 2017) or coattention (Xiong 2017) from previous squad work.
- The sentence "We define ... to be the embeddings of the l words of the sentence that contains s." is not very clear. Do you mean that the sentence contains l words? It could be interpreted that the span has l words.
- There is a typo in your 3.7 "level 1 complexity": there is an extra O inside the big O notation.
review_title: Intuitive model for scaling question answering 

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a method that scales reading comprehension QA to large quantities of text with much less document truncation than competing approaches. The model also does not consider the first mention of the answer span as gold, instead formulating its loss function to incorporate multiple mentions of the answer within the evidence. The reported results were state-of-the-art(*) on the TriviaQA dataset at the time of the submission deadline. It's interesting that such a simple model, relying mainly on (weighted) word embedding averages, can outperform more complex architectures; however, these improvements are likely due to decreased truncation as opposed to bag-of-words architectures being superior to RNNs. 

Overall, I found the paper interesting to read, and scaling QA up to larger documents is definitely an important research direction. On the other hand, I'm not quite convinced by its experimental results (more below) and the paper is lacking an analysis of what the different sub-models are learning. As such, I am borderline on its acceptance.

* The TriviaQA leaderboard shows a submission from 9/24/17 (by "chrisc") that has significantly higher EM/F1 scores than the proposed model. Why is this result not compared to in Table 1? 

Detailed comments:
- Did you consider pruning spans as in the end-to-end coreference paper of Lee et al., EMNLP 2017? This may allow you to avoid truncation altogether. Perhaps this pruning could occur at level 1, making subsequent levels would be much more efficient.
- How long do you estimate training would take if instead of bag-of-words, level 1 used a biLSTM encoder for spans / questions?
- What is the average number of sentences per document? It's hard to get an idea of how reasonable the chosen truncation thresholds are without this.
- In Figure 3, it looks like the exact match score is still increasing as the maximum tokens in document is increased. Did the authors try truncating after more words (e.g., 10k)?
- I would have liked to see some examples of questions that are answered correctly by level 3 but not by level 2 or 1, for example, to give some intuition as to how each level works.
- "Krasner" misspelled multiple times as "Kramer"
review_title: review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a lightweight neural network architecture for reading comprehension, which 1) only consists of feed-forward nets; 2) aggregates information from different occurrences of candidate answers, and demonstrates good performance on TriviaQA (where documents are generally pretty long).

Overall, I think it is a nice demonstration that non-recurrent models can work so well, but I also don’t find the results strikingly surprising. It is also a bit hard to get the main takeaway messages. It seems that multi-loss is important (highlight that!), summing up multiple mentions of the same candidate answers seems to be important (This paper should be cited: Text Understanding with the Attention Sum Reader Network https://arxiv.org/abs/1603.01547). But all the other components seem to have been demonstrated previously in other papers. 

An important feature of this model is it is easier to parallelize and speed up the training/testing processes. However, I don’t see any demonstration of this in the experiments section.

Also, I am a bit disappointed by how “cascades” are actually implemented. I was expecting some sophisticated ways of combining information in a cascaded way (finding the most relevant piece of information, and then based on what it is obtained so far trying to find the next piece of relevant information and so on). The proposed model just simply sums up all the occurrences of candidate answers throughout the full document. 3-layer cascade is really just more like stacking several layers where each layer captures information of different granularity. 

I am wondering if the authors can also add results on other RC datasets (e.g., SQuAD) and see if the model can generalize or not. 

review_title: Official review

Meta-review: The authors did a good job addressing reviewer concerns and analyzing and  testing their model on interesting datasets with convincing results.
Decision: Accept (Poster)