Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents an architecture for answer extraction task and evaluates on the SQUAD dataset. The proposed model builds fixed length representations of all spans in the answer document based on recurrent neural network. It outperforms a few baselines in exact match and F1 on SQUAD.

It is unfortunate that the blind test results are not obtained yet due to the copyright issue. There are quite a few other systems/submissions on the SQUAD leader board that were available for comparison.

Given that there's no result on the test set reported, the grid search for hyperparameters on the dev set directly is also a concern, even though the authors did cross validation experiments.

review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors proposed RASOR to address the problem of finding the best answer span according to a given question. The focus of the paper is mainly on how to model the relationship between question and the answer spans. The idea proposed by this paper is reasonable, but not ground breaking. The analysis is interesting and potentially useful. I would hope the authors can go extra miles to analyze different choices of boundary prediction models and make a more convincing case for the necessity of modeling the score of the span globally.

The main idea behind RASOR is to globally normalize and rank the scores of the possible answer spans. RASOR is able to achieve this by first modeling the hidden vectors of all words with LSTMs. Then, the representation of a text span is formed by concatenating the corresponding hidden vectors of the start and the end word of the corresponding chunk. The approach is reasonable, but not earth shattering. Also, the table 6 shows that the improvement over end-prediction point is not very large.

I appreciate the fact that authors conduct several analysis experiments as some of them are quite interesting. For example, it seems that question independent representation is also very import to the performance. In addition to the current analysis, I also want to get a clear idea on what makes the current model be better than the Match-LSTM. Is it hyper-parameter tuning? Or it is due to the use of the question independent representation?

Another good thing about the proposed model is that it is relatively simple, so there is a chance that the proposed techniques can be combined with other newly proposed ones. 


review_title: Review

final_score: rating: null
review_content: 1.	I found the descriptions of passage independent representation a bit confusing. What is word_q in equation 10? Why do we need to use attention structure here?
2.	In page 6, the authors mention that the reason why Wang & Jiang’ model does not perform so well is due to “both training and evaluation are greedy, making their system susceptible to search errors when decoding.” However, the endpoints version of RASOR seems also to be greedy, but it seems to perform pretty well, as it outperformed Wang & Jiang (2016) already. Could the authors clarify this point for me?

review_title: Pre-review questions

final_score: rating: 7: Good paper, accept
review_content: This paper proposes RaSoR, a method to efficiently representing and scoring all possible spans in an extractive QA task. While the test set results on SQuAD have not been released, it looks likely that they are not going to be state-of-the-art; with that said, the idea of enumerating all possible spans proposed in this paper could potentially improve many architectures. The paper is very well-written and the analysis/ablations in the final sections are mostly interesting (especially Figure 2, which confirms what we would intuitively believe). Based on its potential to positively impact other researchers working on SQuAD, I recommend that the paper is accepted. 
review_title: review

Meta-review: The program committee appreciates the authors' response to concerns raised in the reviews. Unfortunately, most reviewers are not leaning sufficiently towards acceptance. In particular, it is unfortunate that authors can not evaluate their model on the leaderboard due to copyright issues. The role of standard datasets and benchmarks is to allow for meaningful comparisons. Evaluation on non-standard splits defeats this purpose. Fortunately, sounds like authors are working on getting their model evaluated on the leaderboard. Resolving that and incorporating reviewers' feedback will help make the paper stronger.
Decision: Reject