Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper incorporates an answer inspection encoder verifying whether the answers selected by the Machine Reading Comprehension (MRC) component is valid, into the MRC reader. For training, the model includes two additional losses and trained jointly. Evaluation is on SQuAD V2.0 which is constructed from Wikipedia and contains unanswerable questions generated by crowd sources. The approaches are verified in the settings with/without including BERT to show the generalization of the answer inspector. 

This is an interesting paper which focuses on the answer verification and validation, and shows the effectiveness of the proposed model. It also gives a good ablation study showing the contributions of each component, and provides examples to illustrate why it works. 

However, there are a few concerns detailed as follows:

1. The model is only evaluated on SQuAD 2.0, which is over explored by many works. I’m wondering if this could be generalized to other MRC tasks, e.g. MSMARCO or DuReader. It would be nice to see some experiments on them or other datasets.

2. It seems that the performance of state-of-the-art SOTA system on SQuAD is much higher than the proposed approaches. I would like to see some discussion on what are the pros and cons between them.

3. How sensitive are the gammas in Eq 1? 

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: 
Paper Summary:

This paper proposes a neural question requirement inspection models called NeurQuRI.  It is different from existing answer verifiers in that NeurQuRI pinpoints where the mismatch occurs between the question and the candidate answer in unanswerable cases. Experiments with SQuAD 2.0 show the effectiveness of NeurQuRI.

Strengths:

—NeurQuRI improved the accuracy of three popular reading comprehension models, which are BERT, DocQA and QANet, on SQuAD 2.0.

—NeurQuRI requires 33M additional model parameters to check answerability of the candidate answer.  It is quite less than those of the previous answer verifier (150M) proposed by Hu et al. (2019).

—NeurQuRI can be easily extended to other question answering models and tasks.

Weaknesses:

—The authors do not experimentally compare their model to the answer verifier proposed by Hu et al. (2019). The base model of Hu et al. (2019), RMR, is different from the base models used in this paper, BERT, DocQA, and QANet.

—The authors use only one dataset (SQuAD 2.0) to evaluate their method.  NewsQA, MS MARCO, CoQA and QuAC also contain unanswerable questions.

Comments/Suggestions:

—I think that coverage mechanisms used in NMT (Tu et al., 2016) and summarization (See et al., 2017) should be included in the section of related work.

—The idea similar to NeurQuRI is used in a multi-hop QA model, proposed by Nishida et al. (2019), to find the evidence sentences that cover important information with respect to the question sentence.

—It is worthwhile to present the results of ablation tests with respect to the modified answerability label (described in Section 2.2).

Review Summary:

The paper is well motivated.  However, I think the authors need to compare NeurQuRI with the answer verifier proposed by Hu et al. (2019).  Also, I think this paper can benefit a lot with a more comprehensive analysis with other datasets such as CoQA.

References:
Hu et al.: Read + Verify: Machine Reading Comprehension with Unanswerable Questions. AAAI 2019: 6529-6537
Tu et al.: Modeling Coverage for Neural Machine Translation. ACL (1) 2016
See et al.: Get To The Point: Summarization with Pointer-Generator Networks. ACL (1) 2017: 1073-1083
Nishida et al.: Answering while Summarizing: Multi-task Learning for Multi-hop QA with Evidence Extraction. ACL (1) 2019: 2335-2345

EDIT Nov. 20, 2019: 
I appreciate the authors' revision.  
The revision has satisfied my concerns, and I decided to increase the score of the paper (weak reject -> weak accept).
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: The authors propose a Neural Question Requirement module that extracts a list of condition from the question which should be met by the candidate answer in the question answering problems. Authors claim that they propose a novel, attention-based loss function in order to check whether a condition is met.

Evaluation of the methodology is performed on the SquAD 2.0 dataset. It is organized behind combining existing, state of the art solutions like BERT  or QANet. It is improving these solutions consistently.
review_title: Official Blind Review #3

Meta-review: This paper extracts a list of conditions from the question, each of which should be satisfied by the candidate answer generated by an MRC model. All reviewers agree that this approach is interesting (verification and validation) and experiments are solid. One of the reviewers raised concerns are promptly answered by authors, raising the average score to accept. 

Decision: Accept (Poster)