Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: In this paper, two linguistic verifiers are proposed to improve the model performance on machine reading comprehension datasets, such as SQuAD v2, NewsQA and TriviaQA. The first verifier rewrites the question by replacing its interrogatives with the predicted answer phrases. Then it computes a score between the rewritten question and the context, so that the answer candidates are position-sensitive. The second verifier leverages a hierarchical attention network, so that the long context can be split in to shorter segments, which are then recurrently connected to conduct answerability classification and boundary determination. 

The Empirical results of this proposed method is very strong. Apparently, it achieves a new state-of-the-art performance on the dev set of SQuAD v2. It also outperforms the a bunch of strong baseline methods on the NewsQA dataset. Finally, the proposed model also exceeds the BERT model on TriviaQA.

Overall, it is a good paper.

However, I have some comments:

1. In table 2, what does “Regular Track” mean? 

2. In your tables, could you separate the ensemble methods and the single models? It would be much easier to draw a fair comparison.

3. Are you results achieved by ensemble or a single model?

4. You used Albert-xxlarge, but some methods in the tables used smaller pretrained models. For example, in TriviaQA, your baseline is BERT-Large. It might be a bit hard to tell if the improvement is obtained by a better CLM or the proposed modules. So could you do a fair ablation study to verify this?

5. The font of figure 1 looks weird and a bit ugly. I suggest the authors make it more reader friendly. 

6. Will you submit your model to the SQuAD v2 leaderboard? I am very interested in seeing its performance on the test set. And I am willing to raise my score if the result aligns with that of dev set (I expect it will top the leaderboard).

***********************************
Post rebuttal: The author has addressed most of my questions, and the SQuAD v2 test result is on par with the state-of-the-art, partially indicating the proposed method is effective. So I am happy to increase my rating and champion for the acceptance.

review_title: New modules with strong empirical results on MRC

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes two types of linguistic verifiers for machine reading comprehension task in span extraction form. One is a rewritten question oriented verifier that checks the linguistic correctness of the extracted answers, and the other is based on a hierarchical attention network for answerability classification and boundary determination. The two verifiers are trained independently and then combined together via interpolation. Overall, the paper is well organized and easy to follow.

Reasons to accept the paper:
1. The rewritten question oriented verifier could improve the linguistic correctness of the extracted answers.
2. The HAN-based verifier considers the entire document instead of each segment independently, which may enable general transformer-based models to handle long-text document.

Reasons to reject the paper:
1. Some important baselines are not included in the experimental analysis, such as GPT-3 Few-Shot [1] on TriviaQA which achieves 71.2, and RAG [2] on TriviaQA which achieves 68.0.
2. Some important details are missing in the experimental analysis. For example, in Table 2, it is not clear what "DA Verifier" means, and which verifier is used in the method "ALBERT + verifier".
3. It is not clearly discussed the additional computational time and cost spent to train the two proposed verifiers, compared to the baseline without verifiers. For a new method that has marginal performance gain, the extra computational cost should be considered.
4. The illustration of HAN-based verifier in Fig. 1(b) is not complete, which should have included the part for answer prediction and verification loss, etc.


References

[1] Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., ... & Agarwal, S. (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165.

[2] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., ... & Riedel, S. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. arXiv preprint arXiv:2005.11401.

review_title: Official Blind Review #2

final_score: rating: 5: Marginally below acceptance threshold
review_content: MACHINE READING COMPREHENSION WITH ENHANCED LINGUISTIC VERIFIERS

The authors propose two linguistic verifiers for improving extractive question answering performance when the question is answerable. The first replaces interrogatives in the question (who etc.) with candidate answers and evaluates this both in isolation and in combination with the answer-containing sentence to do answer verification. The second verifier jointly encodes individual sentences and spans with questions in a hierarchical manner to improve answer prediction performance. Solid gains on Squad, NewsQA, and TriviaQA are reported for both methods when applied in isolation, and in combination.

Strengths:

- The techniques are sound and lead to solid gains on 3 benchmark datasets.
- The approaches, while relatively straightforward, illustrate that explicit verification and hierarchical evaluation continue to improve application results, despite the high capacity and efficacy of the SOTA deep architectures.

Limitations:

- The paper is understandable but the presentation could be significantly improved. Figure 1a for example, is a bit overwelming, and should probably be replaced with something more focused, and moved into supplementary material. Several sentences I couldn't understand, for example "Minimizing span losses of start and end positions of answers for answerable questions is overwhelming in current pretraining+fine-tuning frameworks." Overall I feel that the paper could use some additional polishing.
- A similar hierarchical (HAN) approach was previously proposed for verifying unanswerable questions, but their approach for answerable questions appears to be more effective.
- The paper has lower novelty wrt ML elements. The component architectures/models that make up their system are well established.
- The replacing of interrogatives with the answer and the associated rules for doing so feel like they have somewhat limited scope (e.g. factoid questions, single interrogative questions, etc.). When there is more than one interrogative, the authors back off to simply appending the answer to the question... perhaps this can be done all the time without compromising the performance gains? 
- Verification (esp. for the HAN verifier, where extra forward passes are done for each sentence and sub-paragraph) is more more computationally demanding, but this is not discussed.

Overall Assessment:

A solid applications paper on extractive question answering. However, I feel that the paper is perhaps better suited for an NLP-application focused audience (e.g. NAACL, deadline approaching), since the results are strong, but the paper has lower novelty wrt core ML. Furthermore, the manuscript is in need of significant revision before it can be considered for acceptance at ICLR.

quality 5/10 (+results on multiple benchmark datasets, -manuscript needs substantial revision)
clarity 5/10 (+understandable for the most part, -manuscript/figures not clear in many places)
originality 6/10 (+novel approaches to QA verification, -lower novelty wrt ML elements)
significance 6/10 (+strong QA results, +demonstrates value in explicit verification/hierarchical processing in DL applications, -perhaps more suitable for an NLP-applications focused audience)
overall (5)

Post-rebuttal:

Authors, thank you for your feedback. The additional results around relative speed and performance have strengthened the paper. However, I still feel that the paper still needs significant polishing before final publication (figures, grammar, presentation), and that the paper is better suited for an NLP-focused conference, and so I have not updated my final score.
review_title: Good results, demonstrates the value of explicit verification/hierarchies in DL, lower novelty wrt ML elements, manuscript needs significant revision

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work addresses two main challenges of span-extraction style machine reading comprehension (MRC) tasks: how to evaluate the syntactic completeness of predicted answers and how to utilize the rich context of long documents. To handle such challenges, Question Rewritten Verifier (QRV) and Hierarchical Attention Network (HAN) are proposed respectively. The former uses a question-rewritten method to replace the interrogatives in the question with the answer span for further verification. The latter adopts a hierarchical multi-granularity (sentence, segment, and paragraph) cross-attention to extract and utilize the context information of long paragraphs. Compared with the strong baselines, both the verifiers and their combination achieved relatively significant accuracy improvement on three mainstream span-extraction MRC tasks: SQuAD2.0, NewsQA, and TriviaQA.

-------------------------------------------
Strengths:

1. The idea of bringing the answer back to the question for further validation is sound and it is reasonable for humans to do this process to verify the candidate answer in real-world practice. 

2. The question rewritten strategy is simple and effective, which brings improvements. HAN also handles the problem of long sequence well.

3. The overall method achieves state-of-the-art results. The significance test shows significant improvements over baselines.

-------------------------------------------
Weaknesses:

1. The design of the training target (loss) in QRV is complex and not interpretable enough. There are many loss functions. How about their contributions to the final performance?

2. There is no test result reported for SQuAD2.0, though it is possible to obtain the results without making it public. Therefore, the clarity, “Due to anonymous issues, we have not submitted our results in an anonymous way to obtain results on the hidden test set.”, is not quite convincing.

3. The improvement of accuracy is mainly reflected in the questions of HasAns, which has no obvious contribution to the recognition accuracy of NoAns, which is one of the main challenges of the current MRC tasks.

-------------------------------------------
Questions:

1. (Section 1 page 2 line 26) The paragraphs are divided into segments, with fixed length (e.g.,512 tokens with strides such as 128 or 256) and then divides the segment into sentences. So when dividing the paragraph, what if the dividing point is in the middle of a sentence? Would the incomplete sentence be discard？If not, how to further divide the segment to sentence level? Further clarification of the process would be beneficial.

2. (Section 2.1 page 3 line 8) When failing to find the alignment, the answer text is attached at the left-hand side of the question. It obviously damages the sentence structure. So will this affect the judgment of the model in the following process? In another word, would it have an impact on the performance of the final model (Increase or decrease) if question-written including subsequent loss calculations were not done on such questions?

3. (Section 2.2 page 4 line 17) Multiple losses are employed, but the paper did not distinguish the practical effectiveness of each loss. My concern is whether each of the objectives is necessary since the experiment results in Table 3 has verified that  $l’_{3}$ does not significantly improve the accuracy of the model. Would the authors further verify the contribution of other losses to model performance (except for apparently indispensable l1 and l2)?

4. (Figure 1 (b)) Do the tunable CLMs of sentence-level and segment level share parameters? Besides, may the authors list the number of parameters of each model (QRV, HAN, and Combination)?

----------------------------------
Minor Issues:

The citation format is not consistent, please check the usage of \citep{} and \citet{}.

review_title: Official Blind Review #1

Meta-review: The authors propose two linguistic verifiers for improving extractive question answering when the question is answerable. The first replaces the interrogative in the question with candidate answers and evaluates the result both in isolation and in combination with the answer-containing sentence to do answer verification. The second jointly encodes individual sentences and spans with questions in a hierarchical manner to improve use of context in answer prediction performance.

The reviews for this paper are roughly on the cusp: 2 reviewers rate the paper a bit below the acceptance threshold, 1 a bit above, and then 1 now rates the paper as a solid Accept. 

Pros

- The main strength of the paper, certainly as emphasized by the most positive reviewer is the strong empirical results. Especially on SQuAD v2, the method here seems to roughly equal the current leading system on the leaderboard.
- The paper also proposes two methods for improving question answering that make sense, are relatively simple, and work

Cons
- The writing and presentation of the paper is not that great. Even at the level of the introduction, the writing just is not very focused: The first page has a lot of background and tutorial information on MRC that just doesn't get to the point of where this paper is situated and what it contributes.
- Neither of the proposed systems are that novel (though it is interesting to see that they still have value even in the age of large contextual language models)
- The paper lacks ML novelty
- The methods appear to be significantly more expensive to run
- Some empirical comparisons appear to be lacking

As well as the missing comparisons mentioned by some reviewers, I think that there are a number of other missing relevant datapoints. While not denying that gathering the available results for NewsQA/TriviaQA is much less straightforward than with that nice leaderboard for SQuAD, aren't there are lot of systems with better results on TriviaQA that aren't mentioned in the paper. These include: RoBERTa and SpanBERT (mandarjoshi); BigBird-ETC see https://proceedings.neurips.cc/paper/2020/file/c8512d142a2d849725f31a9a7a361ab9-Paper.pdf; Longformer; SLQA see https://www.aclweb.org/anthology/P18-1158.pdf .

But, overall, I think the decision on this paper comes down to focus and contributions. Not withstanding the growing size of ICLR, I would like to think that it is not just another ML and ML applications conference, but it is a conference centered on representation learning. The present paper, no matter its quality and strong results, just isn't a contribution to representation learning. It is a much better fit to an NLP conference where it would be a strong contribution to question answering, showing the continuing value of linguistic methods like question rewriting in answer validation. But this just isn't a contribution within the focus of representation learning. Just as R4 does, I encourage the authors to clean up the presentation of the paper a bit and to submit it to an NLP conference, where it would be a strong contribution, for the reasons that R3 emphasizes.

Decision: Reject