Venue: AKBC 2019 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary

This paper addresses the ARC dataset by reformulating the question using embeddings from ConceptNet. Their model selects a few terms from the question using the embeddings from ConceptNet, rewrites the query based on the selected terms, retrieves the documents and solves the query. The empirical result shows that embeddings from ConceptNet is beneficial, and the overall result is comparable to recent performance on ARC dataset.

Quality
pros
1) This paper contains a thorough study of recent QA models and datasets.
2) This paper describes the model architecture, conducts ablation studies of different Essential terms classification, and includes thorough comparisons with recent models on ARC challenges.

cons
- Although the paper includes recent works on QA models/datasets, it doesn’t contain much studies on query reformulations. For example,  "Ask the Right Questions: Active Question Reformulation with Reinforcement Learning” (Buck et al., ICLR 2018) is one of the related works that the paper didn’t cite.
- The paper does not have any example of reformulated queries or error analysis.

Clarity

pros
1) The paper describes the framework and model architecture carefully.

cons
1) It is hard to understand how exactly they reformulate the query based on selected terms. (I think examples would help) For example, in Fig 2, after “activities”, “used”, “conserve” and “water” were selected, how does rewriter write the query? The examples will help.
2) Similar to the above, it would be helpful to see the examples of decision rules in Section 5.2.
3) It is hard to understand how exactly each component of the model was trained. First of all, is rewrite module only trained on Essential Terms dataset (as mentioned in Section 3.1.3) and never fine-tuned on ARC dataset? Same question for entailment modules: is it only trained on SciTail, not fine-tuned on ARC dataset? How did decision rules trained? Are all the modules trained separately, and haven’t been trained jointly? What modules were trained on ARC dataset? All of these are a bit confusing since there’re many components and many datasets were used.

Originality & significance

pros
* Query reformulation methods have been used on several QA tasks (like Buck et al 2018 above), and incorporating background knowledge has been used before too (as described in the paper), but I think it’s fairly original to do both in the same time.

cons
* It is a bit disappointing that the only part using background knowledge is selecting essential terms using ConceptNet embedding. I think the term “using background knowledge” is too general term for this specific idea.

In general, I think the paper has enough contribution to be accepted, if some descriptions are better clarified.
review_title: Good paper, some clarifications needed

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper focuses on the recently introduced ARC Challenge dataset, which contains 2,590 multiple choice questions authored for grade-school science exams. The paper presents a system that reformulates a given question into queries that are used to retrieve supporting text from a large corpus of science-related text. The rewriter is able to incorporate background knowledge from ConceptNet. A textual entailment system trained on SciTail that identifies support in the retrieved results.  Experiments show that the proposed system is able to outperform several baselines on ARC.

* (Sec 2.2)  "[acl-2013] Paraphrase-driven learning for open question answering" and "[emnlp-2017] Learning to Paraphrase for Question Answering" can be added in the related work section.
* (Sec 3.1) Seq2seq predicts 0 and 1 to indicate whether the word is salient. A more straightforward method is using a pointer network for the decoder, which directly selects words from the input. This method should be more effective than seq2seq used in Sec 3.1.1.
* (Sec 3.1) How about the performance of removing the top crf layer? The LSTM layer and the classifier should play the most important role.
* How to better utilize external resources is an interesting topic and is potentially helpful to improve the results of answering science exam questions. For example, the entailment module described in Sec 5.1 can be trained on other larger data, which in turn helps the problem with smaller data. I would like to see more details about this.
* Are the improvements significant compared to the baseline methods? Significance test is necessary because the dataset is quite small.
* Experiments on large-scale datasets are encouraged.
review_title: good qa paper

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces an end-to-end system to answer science exam questions for the ARC challenge. The system is a combination of several existing techniques, including (i) query rewriting based on seq2seq or NCRF++, (ii) answer retriever, (iii) entailment model based on match-LSTM, and (iv) knowledge graph embeddings. The description of the system is clear, and there is abundant ablation study. However, I have following concerns about this paper:

1. There seems to be no new techniques proposed in the system. Hence, the novelty of this work is questioned.
2. I do not understand why the authors use TransH, which is a KG embedding model that differentiates one entity into different relation-specific representations.
3. The system is significantly outperformed by Sun et al. 2018 and Ni et al. 2018.
review_title: The system is a combination of many existing techniques, and is outperformed by several works.

Meta-review: metareview: This paper presents a method for finding important tokens in the question, and then use prior knowledge from conceptNet to answer questions in the ARC dataset. 

Pros. 
Combining finding essential terms, using domain knowledge, and textual entailment. 

Cons. 
None of the proposed methods are novel. 
The paper combines a few components to answer questions in ARC; the intro and abstract are written a bit more general than what the paper actually does.
The paper studies different ways to incorporate concept net; the results between dev and test are not consistent. Some methods achieve better results on the dev set, but the authors use a model (which is not best in dev) to be used on the test set. 

 
confidence: 3: The area chair is somewhat confident
Decision: Accept (Poster)