SlowGuess/NAACL_2021_OCR huggingface dataset:
Guihong Cao, and Songlin Hu. 2020. Graph-based reasoning over heterogeneous external knowledge for commonsense question answering. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8449-8456. AAAI Press.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381-2391, Brussels, Belgium. Association for Computational Linguistics.
Peng Qi, Xiaowen Lin, Leo Mehr, Zijian Wang, and Christopher D. Manning. 2019. Answering complex open-domain questions through iterative query generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2590-2602, Hong Kong, China. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418-5426, Online. Association for Computational Linguistics.
Minjoon Seo, Jinhyuk Lee, Tom Kwiatkowski, Ankur Parikh, Ali Farhadi, and Hannaneh Hajishirzi. 2019. Real-time open-domain question answering with dense-sparse phrase index. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4430-4441, Florence, Italy. Association for Computational Linguistics.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pages 4444-4451. AAAI Press.
Haitian Sun, Tania Bedrax-Weiss, and William Cohen. 2019. PullNet: Open domain question answering with iterative retrieval on knowledge bases and text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the
9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2380-2390, Hong Kong, China. Association for Computational Linguistics.
Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William Cohen. 2018. Open domain question answering using early fusion of knowledge bases and text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4231-4242, Brussels, Belgium. Association for Computational Linguistics.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Computational Linguistics.
In this appendix, we show more details of our dataset construction (Appx. A), details of model implementation and experiments for reproducibility (Appx. B), and more related works (Appx. C). As we have submitted our code as supplementary material with detailed instructions for running baselines, we will skip some minor details here. We will make our code and data public after the anonymity period.
A Constructing OpenCSR Datasets
A.1 Reformatting Questions and Answers
In this section, we introduce how we reformat the existing three datasets and crowd-source annotations of multiple answers for evaluating OpenCSR. To convert a multiple-choice question to an open-ended question, we first remove questions where the correct answer does not contain any concept in  $\mathcal{V}$  and the few questions that require comparisons between original choices, as they are designed only for multiple-choice QA, e.g., "which of the following is the most ..." Then, we rephrase questions with long answers to be an open-ended question querying a single concept.
For example, an original question-answer pair such as (Q:"The Earth revolving around the sun can cause ---", A:"constellation to appear in one place in spring and another in fall") is now rephrased to  $(\mathbf{Q}^{*} = "$  "The Earth revolving around the sun can cause what to appear in one place in spring and another in fall?"),  $A^* =$  "constellation"). Specifically, we combine the original question (Q) and original correct choice (A) to form a long statement and rephrase it to be a new question  $(\mathbf{Q}^{*})$  querying a single concept  $(\mathrm{A}^{*})$  in the original answer, where we use the least frequent concept as the target. This question-rephrasing largely improves the number of answerable questions, particularly for the OBQA dataset. All are English data.
A.2 Crowd-sourcing More Answers
Note that there can be multiple correct answers to an open-ended question in OpenCSR while the original datasets only provide a single answer. Thus, we use Amazon Mechanical Turk $^7$  (AMT) to collect more answers for the test questions to have a more precise OpenCSR evaluation.
Figure 6: Distribution of # answers of test questions.
We design a three-stage annotation protocol as follows:
- S1) Multiple-Choice Sanity Check. We provide a question and 4 choices where only one choice is correct and the other 3 are randomly sampled. Only the workers who passed this task, their following annotations will be considered. This is mainly designed for avoiding noise from random workers.
- S2) Selection from Candidates. To improve the efficiency of annotation, we take the union of top 20 predictions from BM25, DPR, DrKIT, and DrFact and randomly shuffle the order of these concepts (most of them are about  $60\sim 70$  candidates). workers can simply input the ids of the concepts that they think are good answers to the question (i.e., a list of integers separated by comma). There are three different workers for each question and we take the candidates which are selected by at least two workers. Note that we also put the correct answer we already have in the candidates and use them as another sanity check to filter out noisy workers.
- S3) Web-based Answer Collection. We generate an URL link to Google Search of the input question to help workers to use the Web for associating more correct answers to the question (the input here is a string for a list of concepts separated by comma). We also provide our concept vocabulary as a web-page so one can quickly check if a concept is valid.
After careful post-processing and multiple rounds of re-assignment, we have in total 15k answers for 2k questions, and the distribution of number of answers are in Figure 6 and Table 1.
<sup>7</sup>https://www.mturk.com/
B Details of Implementation and Our Experiments
B.1 DrFact Implementation
We present some concrete design choices within our DrFact implementation which are abstractly illustrated in the main content of the paper.
(1) Pre-training Dense Fact Index  $D$ . As we mentioned in Sec. 4, we follow the steps of Karpukhin et al. (2020) to pre-train a bi-encoder question answering model on top of BERT (Devlin et al., 2019). To create negative examples, we use the BM25 results which do not contain any answer concept. We use BERT-base (uncased_L-12_H-768_A-12) in our implementation and thus  $d = 768$  in our experiments.
(2) Sparse Fact-to-Fact Index  $S$ . We use a set of rules to decide if we can create a link  $f_{i} \to f_{j}$  (i.e.,  $S_{ij} = 1$ ) as follows:
-  $i \neq j$ . We do not allow self-link here but use self-following as we described in Sec. 4.
-  $|I| > = 1$  where  $I$  is the set of concepts that are mentioned in both  $f_{i}$  and  $f_{j}$ . Note that we remove the most frequent 100 concepts (e.g., human) from  $I$ .
-  $|I| < |f_i|$ . We do not create links when all concepts in  $f_i$  are mentioned in  $f_j$ , which are usually redundant.
-  $|f_{j}| - |I| > = 2$ . We create links only when there are more than two unseen concepts in  $f_{j}$  which are not in  $f_{i}$ , such that the fact-to-fact links create effective reasoning chains.
We also limit that a fact can be followed by at most  $1k$  different facts. Additionally, we append the links from our distant supervision of justifications as well if they were filtered out before.
(3) Hop-wise Question Encoding  $\mathbf{q}_{\mathrm{t}}$ . We encode the question  $q$  with BERT-base and then use its [CLS] token vector as the dense representation for  $\mathbf{q}$ . For each hop, we append a hop-specific layer to model how the question context changes over the reasoning process —  $\mathbf{q}_{\mathrm{t}} = \mathrm{MLP}_{\theta_t}(\mathbf{q})$ .
(4) Fact Translating Function  $g$ . The translating function accepts both the vector representation of previous-hop facts  $\mathbf{F}_{\mathbf{t} - 1}$  and the hop-wise question vector  $\mathbf{q_t}$  and uses an MLP to map the concatenation of them to a vector used for a MIPS query:  $\mathbf{h}_{\mathbf{t} - 1} = \mathrm{MLP}_{\theta_g}([\mathbf{F}_{\mathbf{t} - 1};\mathbf{q_t}])$ . Thus,  $\mathbf{h}_{\mathbf{t} - 1}$  has the same dimension as a fact vector in  $U$ .
(5) Hop-wise Answer Weights  $\alpha_{t}$ . We use the shared query vector to learn how to aggregate pre
Figure 7: The curve of  $\operatorname{Rec}@\mathbf{K}$  in overall data.
dictions at different hops. For a  $T$ -hop DrFact model, we learn to transform the  $\mathbf{q}$  to a  $T$ -dim vector where  $\alpha_{t}$  is the  $t$ -th component.
B.2 Hyper-parameters and Training Details
We now present the details and final hyperparameters that we used in our experiments. For all methods, we tune their hyper-parameters on the validation set and then use the same configurations to train them with the combination of the training and validation sets for the same steps.
BM25. We use the off-the-shelf implementation by elasticsearch $^{8}$ , which are open-source and unsupervised. For the run-time analysis, we use Intel(R) Xeon(R) CPU @ 2.00GHz and the localhost webserver for data transfer.
DPR. We use the source code released by the original authors. The creation of negative contexts are the same when we pre-train our dense fact index  $D$ , which are sampled from BM25 results.
DrKIT. We use the official source code<sup>10</sup> for our experiments. We did minimal modifications on their code for adapt DrKIT towards building dense index of mentions for the OpenCSR corpus and datasets. For fair comparisons between DPR, DrKIT and DrFact, we all use BERT-base as question and mention/fact encoder. We use 200 as the dimension of mention embeddings and  $T = 2$  as the maximum hops. We found that using  $T = 3$  will cause too much memory usage (due to denser entity-to-mention matrix) and also result
$^{8}$ https://github.com/elastic/  
elasticsearch  
 $^{9}$ https://github.com/facebookresearch/  
DPR  
 $^{10}$ https://github.com/google-research/  
language/tree/master/language/labs/drkit
in a very slow training speed. Non-default hyperparameters are: train_batch_size=8 due to the limit of our GPU memory, entity_score_threshold=5e-3 (out of {5e-2, 5e-3, 5e-4, 1e-4}) to filter numerous long-tail intermediate concepts for speeding up training and inference.
DrFact. Similar to DrKIT, we also implement DrFact in TensorFlow for its efficient implementation of tf.RaggedTensor which are essential for us to compute over large sparse tensors. We record the default hyper-parameters in our submitted code. We use a single V100 GPU (16GB) for training with batch size of 24 (using 15GB memory) and learning rate as 3e-5, selected from {1e-5, 2e-5, 3e-5, 4e-5, 5e-5}. The entity_score_threshold=1e-4, and fact_score_threshold=1e-5, which are all selected from {1e-3, 1e-4, 1e-5} based on the dev set.
Model Parameters. DPR, DrKIT and DrFact are all based on the BERT-base, which are 110 million parameters (after pre-training index). DrKIT and DrFact additionally have several MLP layers on top of '[CLS]' token vectors, which are all less than 1 million parameters. The MCQA-reranker model is based on BERT-Large, and thus has 345 million parameters.
C Discussion on Other Related Work
Other Open-Domain QA models. Recent open-domain QA models such as REALM (Guu et al., 2020), Path-Retriever (Asai et al., 2020), ORQA (Lee et al., 2019), and RAG (Lewis et al., 2020b), mainly focus on QA over the full Wikipedia corpus like DrKIT (Dhingra et al., 2020) does. Some of them explicitly use the links between pages to form reasoning chain, while a few them rely on expensive QA-oriented pretraining. Moreover, as DPR (Karpukhin et al., 2020) already shows better performance (see their Table 4) than most prior works with a simpler method, we thus use DPR as the major baseline for evaluation in this work.