SlowGuess/NAACL_2021_OCR huggingface dataset:
Then, to get a 2-hop evidence chain, we first check if a question-only fact can be linked to an answer-only fact through the sparse fact-to-fact matrix  $S$ . Similarly, we can also get 3-hop distant evidence. In this manner, we can collect the set of supporting facts at each hop position, denoted as  $\{F_1^*, F_2^*, \ldots, F_T^*\}$ .
The final learning objective is thus to optimize the sum of the cross-entropy loss  $l$  between the final weighed set of concepts  $A$  and the answer set  $A^{*}$ , as well as the auxiliary loss from distant evidence — i.e., the mean of the hop-wise loss between the predicted facts  $F_{t}$  and the distant supporting facts at that hop  $F_{t}^{*}$ , defined as follows:
$$
\mathcal {L} = l (A, A ^ {*}) + \frac {1}{T} \sum_ {t = 1} ^ {T} l \left(F _ {t}, F _ {t} ^ {*}\right)
$$
Fact corpus and concept vocabulary
We use the GenericsKB-Best corpus as the main knowledge source<sup>2</sup>. In total, we have 1,025,413 unique facts as our  $\mathcal{F}$ . We use the spaCy toolkit to prepossess all sentences in the corpus and then extract frequent noun chunks within them as our concepts. The vocabulary  $\nu$  has 80,524 concepts, and every concept is mentioned at least 3 times.
Datasets for OpenCSR
To facilitate the research on open-ended commonsense reasoning (OpenCSR), we reformatted three existing multi-choice question answering datasets to allow evaluating OpenCSR methods. We choose three datasets: QASC, OBQA, and ARC, as their questions require commonsense knowledge about science and everyday objects and are presented in natural language. By applying a set of filters and rephrasing rules, we selected those open-ended commonsense questions that query concepts in our vocabulary  $\mathcal{V}$ .
It was constructed from multiple commonsense knowledge corpora and only kept naturally occurring generic statements, which makes it a perfect fit for OpenCSR.
Stat. \ Data	ARC	QASC	OBQA	Overall
# All Examples	6,600	8,443	5,288	20,331
# Training Set	5,355	6,883	4,199	16,437
# Validation Set	562	731	463	1,756
# Test Set	683	829	626	2,138
Avg.#Answers	6.8	7.6	7.7	7.5
Single-hop %	66.91%	59.35%	50.80%	59.02%
As we know that there can be multiple correct answers for a question in OpenCSR, we employed crowd-workers to collect more answers for each test question based on a carefully designed annotation protocol. In total, we collect 15,691 answers for 2,138 rephrased questions for evaluation, which results in 7.5 answers per question on average. Please find more details about crowdsourcing and analysis in Appendix A.
We show some statistics of the OpenCSR datasets and our new annotations in Table 1. To understand the multi-hop nature and the difficulty of each dataset, we use a heuristic to estimate the percentage of "single-hop questions", for which we can find a fact (from top-1k facts retrieved by BM25) containing both a question concept and an answer concept. The ARC dataset has about  $67\%$  one-hop questions and thus is the easiest, while OBQA has only  $50\%$ .
Evaluation metrics.
Recall that, given a question  $q$ , the final output of every method is a weighted set of concepts  $A = \{(a_{1},w_{1}),\ldots \}$ . We denote the set of true answer concepts, as defined above, as  $A^{*} = \{a_{1}^{*},a_{2}^{*},\dots \}$ . We define Hit@K accuracy to be the fraction of questions for which we can find at least one correct answer concept  $a_{i}^{*}\in A^{*}$  in the top-  $K$  concepts of  $A$  (sorted in descending order of weight). As questions have multiple correct answers, recall is also an important aspect for evaluating OpenCSR, so we also use Rec@K to evaluate the average recall of the top-K proposed answers.
We present baseline methods and an optional re-ranker component for boosting the performance on OpenCSR. Table 3 shows a summary of the comparisons of the three methods and our DrFact.
Direct Retrieval Methods. The most straightforward approach to the OpenCSR task is to directly
Table 1: Statistics of datasets for OpenCSR (v1.0).
Methods	BM25	DPR	DrKIT	DrFact (ours)
Knowledge Corpus Structure	A set of docs	A set of docs	Mention-Entity Bipartite Graph	Concept-Fact Hypergraph
Multi-hop Formulation	N/A	N/A	Entity-Following	Fact-Following
Index for Dense Retrieval	N/A	Dense Fact Embeddings	Dense Mention Embeddings	Dense Fact Embeddings
Sparse Retrieval Method	BM25	N/A	Entity-Entity/Mention Co-occurrence	Fact-to-Fact, Concept-to-Fact Matrix
# models for Multi-Hop	N/A	N/A	Multiple Models	A single model (self-following)
Intermediate Supervision	N/A	N/A	N/A	Auxiliary Learning
Table 3: Comparisons of the four retrieval methods.
retrieve relevant facts, and then use the concepts mentioned in the top-ranked facts as answer predictions. BM25 is one of the most popular unsupervised method for retrieval, while the Dense Passage Retrieval (DPR) model is a state-of-the-art trainable, neural retriever (Karpukhin et al., 2020). Following prior work with DPR, we used BM25-retrieved facts to create positive and (hard-)negative examples as supervision. For both methods, we score a concept by the  $\max^3$  of the relevance scores of retrieved facts that mention it.
DrKIT. Following Dhingra et al. (2020), we use DrKIT for OpenCSR, treating concepts as entities. DrKIT is also an efficient multi-hop reasoning model that reasons over a pre-computed indexed corpus, which, as noted above (Sec. 2), differs from our work in that DrKIT traverses a graph of entities and entity mentions, while DRFACT traverses a hypergraph of facts.
Multiple-choice style re-ranking (MCQA). A conventional approach to multiple-choice QA (MCQA) is to fine-tune a pre-trained language model such as BERT, by combining a question and a particular concept as a single input sequence in the form of “[CLS] question [SEP] choice” and using [CLS] vectors for learning to score choices. We follow this schema and train<sup>4</sup> such a multiple-choice QA model on top of BERT-Large, and use this to re-rank the top- $K$  concept predictions.
Main results. For a comprehensive understanding, we report the Hit@K and Rec@K of all methods, at K=50 and K=100, in Table 2. The overall results are the average over the three datasets.
<sup>3</sup>We also tried mean and sum, but max performs the best.  
<sup>4</sup>Specifically, we fine-tune BERT-Large to score truth answers over 9 sampled distractors, and use it to rank the top-500 concepts produced by each above retrieval method.
ARC	QASC	OBQA	Overall
Metric = Hit@K (%)	H@50	H@100	H@50	H@100	H@50	H@100	H@50	H@100
BM25 (off-the-shelf)	56.95	67.35	58.50	66.71	53.99	66.29	56.48	66.78
DPR (Karpukhin et al., 2020)	68.67	78.62	69.36	78.89	62.30	73.80	66.78	77.10
DrKIT (Dhingra et al., 2020)	67.63	77.89	67.49	81.63	61.74	75.92	65.62	78.48
DRFACT (Ours)	71.60	80.38	72.01	84.56	69.01	80.03	70.87	81.66
BM25 + MCQA Reranker	76.87	80.38	75.75	80.22	79.23	84.03	77.28	81.54
DPR + MCQA Reranker	76.72	83.16	81.66	87.45	77.16	83.39	78.51	84.67
DrKIT + MCQA Reranker	78.44	83.37	84.00	86.83	79.25	84.03	80.56	84.74
DRFACT + MCQA Reranker	84.19	89.90	89.87	93.00	85.78	90.10	86.61	91.00
Metric = Rec@K (%)	R@50	R@100	R@50	R@100	R@50	R@100	R@50	R@100
BM25 (off-the-shelf)	21.12	28.08	16.33	20.13	14.27	20.21	17.24	22.81
DPR (Karpukhin et al., 2020)	28.93	38.63	23.19	32.12	18.11	26.83	23.41	32.53
DrKIT (Dhingra et al., 2020)	27.57	37.29	21.25	30.93	18.18	27.10	22.33	31.77
DRFACT (Ours)	31.48	40.93	23.29	33.60	21.27	30.32	25.35	34.95
BM25 + MCQA Reranker	39.11	42.96	29.03	32.11	36.38	39.46	34.84	38.18
DPR + MCQA Reranker	43.78	51.56	40.72	48.25	36.18	43.61	40.23	47.81
DrKIT + MCQA Reranker	43.14	49.17	39.20	44.37	35.12	39.85	39.15	44.46
DRFACT + MCQA Reranker	47.73	55.20	44.30	50.30	39.60	45.24	43.88	50.25
Table 2: Results of the Hit@K and Rec@K ( $K = 50/100$ ) on OpenCSR (v1.0). We present two groups of methods with different inference speed levels. The upper group is retrieval-only methods that are efficient ( $< 0.5\mathrm{sec/q}$ ), while the bottom group are augmented with a computationally expensive answer reranker ( $\geq 14\mathrm{sec/q}$ ).
We can see that DRFACT outperforms all baseline methods for all datasets and metrics. Comparing with the state-of-the-art text retriever DPR, DRFACT improves by about  $4.1\%$  absolute points in Hit@50 accuracy overall. With the expensive yet powerful MCQA reranker module DRFACT gives an even large gap ( $\sim 8\%$  gain in H@50 acc).
The performance gains on the QASC and OBQA datasets are larger than the one on ARC. This observation correlates the statistics that the former two have more multi-hop questions and thus DRFACT has more advantages. As shown in Figure 4, we can see that DRFACT consistently outperforms other retrieval methods at different  $K$  by a considerable margin.
Interestingly, we find that with the MCQA reranker, DrKIT does not yield a large improvement over DPR, and it usually has a lower than other methods. We conjecture this is because that entity-centric reasoning schema produces too many possible concepts and thus is more likely to take more irrelevant concepts at the top positions.
The results on  $\mathbf{Rec}@\mathbf{K}$  in bottom section of Table 2 show that even our DRFACT+MCQA model only recalls about  $50\%$  of the correct answers in top-100 results on average. This suggests that OpenCSR is still a very challenging problem and
Figure 4: The curve of Hit@K accuracy in overall. Please find the curve of Rec@K in Figure 7.
future works should focus on improving the ability of ranking more correct answers higher.
Run-time efficiency analysis. We use Table 4 to summarize the online inference speed of each OpenCSR method. At inference time, DPR will make one call to BERT-base for encoding a question and do one MIPS search. Similarly, DrKIT and DRFACT with  $T$  hops will make one call to BERT-base for query encoding and do  $T$  MIPS searches. However, since the entity-to-mention
Methods	Major Computations	Speed (sec/q)
BM25	Sparse Retrieval	0.14
DPR	BERT-base + MIPS	0.08
DrKIT	BERT-base + T*(MIPS+ sp_e2m)	0.47
DRFACT	BERT-base + T*(MIPS+ sp_f2f)	0.23
X+ MCQA	X + K * BERT-Large	+ 14.12
Table 4: The major competitions of each method and their online (batch-size=1) inference speed in sec/q.
ARC	QASC	OBQA	Overall
T=1	69.3%	70.1%	65.0%	68.1%
T=2	71.1%	72.2%	68.3%	70.5%
T=3✓	71.6%	72.0%	69.0%	70.9%
w/o. Self-follow	70.9%	70.4%	68.4%	69.9%
w/o. Aux. loss	70.6%	70.1%	68.0%	69.6%
Table 5: Ablation study of DRFACT (H@50 test acc).
matrix  $(\mathrm{sp}_{e2m})$  of DrKIT is much larger than the fact-to-fact matrix  $(\mathrm{sp}_{f2f})$  of DRFACT, DrKIT is about twice as slow as DRFACT. The MCQA is much more computationally expensive, as it makes  $K$  calls to BERT-Large for each combination of question and choice. Note that in these experiments we use  $T = 2$  for DrKIT,  $T = 3$  for DRFACT and  $K = 500$  for the MCQA re-rankers.
Ablation study. Varying the maximum hops  $(\mathrm{T} = \{1,2,3\})$  — i.e., the number of calls to Fact-Follow — indicates that overall performance is the best when  $T = 3$  as shown in Table 5. The performance with  $T = 2$  drops  $0.7\%$  point on OBQA. We conjecture this is due to nature of the datasets, in particular the percentage of hard questions. We also test the model (with  $T = 3$ ) without the auxiliary learning loss (Sec. 4.4) or the self-following trick. Both are seen to be important to DRFACT. Self-following is especially helpful for QASC and OBQA, where there are more multi-hop questions. It also makes learning and inference more faster than an alternative approach of assembling multiple models with different maximum hops as done in some prior works.
Qualitative analysis. We show a concrete example in Fig. 5 to compare the behaviour of DPR and DRFACT in reasoning. DPR uses purely dense retrieval without any regularization, yielding irrelevant facts. The fact  $f_{2}$  matches the phrase "sepa
Q: "What will separate iron filings from sand?"
f1= angle irons reinforce the thinnest section of the ring.
f2= sieves are used for separating fossils from sand...
f3= stainless steel has a rough surface just after filing." DPR
iron filings show the magnetic fields. (in F0)
DrFact
magnets produce a magnetic field with a north ... (in F1)
magnets attract magnetic metals through magnetism (in F2)
Figure 5: A case study to compare DPR and DRFACT.
rating...from sand," but does not help reason about the question. The  $f_{3}$  shows here for the semantic relatedness of "steel" and "iron" while "filling" here is not related to question concepts. Our DRFACT, however, can faithfully reason about the question via fact-following over the hypergraph, and use neural fact embeddings to cumulatively reason about a concept, e.g., magnet. By backtracking with our hypergraph, we can use retrieved facts as explanations for a particular prediction.
We introduce and study a new task — open-ended commonsense reasoning (OpenCSR) — which is both realistic and challenging. We construct three OpenCSR versions of widely used datasets targeting commonsense reasoning with a novel crowdsourced collection of multiple answers, and evaluate a number of baseline methods for this task. We also present a novel method, DRFACT. DRFACT is a scalable multi-hop reasoning method that traverses a corpus (as a hypergraph) via a differentiable "fact-following" reasoning process, employing both a neural dense index of facts and sparse tensors of symbolic links between facts, using a combination of MIPS and sparse-matrix computation. DRFACT outperforms several strong baseline methods on our data, making a significant step towards adapting commonsense reasoning approaches to more practical applications. Base on the multi-hop reasoning framework of DRFACT, we hope the work can benefit future research on neural-symbolic commonsense reasoning.
Xiang Ren is supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via Contract No. 2019-19051600007, the DARPA MCS program under Contract No. N660011924033 with the United States Office Of
We note the MCQA-eranker could be speed up by scoring more choices in parallel. All run-time tests were performed on NVIDIA V100 (16GB), but MCQA with batch-size of 1 requires only  $\sim 5\mathrm{GB}$ . This suggests more parallel inference on a V100 could obtain  $4.5\mathrm{sec} / \mathrm{q}$  for MCQA.
Naval Research, the Defense Advanced Research Projects Agency with award W911NF-19-20271, and NSF SMA 18-29268. We thank all reviewers for their constructive feedback and comments.
* Ethical Considerations
Crowd-workers. This work presents three datasets for addressing a new problem, open common-sense reasoning. The datasets are all derived from existing multiple-choice CSR datasets, and were produced by filtering questions and using crowd-workers to annotate common-sense questions by suggesting additional answers. Most of the questions are about elementary science and common knowledge about our physical world. None of the questions involve sensitive personal opinions or involve personally identifiable information. We study posted tasks to be completed by crowd-workers instead of crowd-workers themselves, and we do not retrieve any identifiable private information about a human subject.
Data bias. Like most crowdsourced data, and in particular most common-sense data, these crowdsourced answers are inherently subject to bias: for example, a question like "what do people usually do at work" might be answered very differently by people from different backgrounds and cultures. The prior multiple-choice CSR datasets which our datasets are built on are arguably more strongly biased culturally, as they include a single correct answer and a small number of distractor answers, while our new datasets include many answers considered correct by several annotators. However, this potential bias (or reduction in bias) has not been systematically measured in this work.
Sustainability. For most of the experiments, we use the virtual compute engines on Google Cloud Platform, which "is committed to purchasing enough renewable energy to match consumption for all of their operations globally."6 With such virtual machine instances, we are able to use the resources only when we have jobs to run, instead of holding them all the time like using physical machines, thus avoiding unnecessary waste.
Application. The work also evaluates a few proposed baselines for OpenCSR, and introduced a new model which outperforms them. This raises the question of whether harm might arise from applications of OpenCSR—or more generally, since
OpenCSR is intended as a step toward making multiple-choice CSR more applicable, whether harm might arise more generally from CSR methods. Among the risks that need to be considered in any deployment of NLP technology are that responses may be wrong, or biased, in ways that would lead to improperly justified decisions. Although in our view the current technology is still relatively immature, and unlikely to be fielded in applications that would cause harm of this sort, it is desirable that CSR methods provide audit trails, and recourse so that their predictions can be explained to and critiqued by affected parties. Our focus on methods that provide chains of evidence is largely a reflection of this perceived need.
Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. 2020. Learning to retrieve reasoning paths over wikipedia graph for question answering. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Sumithra Bhakthavatsalam, Chloe Anastasiades, and Peter Clark. 2020. Genericskb: A knowledge base of generic statements. arXiv preprint arXiv:2005.00660.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870-1879, Vancouver, Canada. Association for Computational Linguistics.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.
William W. Cohen, Haitian Sun, R. Alex Hofer, and Matthew Siegler. 2020. Scalable neural methods for reasoning with a symbolic knowledge base. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, and Andrew McCallum. 2019. Multi-step retriever-reader interaction for scalable open-domain question answering. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of
$^{6}$ https://cloud.google.com/sustainability
deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Bhuwan Dhingra, Manzil Zaheer, Vidhisha Balachandran, Graham Neubig, Ruslan Salakhutdinov, and William W. Cohen. 2020. Differentiable reasoning over a virtual knowledge base. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Yair Feldman and Ran El-Yaniv. 2019. Multi-hop paragraph retrieval for open-domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2296-2309, Florence, Italy. Association for Computational Linguistics.
Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, and Xiang Ren. 2020. Scalable multi-hop relational reasoning for knowledge-aware question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1295-1309, Online. Association for Computational Linguistics.
Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. 2020. Accelerating large-scale inference with anisotropic vector quantization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3887-3896. PMLR.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Papat, and Ming-Wei Chang. 2020. Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3929-3938. PMLR.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. IEEE Transactions on Big Data.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UNIFIEDQA: Crossing format boundaries with a single QA system. In Find-
ings of the Association for Computational Linguistics: EMNLP 2020, pages 1896-1907, Online. Association for Computational Linguistics.
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. QASC: A dataset for question answering via sentence composition. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8082-8090. AAAI Press.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086–6096, Florence, Italy. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Patrick S. H. Lewis, Ethan Perez, Aleksandra Pik-tus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020b. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019. KagNet: Knowledge-aware graph networks for commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2829-2839, Hong Kong, China. Association for Computational Linguistics.
Shangwen Lv, Daya Guo, Jingjing Xu, Duyu Tang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang,