SlowGuess/NAACL_2021_OCR huggingface dataset:
global constraints (see Saha et al. (2020)). For each predicted proof, the predicted nodes and edge probabilities from MULTIPROVER, we obtain the corresponding predicted edges using Eqn. 1.
We experiment on synthetic, hand-authored zero-shot, and human paraphrased datasets, following Clark et al. (2020); Saha et al. (2020).
Datasets: The five synthetic datasets DU0-DU5 consist of 100k questions with their own train, validation and test splits (70/10/20) and reasoning depths up to  $D = 0,1,2,3,5$ . Each example in these datasets is annotated with all possible proofs. The second dataset is a Birds-Electricity dataset, consisting of 5k hand-authored samples aimed at evaluating the zero-shot performance of the models. Unlike the previous datasets, all examples in this dataset have a unique gold proof. Third, ParaRules is a human-paraphrased dataset, consisting of 40k examples with all possible proofs, where the facts and rules are paraphrased by crowd-workers. Further details of the datasets and model's hyperparameters can be found in the appendix.
Evaluation Metrics: Following PROVER, QA evaluation is done through accuracy. For proofs, we compute the following metrics: (1) Node Precision, Recall, F1 (2) Edge Precision, Recall, F1, (3) Proof Precision, Recall, F1, and (4) Full Accuracy (FA). For each sample, given a set of gold proofs and predicted proofs, node precision is computed as the fraction of predicted proofs where the predicted node set matches exactly with a gold proof's node set. Similarly, node recall for each sample is computed as the fraction of gold proofs where the corresponding node sets match exactly. The overall node precision, recall and F1 are the respective sample-wise precision, recall and F1 scores averaged over all the samples. Edge metrics are computed similarly but with respect to the edges only and the proof metrics consider both nodes and edges in conjunction. Our final metric, full accuracy evaluates a sample as a whole and is given by the fraction of samples where the answer and all corresponding proofs are exactly correct.
5.1 Comparison of PROVER variants with MULTIPROVER
In Table 1, we compare ML-MULTIPROVER and IT-MULTIPROVER with five variants of PROVER
Node	Edge	Proof	FA
QA	P	R	F1	P	R	F1	P	R	F1
PROVER (Saha et al., 2020)	99.3	89.2	84.9	86.0	87.5	84.2	85.3	87.1	84.0	84.7	81.2
PROVER-all	99.3	87.9	83.8	84.9	87.1	83.6	84.6	85.9	82.8	83.7	80.3
PROVER-top-p	99.3	34.4	88.4	48.4	33.8	87.4	47.7	33.3	86.7	47.2	00.0
PROVER-top-p-classifier	99.3	85.7	84.4	83.8	84.8	84.1	83.5	83.9	83.4	82.6	77.3
PROVER-top-p-threshold	99.3	84.4	88.0	85.0	83.6	87.1	84.4	83.0	86.5	83.8	77.2
ML-MULTIPROVER	99.5	89.4	89.2	89.0	87.7	87.8	87.4	87.2	87.3	87.0	83.8
IT-MULTIPROVER	99.5	90.6	90.2	90.0	89.6	89.4	89.2	89.1	89.0	88.7	85.5
Table 1: Comparison of our MULTIPROVER models with PROVER variations on DU5 test set. Iterative-MULTIPROVER's improvement in Full Accuracy over Multilabel-MULTIPROVER is statistically significant with  $p < 0.001$ .
- (1) PROVER, as introduced in Saha et al. (2020), trained with one proof per example and also generates a single proof, (2) PROVER-all, trained with all proofs as separate examples and generates a single proof per example, (3) PROVER-top- $p$ , an extension of PROVER-all, generating top- $p$  proofs for all examples, (4) PROVER-top- $p$ -classifier, an improvement over the vanilla top- $p$  model, where we first predict the number of proofs by training a RoBERTa classifier with concatenated question and context and then generate those many top proof graphs, and (5) PROVER-top- $p$ -threshold, another improved model over vanilla top- $p$ , where we use the optimization score from Equation 3 to predict the number of proofs to generate, i.e., we stop generating proofs when the score difference between two consecutive proofs exceeds a certain threshold (tuned on the validation set). All models are trained on the DU5 train set and tested on the corresponding test set. Based on Figure 2 which shows that  $98\%$  of the dataset contains samples with  $\leq 3$  proofs, we set max-proofs,  $p = 3$ .  $87\%$  of the examples in the dataset have a single gold proof, thereby making PROVER a strong baseline.
We observe that PROVER-all has a slightly lower proof F1 than PROVER, because the model likely gets confused with multiple possible proofs for the same context and question. PROVER-top- $p$ 's huge drop in precision is unsurprising because the subsequent non-empty proofs are always incorrect, causing full accuracy to drop to  $0\%$ . When we perform careful inference over PROVER either by predicting the number of proofs or by thresholding and do not generate a fixed  $p$  number of proofs for all examples, we observe a boost in precision over the vanilla top- $p$  model, with very little drop in recall. However, PROVER continues to be a stronger baseline than all the top- $p$  variants because of a lot of single-proof examples in the dataset.
Both MULTIPROVER models improve significantly on the state-of-the-art proof F1, while retaining a near perfect QA accuracy. IT-MULTIPROVER is a significantly stronger model because of its explicit conditioning mechanism and obtains up to a statistically significant $^2$  ( $p < 0.001$ )  $4\%$  improvement on proof F1 and full accuracy. While our model is expected to improve the proof recall compared to PROVER and PROVER-all because of the generation of multiple proofs, the improvement in precision is particularly important as it shows that the subsequently generated proofs by IT-MULTIPROVER are mostly correct. Similarly, its improvement in proof recall compared to PROVERTOP- $p$  also shows the strength of the model considering that PROVER-top- $p$  generates the maximum number of proofs for every sample. Overall, IT-MULTIPROVER outperforms all other models in all metrics. In summary, careful inference strategies over a single-proof generation model like PROVER are largely ineffective for generating multiple proofs and an effective proof-set generation model needs to exploit and learn the inter-proof correlations during the training phase itself. Our experiments on the ParaRules dataset demonstrate similar findings, details of which and the effect of varying  $p$  for MULTIPROVER is in the appendix.
Iterative-MULTIPROVER performs equally well on the subset of questions where the context has negations, achieving a high proof F1 of 90.8. As part of error analysis, we find that  $58\%$  of Iterative-MULTIPROVER's wrongly predicted proofs have more nodes and edges than those in the gold proof, suggesting that our model tends to overestimate the essential rules and facts and their inter-connections. In the following subsections, we analyze MULTIPROVER's generalization capabilities in three dif
2We use bootstrap test (Efron and Tibshirani, 1994) for calculating the statistical significance score.
QA	Node	Edge	Proof	FA
P	R	F1	P	R	F1	P	R	F1
PROVER (Saha et al., 2020)	86.5	81.3	81.3	81.3	81.4	81.4	81.4	80.7	80.7	80.7	80.7
PROVER-all	85.9	80.9	80.9	80.9	80.4	80.4	80.4	80.2	80.2	80.2	80.0
ML-MULTIPROVER	85.1	79.2	79.9	79.4	79.4	79.9	79.5	78.7	79.1	78.8	78.1
IT-MULTIPROVER	86.3	82.7	83.3	82.9	82.4	83.0	82.6	82.2	82.7	82.3	81.8
Table 2: Comparison of all models on the zero-shot Birds-Electricity dataset containing one gold proof per sample. Iterative-MULTIPROVER's improvement in Full Accuracy over PROVER is statistically significant with  $p < 0.001$ .
d	MP	Proof F1	Full Acc
PR	ML	IT	PR	ML	IT
0	7.2	93.8	97.8	98.2	92.6	96.7	97.0
1	10.3	88.0	92.8	93.5	85.7	91.0	91.7
2	15.7	80.8	86.1	87.1	76.5	81.8	83.7
3	17.7	78.0	80.7	83.0	72.2	75.9	78.0
4	19.9	71.1	72.3	77.2	65.9	66.4	70.1
5	23.1	67.7	64.9	70.6	61.0	58.7	63.7
ferent contexts - zero-shot settings, higher depth questions and training with less training data.
5.2 Generalization to Zero-Shot Dataset with Single Gold Proofs
The Birds-Electricity test-only dataset evaluates the zero-shot performance. It contains examples with single gold proofs; hence, if a multiple-proof generation model like MULTIPROVER transfers well to it, this indicates strong generalization capabilities because along with generating correct proofs, it also needs to infer the correct number of proofs. With that motivation, in Table 2, we compare PROVER and PROVER-all, both trained on DU5 to generate a single proof, with our MULTIPROVER models, also trained on DU5 and find that IT-MULTIPROVER obtains state-of-the-art result on all proof-related metrics, while retaining the QA performance. Note that IT-MULTIPROVER has two important design choices which explains its good performance on out-of-domain transfer – (1) it trains on all proofs jointly, (2) explicit proof conditioning. Both of these, when combined, enable it to learn the correlations between the proofs to identify the degree of relevance of facts and rules, ranging from essential to sometimes useful to irrelevant, for a given question. Thus, on out-of-domain test data, it assigns soft prior relevance scores to the context which helps it to better learn the significantly smaller space of correct proofs and be more accurate even for a single-proof dataset.
Table 3: Comparison of PROVER-all and MULTIPROVER models on the subset of samples in DU5 test set requiring d depth of reasoning.
Count	QA	Proof F1	Full Acc
ML	IT	ML	IT	ML	IT
10k	87.2	86.1	41.5	41.4	39.0	39.5
30k	97.7	98.2	74.3	74.9	71.2	72.0
50k	99.4	99.4	83.7	84.5	80.0	81.0
70k (All)	99.5	99.5	87.0	88.7	83.8	85.5
Table 4: Comparative study between the two MULTIPROVER models with varying amount of training data on DU5. Count = number of training examples.
5.3 Generalization to Higher Depths
The DU5 dataset consists of questions requiring reasoning up to a maximum depth of 5. Thus, we test the generalization capabilities of the MULTIPROVER models on higher depth questions. Specifically, in Table 3, we compare the DU5-trained models of PROVER-all, ML-MULTIPROVER and IT-MULTIPROVER on the subset of DU5 test examples with varying depths of reasoning  $(d)$ . Each row also shows the percentage of examples with multiple gold proofs (MP) which, unsurprisingly, increases as the depth increases. We observe that much of IT-MULTIPROVER's improvement compared to ML-MULTIPROVER comes at higher depths where the presence of multiple proofs is a more frequent phenomenon. At depth-5, where  $23\%$  of the examples have  $>1$  correct proof, IT-MULTIPROVER obtains a  $6\%$  improvement over ML-MULTIPROVER. This shows that joint training with all proofs and explicit conditioning between them leads to better generalization at higher depths.
5.4 Generalization with Less Training Data
Collecting proofs for supervised training is expensive in most real-world scenarios. Hence, on top of the zero-shot and depth generalization results presented so far, we ask if our MULTIPROVER models can learn from less training data. Table 4 shows that these models obtain near perfect QA accuracy with only  $40\%$  of the training data (30k examples). However, proof generation proves to be challenging and only improves with sufficient training
Rules:
R1: All green, white people are round.
R2: Quiet people are white.
R3: All green, young people are nice.
R4: If someone is quiet and green then they are kind.
R5: White people are nice.
R6: Quiet people are
R7: All green, white people are nice.
R8: If someone is kind and white then they are green.
R9: All nice, quiet people are kind.
Facts:
F1: Bob is quiet. F2: Bob is young.  
F3: Charlie is quiet. F4: Charlie is young
F5:Fiona is nice.F6:Fiona is quiet.
F7:Fiona is round.F8:Fiona is white.
F9: Gary is green. F10: Gary is nice.
F11: Gary is quiet. F12: Gary is young.
Q1: Fiona is not kind. [Answer : F ]
F1: Anne is cold. F2: Bob is cold.  
F3: Bob is young. F4: Fiona is big.
F5:Fiona is young.F6:Harry is big.
E7: Harry is blue. E8: Harry is cold.
F9: Harry is blue. F10: Harry is cold.
F11: Harry is red. F12: Harry is young.
Q2: Bob is not cold. [Answer: F]
Rules:
Facts:
data. Another interesting observation is that while both MULTIPROVER models perform comparably with less training data, IT-MULTIPROVER starts to outperform ML-MULTIPROVER upon training with more examples. IT-MULTIPROVER consists of more trainable parameters because of its multiple node and edge encoders, which get learned better with more data. See appendix for runtime and parameter space of these models.
5.5 Comparison of MULTIPROVER with the Skyline Single-Proof Generation Model
We find that an ideal (skyline) single-proof generation model's proof recall for the DU5 dataset is upper-bounded by  $92\%$  as it contains about  $87\%$  of single-proof examples. This is computed by considering exactly 1 correct proof per question. Hence, we ask how well our MULTIPROVER models compare with this ideal performance (Figure 7). Our results are encouraging, not only because IT-MULTIPROVER generates more correct proofs than all other models but also because it almost matches the performance of the skyline single-proof generation model. The PROVER model is  $9.2\%$  worse as compared to the skyline single-proof generation model while IT-MULTIPROVER reduces this gap to  $3\%$ . Given the dataset mostly contains single-proof examples, the skyline is a strong upper-bound on proof generation performance and IT-MULTIPROVER significantly reduces the gap. See appendix for ablations of IT-MULTIPROVER, including the effect of Hungarian Loss.
6 Qualitative Analysis of MULTIPROVER
Fig. 6 shows the sets of proofs correctly generated by Iterative-MULTIPROVER for two randomly chosen questions. For  $Q_{1}$ , it generates all the possible proofs by identifying the common subgraph
Figure 6: Figure showing all the proofs correctly generated by our Iterative-MULTIPROVER model for two randomly chosen questions corresponding to two different rule-bases.
Figure 7: Comparison of proof recall for all models with that of the skyline single-proof generation model.
$F_{6} \rightarrow R_{9}$ .  $Q_{2}$  is interesting, because (i) the single-node proof  $F_{2}$  is significantly different from the other proofs in both structure and size, and (ii) the two larger proofs have two distinct common subgraphs. Here, PROVER performs simple lookup in the rule-base to generate the proof  $F_{2}$ , thereby limiting our understanding of its reasoning capabilities. However, MULTIPROVER, through its ability to also generate the larger and more complex proofs enhances the transparency and verification of its reasoning abilities, and hence is a crucial step towards bridging the gap between neural and symbolic approaches.
We proposed Multilabel-MULTIPROVER and Iterative-MULTIPROVER, two variants of a proof-set generation model where the former performs implicit conditioning between the proofs to generate them in parallel while the latter generates a proof-set through explicit conditioning on the previously generated proofs. Both models obtain strong proof F1 improvements on synthetic and human-paraphrased datasets and Iterative-MULTIPROVER also obtains state-of-the-art proof F1 on a zero-shot dataset with single proofs. MULTIPROVER's modeling is fairly generic and similar methods can be used in generating a set of structured explanations for other NLP tasks like multi-hop QA.
Despite the overwhelming success of pre-trained language models for various NLP tasks, a common criticism is their lack of interpretability. Generating structured proofs from such models allows us to explain their reasoning capabilities and also bridges the gap between neural and symbolic systems. In this work, we take a step closer towards improving the interpretability of rule-based reasoning by generating a set of multiple proofs, each providing a diverse rationale for the reasoning process. We experiment with a wide variety of rule-bases ranging from synthetic to hand-authored to human-paraphrased rule-bases. Our results show good generalization performance of our models across three different aspects – (1) zero-shot settings, (2) questions requiring higher depths of reasoning, and (3) availability of less training data. We hope our models and findings will inspire future work on generating multiple structured explanations for different compositional reasoning tasks in NLP.
We thank the reviewers and Peter Hase for their helpful feedback. This work was supported by DARPA MCS Grant N66001-19-2-4031, NSF-CAREER Award 1846185, DARPA YFA17-D17AP00022, ONR Grant N00014-18-1-2871, Microsoft Investigator Fellowship, and Munroe & Rebecca Cobey Fellowship. The views in this article are those of the authors and not the funding agency.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533-1544.
Jonathan Berant and Percy Liang. 2014. Semantic parsing via paraphrasing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1415-1425.
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-SNLI: Natural language inference with natural language explanations. In Advances in Neural Information Processing Systems, pages 9539-9549.
Kyunghyun Cho, Bart van Merrienboer, Caglar Güçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP.
Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2020. Transformers as soft reasoners over language. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 3882-3890. International Joint Conferences on Artificial Intelligence Organization. Main track.
Bo Dai, Sanja Fidler, Raquel Urtasun, and Dahua Lin. 2017. Towards diverse and natural image descriptions via a conditional gan. In Proceedings of the IEEE International Conference on Computer Vision, pages 2970-2979.
Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C Wallace. 2020. Eraser: A benchmark to evaluate rationalized nlp models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4443-4458.
Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608.
Bradley Efron and Robert J Tibshirani. 1994. An introduction to the bootstrap. CRC press.
Tian Gao, Jie Chen, Vijil Chenthamarakshan, and Michael Witbrock. 2019. A sequential set generation method for predicting set-valued outputs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 2835-2842.
Kevin Gimpel, Dhruv Batra, Chris Dyer, and Gregory Shakhnarovich. 2013. A systematic exploration of diversity in machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1100-1111, Seattle, Washington, USA. Association for Computational Linguistics.
Nicolas Gontier, Koustuv Sinha, Siva Reddy, and Chris Pal. 2020. Measuring systematic generalization in neural proof generation with transformers. Advances in Neural Information Processing Systems, 33.
Peter Hase and Mohit Bansal. 2020. Evaluating explainable AI: Which algorithmic explanations help users predict model behavior? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Alon Jacovi and Yoav Goldberg. 2020. Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness? In ACL.
Unnat Jain, Ziyu Zhang, and Alexander G Schwing. 2017. Creativity: Generating diverse questions using variational autoencoders. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6485-6494.
Peter Jansen and Dmitry Ustalov. 2019. Textgraphs 2019 shared task on multi-hop inference for explanation regeneration. In Proceedings of the Thirteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-13), pages 63-77.
Peter A Jansen, Elizabeth Wainwright, Steven Marmorstein, and Clayton T Morrison. 2018. Worldtree: A corpus of explanation graphs for elementary science questions supporting multi-hop inference. arXiv preprint arXiv:1802.03052.
Harsh Jhamtani and Peter Clark. 2020. Learning to explain: Datasets and models for identifying valid reasoning chains in multihop question-answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 137-150.
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. Qasc: A dataset for question answering via sentence composition. In Proceedings of the AAAI Conference on Artificial Intelligence, 05, pages 8082-8090.
Adam R Kosiorek, Hyunjik Kim, and Danilo J Rezende. 2020. Conditional set generation with transformers. arXiv preprint arXiv:2006.16841.
H. W. Kuhn and Bryn Yaw. 1955. The hungarian method for the assignment problem. Naval Res. Logist. Quart, pages 83-97.
Ilya Kulikov, Alexander H Miller, Kyunghyun Cho, and Jason Weston. 2018. Importance of a search strategy in neural dialogue modelling. In iNLG.
Sawan Kumar and Partha Talukdar. 2020. NILE: Natural language inference with faithful natural language explanations. In ACL.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam R. Kosiorek, Seungjin Choi, and Yee Whye Teh. 2018. Set transformer: A framework for attention-based permutation-invariant neural networks. In ICML.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 107-117.
Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. A simple, fast diverse decoding algorithm for neural generation. arXiv preprint arXiv:1611.08562.
Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. 2019. Reasoning over paragraph effects in situations. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 58-62, Hong Kong, China. Association for Computational Linguistics.