SlowGuess/NAACL_2022_OCR huggingface dataset:
Baselines. We consider the following baselines<sup>4</sup>.
- RuleTaker (RT) (Clark et al., 2020): a RoBERTa based model that only predicts answers.
PROVER (PV) (Saha et al., 2020): a method that treats the proof as a graph and predicts all its nodes and edges at once, also using RoBERTa model as the backbone, same as IBR.
PROBR (PB) (Sun et al., 2021): it improves PROVER by introducing the probabilistic graph that jointly considers the answer, nodes and edges.
- EVR (Liang et al., 2021): an iterative model that predicts the next proof item by generating textual sub-questions based on logical operator. Note that this model is not applicable for samples whose proof strategy is Fail-proof discussed in §3.1, so we make comparison with it separately.
Metrics. We closely follow previous works to evaluate the performance of models via answer prediction (QA) accuracy and proof generation (PA) accuracy. Since some samples may have multiple gold proofs, a generated proof will be considered correct, as long as its nodes and edges match with the nodes and the edges in any of the gold proofs. Full Accuracy (FA) is also included, where a sample is regarded as correct only both the predicted answer and proof are correct.
4.2 Results under Fully-Supervised Training
We train IBR on the training split of the DU5 dataset and evaluate on the test split of DU5. We compare the performance of IBR with baselines except for EVR in Table 1, while with EVR in Table 2 where only partial test split is included, excluding samples whose proof strategy is Fail-proof. Because EVR always fails on these samples (EVR on these excluded samples is given in Appendix A.5).
Obviously, IBR achieves the best proof generation accuracy (PA) as well as full accuracy (FA) among all baseline models, on samples with every depth. Our model also shows a greater advantage on samples with deeper proof path, e.g., 81.7 vs.
2Refer to Appendix A.1 for implementation details.
3More details are given in Appendix A.2
Results of baselines are obtained from the original papers or by running the released code.
D	0	1	2	3	4	5	all
Cnt	6299	4434	2915	2396	2134	2003	20192
QA	RT	100	98.4	98.4	98.8	99.2	99.8	99.2
PV	100	99.0	98.8	99.1	98.8	99.3	99.3
PB	100	99.9	99.9	100	100	100	99.9
IBR	100	99.2	99.2	98.9	99.3	99.6	99.4
PA	PV	98.4	93.2	84.8	80.5	72.5	65.1	87.1
PB	98.4	94.3	86.1	82.0	76.1	72.2	88.8
IBR	99.5	95.6	93.0	90.7	86.5	81.7	93.5
FA	PV	98.4	93.1	84.8	80.5	72.4	65.1	87.1
PB	98.4	94.3	86.1	82.0	76.1	72.2	88.8
IBR	99.5	95.6	92.9	90.7	86.5	81.6	93.5
Table 1: Results of different models on varying proof depth (D) under the fully-supervised setting. Cnt: sample count, RT: RuleTaker, PV: PROVER, PB: PROBR.
D	0	1	2	3	4	5	all
Cnt	1934	1934	1934	1934	1934	1934	11604
QA	EVR	99.4	99.3	96.9	93.3	88.9	88.3	94.4
IBR	100	99.3	99.6	99.3	99.6	99.5	99.5
PA	EVR	95.8	92.5	87.7	79.3	77.3	68.8	83.6
IBR	98.8	96.4	94.7	92.2	88.7	83.6	92.4
FA	EVR	95.8	92.5	87.7	79.3	77.3	68.8	83.6
IBR	98.8	96.3	94.6	92.2	88.7	83.5	92.3
72.2 on PA when depth is 5, illustrating the superiority of iterative models on complex proof paths. Besides, despite not being the best in answer accuracy (QA), there is a very narrow gap between our model and the best one, which proves that IBR is still a comprehensive model covering both subtasks. When compared to EVR, also an iterative model, IBR shows significantly stronger performance on all metrics, benefiting from our elaborate two-fold reasoning process at each step.
4.3 Using Fewer Training Samples
We also explore the performance of IBR when training using fewer data, ranging from 10k to 30k to all the examples (70k) in DU5. The comparison between our model, PROVER (PV), and PROBR (PB) is shown in Table 3, in all three metrics. Our model significantly has the best proof generation performance than the other two baselines in all cases, due to the iterative architecture requiring less global modeling capability and thus fewer training samples. Although PB shows a promising answer prediction accuracy under fewer-data settings, the performance of IBR is close to it while better than
Table 2: Results of IBR and EVR on a partial test split of DU5 (exclude Fail-proof samples). The models are trained on the train split of DU5.
Data	QA	PA	FA
PV	PB	IBR	PV	PB	IBR	PV	PB	IBR
70k	99.3	99.9	99.4	87.1	88.8	93.5	87.1	88.8	93.5
30k	97.8	99.9	98.3	72.5	86.8	89.8	72.4	86.8	89.7
10k	87.1	99.9	94.3	44.0	72.4	75.7	42.7	72.3	75.4
Table 3: Performance comparison using fewer training samples among IBR, PROVER (PV), and PROBR (PB) on the full test split of DU5 after trained on partial DU5 samples.
Data	QA	PA	FA
EVR	IBR	EVR	IBR	EVR	IBR
70k	94.4	99.5	83.6	92.4	83.6	92.3
30k	95.7	99.4	84.4	88.2	84.4	88.1
10k	96.2	97.9	82.8	71.2	82.8	70.8
Table 4: Performance comparison using fewer training samples among EVR and IBR on partial test split of DU5 (without Fail-proof samples) after trained on partial DU5 samples.
PV, e.g., 94.3 vs. 87.1 under 10k. In addition, in Table 4, we also compare with EVR under the same settings but using a different test set that excludes Fail-proof samples. EVR outperforms IBR under the 10k setting for proof generation, but IBR is stronger if more training samples are available.
4.4 Evaluation of Out-of-Domain Data
We further test the out-of-domain performance of IBR against baselines on Birds-Electricity dataset to evaluate their robustness, where B1 and B2 are two sets from the birds domain, and E1-E4 are four sets from the electricity domain. Results are shown in Table 5 and Table 6. Note that Fail-proof samples are still not involved in the comparison for EVR. Overall, our IBR achieves  $2.5\%$  promotion in PA while an equivalent result on QA, compared to PROVER. Despite being the best one on QA, PROBR is also defeated by IBR on both PA and FA. In addition, our model shows more improvement on the hardest E3 and E4 subsets, which further verifies its robustness. When it comes to EVR, we can find its cross-domain capability is relatively weak as it sees a significant drop in PA, and IBR is superior to it without any doubt. Because the cross-domain generation for intermediate texts is much harder, our usage of high-level node features to finished reasoning can alleviate this challenge.
4.5 Generalization Ability
Generalize to higher depths. Following the previous work (Sun et al., 2021), we test the general
Test	B1	B2	E1	E2	E3	E4	all
Cnt	40	40	162	180	624	4224	5270
QA	RT	97.5	100.0	96.9	98.3	91.8	76.7	80.1
PV	95.0	95.0	100.0	100.0	89.7	84.8	86.5
PB	100.0	100.0	100.0	100.0	98.2	95.6	96.3
IBR	100.0	97.5	100.0	100.0	89.2	84.1	86.0
PA	PV	92.5	95.0	95.1	91.7	72.3	80.6	80.7
PB	100.0	100.0	97.5	93.3	79.3	77.7	79.3
IBR	100.0	100.0	95.6	94.4	80.2	82.4	83.2
FA	PV	92.5	95.0	95.1	91.7	71.8	80.6	80.5
PB	100.0	100.0	97.5	93.3	79.3	77.7	79.3
IBR	100.0	97.5	95.6	94.4	78.2	82.4	82.9
Table 5: Out-of-domain performance comparison among RuleTakers (RT), PROVER (PV), and PROBR (PB) on Birds-Electricity dataset after training on DU5.
Test	B1	B2	E1	E2	E3	E4	all
Cnt	28	28	72	90	312	1206	1736
QA	EVR	67.8	64.2	83.3	80.0	76.2	83.8	81.6
IBR	100.0	96.4	100.0	100.0	92.9	100.0	98.6
PA	EVR	32.1	35.7	58.3	50.0	45.5	70.3	63.1
IBR	100.0	100.0	91.6	91.1	91.3	95.2	94.3
FA	EVR	32.1	32.1	58.3	50.0	45.5	70.3	63.1
IBR	100.0	96.4	91.6	91.1	87.1	95.2	93.5
ization ability of IBR by first training the model on the training splits of DU0, DU1, DU2, and DU3, then test them on the test split of DU5 with deeper proof paths respectively<sup>5</sup>. Results are shown in Table 7. We notice that all models suffer performance degeneration especially when the proof depth of the training set is lower, because it is hard for the model to learn complex reasoning based on simple proof paths. However, IBR still realizes the best performance in terms of PA and FA, especially on DU3, where it gets  $4.2\%$  PA/FA promotion to PROBR and even outperforms PROVER trained on the whole DU5 data. These observations again prove that iterative approaches can better learn the detailed reasoning step by step, obtaining a better generalization capability than at-once models.
Generalize to complex language. We also evaluate whether IBR can be applied to samples where questions and statements are expressed in more human-like natural language. Following Clark et al. (2020), we train models on the combined training
Table 6: Out-of-domain performance comparison among EVR and IBR on partial Birds-Electricity dataset (exclude Fail-proof samples) after training on DU5.
Data	QA	PA	FA
PV	PB	IBR	PV	PB	IBR	PV	PB	IBR
DU0	68.7	56.9	53.5	44.4	50.7	47.0	42.8	41.3	47.0
DU1	73.7	97.7	73.1	63.8	63.9	64.6	61.9	63.9	64.5
DU2	89.6	99.9	89.6	72.6	74.5	76.3	72.3	74.4	76.2
DU3	98.6	99.9	98.6	79.1	83.2	87.4	79.1	83.2	87.4
DU5	99.3	99.9	99.4	87.1	88.8	93.5	87.1	88.8	93.4
Table 7: Performance of generalization ability between PROVER (PV), PROBR (PB), and IBR when testing on the test split of DU5, after trained on DU0, DU1, DU2, DU3, and DU5, respectively.
D	0	1	2	3	4	all
Cnt	2968	2406	1443	1036	142	8008
QA	PV	99.7	98.6	98.2	96.5	88.0	98.4
PB	99.8	99.7	99.9	99.8	100	99.8
IBR	99.9	98.8	97.5	96.3	88.7	98.4
PA	PV	99.5	98.0	88.9	90.0	76.1	95.4
PB	99.5	98.0	88.9	90.1	82.4	95.6
IBR	99.8	98.8	91.1	89.0	75.3	95.9
FA	PV	99.4	97.3	88.7	89.9	76.1	95.1
PB	99.4	98.0	88.9	90.1	82.4	95.5
IBR	99.7	98.1	90.9	89.0	75.3	95.7
Table 8: Performance on ParaRules test set, after trained on combined D3+ParaRules training partitions, including PROVER (PV), PROBR (PB), and IBR.
partitions of DU3 and ParaRules then test them on the ParaRules test set. To our best knowledge, it is the dataset that is closest to real-world applications. Table 8 demonstrates that our model sees a slight promotion in PA/FA while a similar accuracy as PROVER in QA, indicating that IBR still has good applicability when doing reasoning on more complicated and natural texts.
To explore the effects between different components in our model, we consider the following ablations: 1) IBR +Gold-Parent: given the gold parent nodes during inference to explore the accuracy of child node prediction; 2) IBR +Gold-Child: given the gold child nodes to verify the accuracy of parent node prediction; 3)  $w / o$  QA: removing QA task in loss to check its impact on proof generation; 4)  $w / o$  node LSTM: using mean pooling rather than LSTM encoding to get the representations of nodes; 5)  $w / o$  focus LSTM: Removing the supplementary LSTM in path focus selection.
Results on the whole DU5 test split are given in Table 9. As the numeric performance shows,
<sup>5</sup>We remove the position embedding in path focus selection to proceed to this test, see Appendix A.1 for details
Models	QA	PA	FA
IBR	99.4	93.5	93.5
IBR +Gold-Parent	99.4	95.6	95.3
IBR +Gold-Child	99.4	99.6	99.3
w/o QA	-	93.7	-
w/o node LSTM	99.5	93.2	93.2
w/o focus LSTM	99.6	92.6	92.4
Table 9: Results of ablation studies on DU5 dataset. We use IBR as the backbone.
giving either gold parent nodes or gold child nodes can benefit the performance especially the later one. This signifies that our parent node prediction achieves promising accuracy while the prediction of child nodes can be further improved. Moreover, IBR can still learn to generate proofs without supervision from answers. And LSTM encoders attribute to a better representation of both the nodes and the path that has been derived.
5.2 Latency Analysis
To demonstrate the computational efficiency of IBR, we compare the per sample inference time of IBR with EVR, also an iterative proof generation model, on the test split of DU5. Additionally, we also compare the per sample inference time of IBR with PROVER and PROBR, both at-onc models. All models are tested on one NVIDIA Tesla-V100 GPU with the same batch size and the beam size of IBR sets to 1 for a fair comparison. As shown in Figure 4, our IBR could achieve up to  $\times 119.5$  speedup compared with EVR, benefiting from our reasoning based on node and path features rather than intermediate texts. It is also noticeable that the runtime of EVR grows linearly with depth, while such an effect is slight on our model. Because EVR needs to infer on all contexts at every step, but IBR uses a simplified parent node prediction based on the derived path. Figure 5 illustrates that IBR is also faster than PROVER because PROVER has some constraints during post-processing in inference, like ensuring proof connectivity, which takes extra time.
This paper presents IBR, a proof generation model via iterative backward reasoning for rule-based QA tasks. We equip the reasoning procedure with detailed hidden state tracking by predicting nodes and edges in the proof path iteratively backward from the question, and allow the model to reason on the elaborate representations of nodes and his-
Figure 4: Per-sample inference runtime (in second) of EVR and IBR on DU5 dataset with varying depths.
Figure 5: Per-sample inference runtime (in second) of PROVER (PV), IBR, and PROBR (PB) on DU5 dataset with varying depths.
tory paths. Our model is more interpretable than previous at- once models, and is also more effective and efficient than former iterative models. Experiments also demonstrate the superiority of IBR to various baselines on proof generation under various settings.
This work was partially supported by the National Natural Science Foundation of China (61876053, 62006062, 62176076), the Shenzhen Foundational Research Funding JCYJ20210324115614039, Shenzhen Science and Technology Program JSGG20210802154400001, and the Joint Lab of HITSZ and China Merchants Securities.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533-1544, Seattle, Washington, USA. Association for Computational Linguistics.
Jonathan Berant and Percy Liang. 2014. Semantic parsing via paraphrasing. In Proceedings of the 52nd Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 1415-1425, Baltimore, Maryland. Association for Computational Linguistics.
Deng Cai and Wai Lam. 2019. Core semantic first: A top-down approach for AMR parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3799-3809, Hong Kong, China. Association for Computational Linguistics.
Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2020. Transformers as soft reasoners over language. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 3882-3890. ijcai.org.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378, Minneapolis, Minnesota. Association for Computational Linguistics.
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 1-13, Hong Kong, China. Association for Computational Linguistics.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611, Vancouver, Canada. Association for Computational Linguistics.
Zhengzhong Liang, Steven Bethard, and Mihai Surdeanu. 2021. Explainable multi-hop verbal reasoning through internal monologue. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1225-1250, Online. Association for Computational Linguistics.
Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. 2019. Reasoning over paragraph effects in situations. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 58-62, Hong Kong, China. Association for Computational Linguistics.
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 3622-3628. ijcai.org.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv preprint, abs/1907.11692.
Mark A Musen and Johan Van Der Lei. 1988. Of brittleness and bottlenecks: Challenges in the creation of pattern-recognition and expert-system models. In Machine Intelligence and Pattern Recognition, 7:335-352.
Allen Newell and Herbert Simon. 1956. The logic theory machine-a complex information processing system. IRE Transactions on information theory, 2(3):61-79.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Swarnadeep Saha, Sayan Ghosh, Shashank Srivastava, and Mohit Bansal. 2020. PRover: Proof generation for interpretable reasoning over rules. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 122-136, Online. Association for Computational Linguistics.
Nan Shao, Yiming Cui, Ting Liu, Shijin Wang, and Guoping Hu. 2020. Is Graph Structure Necessary for Multi-hop Question Answering? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7187-7192, Online. Association for Computational Linguistics.
Changzhi Sun, Xinbo Zhang, Jiangjie Chen, Chun Gan, Yuanbin Wu, Jiaze Chen, Hao Zhou, and Lei Li. 2021. Probabilistic graph reasoning for natural proof generation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages
3140-3151, Online. Association for Computational Linguistics.
Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. 2021. ProofWriter: Generating implications, proofs, and abductive statements over natural language. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP* 2021, pages 3621-3634, Online. Association for Computational Linguistics.
Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. 2019. QuaRTz: An open-domain dataset of qualitative relationship questions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5941-5946, Hong Kong, China. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008.
Leon Weber, Pasquale Minervini, Jannes Münchmeyer, Ulf Leser, and Tim Roktäschel. 2019. NLProlog: Reasoning with weak unification for question answering in natural language. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6151-6161, Florence, Italy. Association for Computational Linguistics.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287-302.
Jason Weston, Antoine Bordes, Sumit Chopra, and Tomás Mikolov. 2016. Towards ai-complete question answering: A set of prerequisite toy tasks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Computational Linguistics.
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. Qanet: Combining local convolution with global self-attention for reading comprehension. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2020. Reclor: A reading comprehension dataset requiring logical reasoning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Luke S. Zettlemoyer and Michael Collins. 2012. Learning to map sentences to logical form: structured classification with probabilistic categorial grammars. ArXiv preprint, abs/1207.1420.
Parameter	Value
Training Epochs	8
Optimizer	AdamW
Batch Size	16
RoBERTa Learning rate	1e-5
QA and Strategy Pre Learning rate	1e-5
Parent Node Pre Learning rate	2e-4
Child Node Pre Learning rate	5e-4
All LSTM Learning rate	1e-3
Dropout Rate	0.1
LSTM hidden state for parent node and child node encoding	1024
LSTM hidden state for path encoding in parent node prediction	1024
Transformer hidden state in path focus selection	1024
LSTM hidden state in path focus selection	256
Seed	42
We implement our model based on PyTorch along with Huggingface-Transformers toolkit<sup>6</sup>. We use RoBERTaLarge model<sup>7</sup> as our backbone encoder to generate token-level representations. Table 10 shows the implementation details of IBR, including learning rates for different modules. All linear layers used in our model have one layer. The model trained after 8 epochs will be used in the evaluation. We remove functional words without lexical meaning like "a" and "the" from facts, rules, and questions to shorten the input length, so each training epoch takes about 2 hours. We select these hyper-parameters according to tuning them empirically based on the performance. All experiments are run on NVIDIA Tesla-V100 GPUs. The main experiment performance of IBR fluctuates by one point.
We next introduce the details of the three datasets used in our experiment. All of them are firstly applied in rule-based QA and proof generation tasks in Clark et al., 2020.
$^{6}$ https://github.com/huggingface/transformers
 $^{7}$ https://huggingface.co/roberta-large
Table 10: Implementation details of IBR.
Split	D	Num	Fail-proof Num	Proof Num	Avg. Node
Train	0	21,359	14,597	6,762	0.62
1	15,380	8,618	6,762	1.82
2	10,112	3,350	6,762	3.37
3	8,389	1,627	6,762	4.98
4	7,456	694	6,762	6.90
5	6,987	225	6,762	9.26
all	69,683	29,111	40,572	3.35
Test	0	6,299	4,365	1,934	0.59
1	4,434	2,500	1,934	1.77
2	2,915	981	1,934	3.36
3	2,396	462	1,934	4.99
4	2,134	200	1,934	6.98
5	2,003	69	1,934	9.47
all	20,181	8,577	11,604	3.33
Table 11: The statistics of train and test split in DU5 dataset. Fail-proof and Proof indicate different proof strategies we discussed in §3.1. Avg. Node indicates the average node number in a proof path.