SlowGuess/NAACL_2022_OCR huggingface dataset:
DU0-DU5: A series of synthesized datasets where rules and facts are all generated via manually designed logical programming, while questions are generated by combining random logical operations among them. Data are divided into 5 subsets according to their maximum reasoning depth (D) in the proof path,  $\mathrm{D} = 0, 1, 2, 3, 5$ . There are 100k questions in each subset, where 70k / 10k / 20k samples in the train / validation / test partition respectively.  $\mathrm{D} = 0$  means that the question can be proven directly using a fact in contexts. In our experiment in §4, we only use the data from DU5 for testing because it covers all possible depths, while the train set is the train split in DU5 except §4.5, where we use train split from DU0, DU1, DU2 and DU3 for training. We provide some statistics of DU5 in Table 11.
Birds-Electricity: It is a set of data that only contains 5k test samples for the evaluation of robustness and out-of-domain performance of models. The Birds data only require reasoning up to depth 1 and 2 (B1 and B2), while Electricity data have reasoning depths ranging from 1 to 4. Both of them include new vocabulary that is not included in DU0-DU5.
ParaRules: A more challenging dataset contains paraphrased samples on the synthesized ones via crowdsourcing. It has 40k questions against about 2k theories. The statements are expressed in a more natural way, posing a discrepancy between DU0-DU5. It has 28k / 4k / 8k samples in the train / validation / test split respectively. In §4.5, we combine it with the extensive DU3 for training,
resulting in a train set containing 119k samples.
A.3 Possible Limitations of Our Model
Since our strategy prediction module and operations corresponding to different strategies in node prediction modules are specially designed for the current datasets, we may need to redesign some specific operations to reach the best performance, if some novel proof types are included in new datasets. But we believe our architecture will still take effect without modification. Besides, the interpretability of IBR is not so strong as former works like EVR that make use of intermediate texts.
A.4 Strategy Accuracy of IBR
D	Cnt	Strategy Accuracy
0	6299	99.9
1	4434	99.1
2	2915	99.3
3	2396	99.0
4	2134	99.2
5	2003	99.7
All	20192	99.4
We provide the strategy prediction accuracy on DU5 in Table 12. It proves that IBR is also well able to make predictions on the proof strategies. This is partly due to RoBERTa's powerful representation capability. On the other hand, there is a certain connection between the answer to the question and the strategy, and there are some common elements at the semantic representation level that can be learned together.
A.5 Performance of EVR and IBR on Fail-proof Samples
As we have discussed in §4.2, EVR (Liang et al., 2021) is not applicable for samples containing Failproof proofs, because it cannot obtain proper intermediate questions to proceed correct following reasoning. Here, we compare our model with EVR on these samples in DU0-DU5, as illustrated in Table 13. Although EVR can achieve promising performance on answer prediction (QA) for these samples, it cannot generate any correct proof path in such cases, which have already been discussed in its original paper.
Table 12: Strategy accuracy of IBR on test split of DU5 after training on training split of DU5.
D	0	1	2	3	4	5	all
Cnt	4365	2500	981	462	200	69	8577
QA	EVR	99.7	99.1	98.9	99.1	98.5	100	99.4
IBR	100	99.1	98.3	97.6	96.5	100	99.3
PA	EVR	0.0	0.0	0.0	0.0	0.0	0.0	0.0
IBR	99.8	95.0	89.5	84.4	65.5	28.9	95.0
FA	EVR	0.0	0.0	0.0	0.0	0.0	0.0	0.0
IBR	99.8	95.0	89.5	84.4	65.5	28.9	95.0
Table 13: The performance of EVR and IBR on the partial test split of DU5 that only contains samples whose proofs strategies are Fail-proof.
A.6 Proof Generation samples
We provide some proof generation samples in Figure 6 for a better understanding of this task, where questions, all contexts, and the proof path generated by our IBR are given (all consistent with the given labels).
$\mathbb{R}_1$  : If someone is nice and kind then they like the bear.
$\mathbf{R}_2$  : If someone sees the dog and they eat the bear then the bear is cold.
$\mathbb{R}_3$  : If someone is big then they eat the cat.
$\mathbb{R}_4$  : If someone is big then they do not see the rabbit.
$\mathbf{R}_5$  : If someone is not big and they do not eat the dog then the dog is cold.
$\mathrm{R}_6$  : If someone is cold then they like the rabbit.
$\mathbb{R}_7$  : If someone likes the rabbit then they see the dog.
$\mathbb{R}_8$  : If the dog eats the cat then the dog is kind.
$\mathbf{R}_9$  : If someone likes the dog and they do not eat the cat then the dog eats the bear.
$\mathrm{F}_1$  : The bear eats the cat.  
 $\mathrm{F}_2$  : The bear eats the rab  
 $\mathrm{F}_3$  : The cat eats the dog.  
 $\mathrm{F}_4$  : The cat eats the rabbi  
 $\mathrm{F}_5$  : The cat likes the bear  
 $\mathrm{F}_6$  : The cat sees the rabbi  
 $\mathrm{F}_7$  : The dog is round.  
 $\mathrm{F}_8$  : The dog likes the bear  
 $\mathrm{F}_9$  : The dog likes the cat.
$\mathrm{F}_{10}$  : The dog sees the bear.   
 $\mathrm{F}_{11}$  : The rabbit eats the bear.   
 $\mathrm{F}_{12}$  : The rabbit is big.   
 $\mathrm{F}_{13}$  : The rabbit is cold.   
 $\mathrm{F}_{14}$  : The rabbit is not kind.   
 $\mathrm{F}_{15}$  : The rabbit does not like the cat.   
 $\mathrm{F}_{16}$  : The rabbit sees the bear.
$\mathbf{Q}_{1}$ : The bear is cold.
Proof generated by IBR:
Proof Depth  $= 3$  ,Strategy: Proof
$\mathbf{Q}_2$  : The dog does not see the dog.  $\mathbf{A}_2$  : False Proof generated by IBR: Proof Depth  $= 3$  ,Strategy: Proof
Q3: The dog eats the bear. A3: False Proof generated by IBR: Proof Depth  $= 1$  ,Strategy
Figure 6: Some proof cases generated by IBR, along with all contexts and questions, including two proof strategies, Proof and Fail-proof.