SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 8: AUSHD with  $90\%$  confidence intervals (in the parenthesis), for synthetic data and for low and regular data regimes ( $N = 1056$  and  $N = 3200$  resp.).
AIT	BALD	Random	GIT (ours)	priv. GIT
bidiag	1056	24.7 (24.1, 25.5)	21.9 (21.1, 22.8)	22.0 (21.5, 22.7)	20.0 (19.5, 20.6)	19.9 (18.6, 20.9)
3200	14.0 (13.0, 15.4)	13.2 (12.5, 14.0)	11.1 (10.5, 12.1)	9.4 (9.0, 9.9)	9.3 (8.0, 10.3)
chain	1056	14.9 (14.4, 15.4)	12.2 (11.8, 12.7)	13.5 (13.1, 13.9)	11.7 (11.3, 12.1)	12.2 (11.4, 13.3)
3200	7.7 (7.3, 8.1)	7.2 (6.8, 7.7)	6.3 (6.0, 6.6)	5.6 (5.2, 6.0)	6.3 (5.2, 8.5)
collider	1056	16.0 (15.2, 16.7)	16.1 (15.5, 16.7)	14.6 (14.1, 15.1)	14.4 (13.4, 15.2)	11.8 (10.9, 13.0)
3200	10.9 (10.2, 11.7)	12.2 (11.6, 12.7)	9.7 (9.2, 10.3)	12.1 (10.9, 13.1)	7.8 (6.9, 8.8)
fulldag	1056	133.0 (131.2, 134.7)	141.6 (139.1, 144.2)	121.7 (120.4, 122.9)	119.8 (118.7, 120.8)	120.7 (119.1, 122.1)
3200	72.8 (71.0, 74.5)	100.6 (97.8, 103.8)	63.4 (62.0, 64.7)	67.9 (66.0, 70.3)	63.4 (61.2, 64.9)
jungle	1056	23.2 (21.9, 24.6)	20.6 (19.6, 21.7)	20.9 (20.1, 21.7)	14.7 (14.1, 15.4)	13.9 (12.4, 15.5)
3200	11.2 (10.7, 11.9)	13.3 (12.3, 14.3)	9.1 (8.8, 9.5)	6.9 (6.5, 7.2)	6.9 (5.5, 8.3)
random	1056	42.1 (40.5, 43.6)	43.1 (41.5, 44.9)	35.6 (34.6, 36.7)	34.6 (33.7, 35.7)	31.9 (30.4, 34.6)
3200	21.3 (20.4, 22.3)	30.7 (29.0, 32.5)	16.5 (15.8, 17.3)	17.0 (16.3, 17.7)	14.5 (13.6, 15.6)
Table 9: AUSHD with  $90\%$  confidence intervals (in the parenthesis), for real-world data and for low and regular data regimes ( $N = 1056$  and  $N = 3200$  resp.).
AIT	CBED	Random	GIT (ours)	priv. GIT
alarm	1056	42.8 (41.8, 43.8)	36.8 (35.8, 37.8)	39.7 (38.6, 40.8)	28.8 (28.3, 29.3)	28.5 (27.0, 29.6)
3200	35.0 (33.6, 36.4)	31.6 (30.3, 33.1)	28.8 (27.6, 30.8)	24.0 (23.4, 24.9)	21.5 (20.7, 23.1)
asia	1056	3.6 (2.9, 4.5)	3.5 (2.8, 4.3)	2.0 (1.8, 2.1)	2.2 (2.0, 2.5)	1.8 (1.7, 1.9)
3200	2.4 (1.9, 3.3)	2.1 (1.9, 2.5)	1.3 (1.2, 1.4)	1.5 (1.4, 1.6)	1.1 (1.0, 1.2)
cancer	1056	2.0 (1.9, 2.1)	2.1 (2.0, 2.3)	2.4 (2.2, 2.6)	2.4 (2.2, 2.5)	2.1 (1.6, 2.6)
3200	1.8 (1.6, 2.0)	2.1 (1.9, 2.2)	2.2 (2.0, 2.3)	2.2 (2.0, 2.4)	2.2 (1.7, 2.6)
child	1056	14.4 (13.7, 15.2)	10.4 (9.6, 11.2)	11.1 (10.7, 11.6)	8.3 (8.0, 8.7)	7.9 (7.0, 9.0)
3200	7.8 (7.1, 8.6)	7.1 (6.5, 8.0)	5.0 (4.7, 5.5)	4.5 (4.2, 4.8)	3.9 (3.2, 4.7)
earthquake	1056	0.5 (0.4, 0.6)	0.5 (0.4, 0.6)	0.4 (0.3, 0.5)	0.6 (0.5, 0.7)	0.4 (0.2, 0.6)
3200	0.2 (0.1, 0.3)	0.2 (0.1, 0.2)	0.1 (0.1, 0.2)	0.3 (0.2, 0.5)	0.1 (0.1, 0.2)
sachs	1056	3.1 (2.9, 3.3)	2.9 (2.6, 3.1)	2.9 (2.7, 3.1)	2.5 (2.4, 2.7)	2.5 (2.2, 2.8)
3200	1.4 (1.3, 1.6)	1.9 (1.7, 2.2)	1.2 (1.1, 1.3)	1.1 (1.0, 1.3)	0.9 (0.8, 1.0)
Table 10: SHD with  $90\%$  confidence intervals (in the parenthesis), for synthetic data and for low and regular data regimes ( $N = 1056$  and  $N = 3200$  resp.).
AIT	CBED	Random	GIT (ours)	GIT-priv.
bidiag	1056	11.4 (10.3, 12.4)	10.1 (9.2, 11.0)	7.8 (7.0, 8.5)	6.3 (5.7, 7.0)	7.4 (6.2, 8.6)
3200	5.2 (4.2, 6.3)	7.8 (6.9, 8.7)	2.8 (2.3, 3.4)	2.4 (1.8, 2.9)	2.2 (0.8, 3.6)
chain	1056	5.6 (4.8, 6.4)	5.4 (4.6, 6.1)	4.3 (3.8, 4.9)	3.6 (3.0, 4.2)	3.6 (2.0, 4.8)
3200	3.2 (2.6, 3.7)	3.9 (3.4, 4.3)	2.2 (1.7, 2.6)	1.8 (1.3, 2.3)	1.8 (0.2, 2.6)
collider	1056	11.0 (10.1, 11.9)	11.8 (11.0, 12.7)	9.8 (9.1, 10.6)	13.3 (12.2, 14.4)	9.8 (7.6, 12.0)
3200	4.8 (3.8, 5.9)	7.9 (6.8, 8.9)	3.7 (2.8, 4.6)	9.7 (7.7, 11.6)	3.4 (1.4, 5.0)
fulldag	1056	64.4 (61.8, 67.0)	91.4 (86.8, 96.0)	52.1 (50.0, 54.3)	55.8 (53.4, 58.0)	53.4 (49.8, 57.0)
3200	32.0 (30.0, 33.8)	75.4 (71.8, 79.0)	25.1 (22.8, 27.2)	27.3 (25.1, 29.8)	20.8 (19.6, 21.8)
jungle	1056	10.4 (9.2, 11.6)	11.6 (10.1, 13.2)	5.7 (5.0, 6.5)	5.1 (4.4, 5.8)	5.2 (3.0, 7.4)
3200	3.5 (3.1, 3.9)	8.3 (7.2, 9.4)	1.9 (1.5, 2.3)	2.2 (1.8, 2.7)	3.0 (2.0, 4.0)
random	1056	18.8 (17.3, 20.3)	27.5 (25.6, 29.5)	11.3 (10.0, 12.5)	12.5 (11.3, 13.5)	11.0 (9.2, 13.0)
3200	8.3 (7.0, 9.4)	22.1 (19.6, 24.4)	5.0 (4.3, 5.8)	5.3 (4.4, 6.1)	3.8 (2.2, 5.4)
F.3 ENCO - large interventional batch experiment
We provide SHD training curves for experiments with the large interventional batch in Figure 9.
F.4 ENCO - monte carlo sampling evaluation
We provide a performance evaluation of GIT with different amount of graphs sampled from the model in Figure 10.
Table 11: SHD with  $90\%$  confidence intervals (in the parenthesis), for real-world data and for low and regular data regimes ( $N = 1056$  and  $N = 3200$  resp.).
AIT	CBED	Random	GIT (ours)	priv. GIT
alarm	1056	35.76 (34.04, 37.52)	28.44 (26.68, 30.16)	26.0 (24.71, 27.29)	19.84 (19.0, 20.68)	25.0 (23.2, 27.0)
3200	26.15 (24.15, 28.23)	24.33 (21.67, 27.0)	16.0 (14.57, 17.14)	20.0 (18.67, 21.33)	15.2 (14.6, 15.8)
asia	1056	2.0 (1.2, 2.68)	1.96 (1.44, 2.4)	0.96 (0.8, 1.12)	1.2 (1.0, 1.36)	1.2 (0.8, 1.4)
3200	1.56 (1.12, 1.92)	1.28 (1.0, 1.48)	0.88 (0.79, 1.0)	1.12 (0.96, 1.24)	0.8 (0.6, 1.2)
cancer	1056	1.72 (1.48, 2.0)	2.2 (2.0, 2.4)	2.28 (2.04, 2.48)	2.12 (1.84, 2.4)	2.2 (1.8, 2.4)
3200	1.8 (1.6, 2.0)	1.96 (1.72, 2.2)	1.84 (1.6, 2.12)	2.0 (1.76, 2.24)	2.4 (2.0, 2.8)
child	1056	7.32 (5.92, 8.68)	6.36 (5.52, 7.16)	3.52 (2.84, 4.2)	3.72 (3.2, 4.24)	2.8 (1.4, 4.0)
3200	3.2 (2.56, 3.8)	4.68 (3.8, 5.48)	1.04 (0.7, 1.35)	2.16 (1.8, 2.52)	1.8 (0.4, 3.0)
earthquake	1056	0.12 (0.0, 0.2)	0.12 (0.0, 0.2)	0.0 (0.0, 0.0)	0.24 (0.08, 0.36)	0.0 (0.0, 0.0)
3200	0.04 (-0.04, 0.08)	0.0 (0.0, 0.0)	0.0 (0.0, 0.0)	0.2 (0.08, 0.32)	0.0 (0.0, 0.0)
sachs	1056	0.84 (0.68, 1.0)	1.28 (0.96, 1.6)	0.6 (0.4, 0.8)	0.52 (0.32, 0.72)	0.4 (0.0, 0.8)
3200	0.48 (0.32, 0.64)	1.48 (1.16, 1.76)	0.24 (0.08, 0.36)	0.48 (0.28, 0.68)	0.0 (0.0, 0.0)
Figure 7: Expected SHD metric for different acquisition methods on top of the ENCO framework, for synthetic graphs with 25 nodes.  $95\%$  bootstrap confidence intervals are shown.
F.5 ENCO - Large Synthetic Graphs
In order to study the scalability of our method, we perform an additional evulation on selected synthetic graphs in which we increase the number of nodes to 100. We comapre the performance of different acquisition methods used with ENCO in Figure 11. We observe that GIT exhibits very good results, converging to lower SHD values with significantly less acquisition steps compared to all the other methods. This confirms the superiority of GIT, even within a larger graph regime.
F.6 ENCO - Correlation Scores
In Figure 12, we present the correlation of scores of the tested targeting methods. Importantly, the high correlation of GIT and GIT-privileged supports the hypothesis that imaginary gradients are a credible proxy of the true gradients and thus validates GIT. Otherwise, correlations are relatively small, suggesting that the studied methods use different decision mechanisms. Understanding this phenomenon is an interesting future research direction.
Below, we provide more details about computing the correlations. Let us denote by  $s_{b,i}^{m}$  the score produced by method  $m$  for the batch  $b$  and the node  $i$ . In order to eliminate effects such as changing
Figure 8: Expected SHD metric for different acquisition methods on top of the ENCO framework, for graphs from BnLearn dataset.  $95\%$  bootstrap confidence intervals are shown.
Figure 9: Expected SHD metric for GIT with an interventional batch size of 1024 samples.  $95\%$  bootstrap confidence intervals are shown, and results were computed using 3 random seeds.
scores scales during the discovery process, we normalize the scores as  $\bar{s}_{b,i}^{m} \coloneqq \frac{s_{b,i}^{m}}{\sum_{j=1}^{N} s_{b,j}^{m}}$ . For every pair of methods  $m$ ,  $m'$  and node  $i$ , we compute Spearman's rank correlation score  $r_{s}\left(\bar{s}_{\cdot,i}^{m}, \bar{s}_{\cdot,i}^{m'}\right)$ . We average over the nodes to get the scalar correlation value  $\mathrm{corr}(m, m') \coloneqq \frac{\sum_{j=1}^{N} r_{s}\left(\bar{s}_{\cdot,i}^{m}, \bar{s}_{\cdot,i}^{m'}\right)}{N}$ .
In addition, we present Pearson's correlations in Figure 13. Conclusions from the analysis of the Spearman's rank correlation hold; in particular, the correlation between GIT and GIT-privileged is high.
Figure 10: Expected SHD metric for GIT with different numbers of graphs samples used to estimate score for interventions (see line 2 in Algorithm 2).  $95\%$  bootstrap confidence intervals are shown, results were computed using 10 random seeds.
Figure 11: Expected SHD metric for different acquisition methods on top of the ENCO framework, for synthetic graphs with 100 nodes.  $95\%$  bootstrap confidence intervals are shown.
F.7 ENCO - Intervention Targets Distribution
In this section, we provide additional histograms and plots with regard to the interventional target distributions obtained by different intervention methods as discussed in Section 5.4 in the main text.
In Figure 14, we present the histograms of the target distributions for the real graphs for each of the intervention acquisition methods. Note that those histograms represent the same information as the node coloring in Figure 4. It may be observed that the distributions obtained by GIT concentrate on fewer nodes than those obtained by the AIT and CBED approaches. The only exceptions being the
Figure 12: Spearman's rank correlation of the scores produced by different acquisition methods, averaged over nodes.
Figure 13: Pearson correlation of the scores produced by different acquisition methods, averaged over nodes. We can see similar trends as in the case of Spearman's rank correlation, in particular, a high correlation of GIT and GIT-privileged.
Figure 14: The histograms of chosen interventional targets in all data acquisition steps for different strategies computed on the real-world data.
sachs and child datasets, for which the entropy of CBED approach is smaller (recall Figure 4). Note, however, that CBED underperforms on those graphs (recall Figure 3 in the main text or see Figure 8). This is in contrast to GIT, which maintains good performance.
Finally, in Figure 15, we present the interventional target distribution on the alarm graph. We observe that each method intervenes on at least one node incident to the critical edges in the Markov
Figure 15: The interventional target distribution for the alarm graph. The green color represents the edges for which there exists a graph in the Markov Equivalence Class that has the corresponding connection reversed. Black color is used to indicate node for each no data is collected. We may observe that each method intervenes on at least one node incident to the critical edges. However, both AIT and CBED do not converge for this dataset and struggle to achieve good results.
Equivalence Class (as indicated by the green color in the plot). However, both AIT and CBED struggle to achieve convergence and suffer low performance, as can be observed in Figure 8.
F.7.1 ENCO - Obtained Synthetic Graphs
Figure 16: The interventional target distributions obtained by different strategies on synthetic data. The probability is represented by the intensity of the node's color. For clarity of the presentation, we choose not to color the critical edges in the corresponding Markov Equivalence Classes. This is because all edges of all the presented graphs would need to be colored. The only exception is the collider graph, which is alone in its Markov Equivalence Class.
In addition, we present the results obtained for the synthetic graphs in Figure 16 and the corresponding histograms in Figure 17. Note that in this case the results are also averaged by different ground truth distributions, which means that any regularities in selecting the nodes come rather from the graph structure than from data distribution.
Interestingly, we may observe that for the jungle and chain graphs GIT often intervenes on the nodes which are the first ones in the topological order (as indicated by low node numbers in the plots). This is again intuitive, as intervening on those nodes can impact more variables lower in the hierarchy. In addition, note that for the chain graph, knowing its Markov Equivalence Class and setting the directionality of an edge automatically makes it possible to determine the directionality of edges for all subsequent nodes in the topological order. Hence intervening on the nodes which are the first ones in the ordering may convey more information and is desired.
We may also observe that the CBED seems to focus only on the first nodes in the topological order, despite the data distribution, which in some graphs (as the chain graph) may be desired, but in others seems to be an oversimplified solution. Note that CBED often struggles to converge – this may be observed in Figure 7.
Figure 17: The histograms of chosen interventional targets in all data acquisition steps for different strategies computed on the synthetic data.
F.7.2 Discussion on Small Real-World Graphs
We provide a more detailed discussion on the differences between the earthquake and cancer graph distributions and the way it affects the GIT method.
Consider Figure 4 in the main text. Note that the middle node in the earthquake graph corresponds to setting off a burglary alarm, an event very unlikely to happen in observational data but which, when occurs, triggers a change in the distributions of the nodes lower in the hierarchy (see the conditional distributions in Table 12). The chance of starting an alarm is also very high in case a burglary has happened (the left-most node in the graph). Hence the GIT concentrates on those two nodes as they have the largest impact on the entailed distribution.
A similar situation can be observed for the cancer graph, where the middle node corresponds to a binary variable indicating the probability of developing the illness. Even though the two parents of the cancer variable (pollution and smoke, represented by nodes 0 and 1, respectively) share a causal relationship with cancer, their impact on the cancer variable is limited. In other words, the chances of developing cancer, no matter whether being subject to high or low pollution or being a smoker or not, remain rather small (see the conditional distributions for cancer variable in Table 13). Hence, the only way in which one can gather more information about the impact of having cancer on the distributions of its child variables (nodes 3 and 4) is by performing an intervention. In consequence, it may be observed that GIT prefers to select nodes that allow to gather data that otherwise would be hard to acquire in the purely observational setting.
Table 12: The conditional distribution in the earthquake graph.
Variable	Parents	Values	Distribution
Burglary	-	[True, False]	[0.01, 0.99]
Earthquake	-	[True, False]	[0.02, 0.99]
Alarm	Burglar=True, Earthquake=True	[True, False]	[0.95, 0.05]
Alarm	Burglar=False, Earthquake=True	[True, False]	[0.29, 0.71]
Alarm	Burglar=True, Earthquake=False	[True, False]	[0.94, 0.06]
Alarm	Burglar=False, Earthquake=False	[True, False]	[0.001, 0.999]
John Calls	Alarm=True	[True, False]	[0.9, 0.1]
John Calls	Alarm=False	[True, False]	[0.05, 0.95]
Mary Calls	Alarm=True	[True, False]	[0.7, 0.3]
Mary Calls	Alarm=False	[True, False]	[0.01, 0.99]
Table 13: The conditional distribution in the cancer graph.
Variable	Parents	Values	Distribution
Pollution	-	[Low, High]	[0.9, 0.1]
Smoker	-	[True, False]	[0.3, 0.7]
Cancer	Pollution=Low, Smoker=True	[True, False]	[0.03, 0.97]
Cancer	Pollution=High, Smoker=True	[True, False]	[0.05, 0.95]
Cancer	Pollution=Low, Smoker=False	[True, False]	[0.001, 0.999]
Cancer	Pollution=High, Smoker=False	[True, False]	[0.02, 0.98]
Xray	Cancer=True	[True, False]	[0.9, 0.1]
Xray	Cancer=False	[True, False]	[0.2, 0.8]
Dyspnoea	Cancer=True	[True, False]	[0.65, 0.35]
Dyspnoea	Cancer=False	[True, False]	[0.3, 0.7]
F.8 ENCO - Experiments with Pre-Initialization
Figure 18: Histograms of intervention targets chosen by GIT. The red color corresponds to the selected node, while the green color indicates the node's parents. The edges on which standard initialization was used are indicated by gray dashed lines. The rest of the solution is given in the initialization.
In addition to the discussion on the target distributions in the case of pre-initializing parts of the graph with the ground truth solution (presented in the main text for synthetic graphs in Section 5.4), we present results of the same experiment computed on the real-world graphs. The results are presented in Figure 18.
Similarly as for the synthetic graphs, here we also observe that the GIT concentrates either on the selected node  $v$  or on its parents (denoted respectively by red and green colors in the plots).