Bayesian Active Causal Discovery with Multi-Fidelity Experiments
Authors: Zeyu Zhang
Chaozhuo Li
Xu Chen
Xing Xie
Venue: NeurIPS 2023 poster (2023)
Keywords: Causal Discovery
Active Learning
Multi-fidelity

Abstract: This paper studies the problem of active causal discovery when the experiments can be done based on multi-fidelity oracles, where higher fidelity experiments are more precise and expensive, while the lower ones are cheaper but less accurate. In this paper, we formally define the task of multi-fidelity active causal discovery, and design a probabilistic model for solving this problem. In specific, we first introduce a mutual-information based acquisition function to determine which variable should be intervened at which fidelity, and then a cascading model is proposed to capture the correlations between different fidelity oracles. Beyond the above basic framework, we also extend it to the batch intervention scenario. We find that the theoretical foundations behind the widely used and efficient greedy method do not hold in our problem. To solve this problem, we introduce a new concept called $\epsilon$-submodular, and design a constraint based fidelity model to theoretically validate the greedy method. We conduct extensive experiments to demonstrate the effectiveness of our model.

Bayesian Active Causal Discovery with Multi-Fidelity Experiments
Zeyu Zhang
Chaozhuo Li Microsoft Research Asia i weiiiiiccoott.omm com
Gaoling School of Artificial Intelligence Renmin University of China
zeyuzhangoruc. edw.ruuuueddd...... cn
Xu Chen
Xing Xie
Gaoling School of Artificial Intelligence Renmin University of China xu. cwwn@ruc......co... edu. cn
Microsoft Research Asia inngxomicrosof..ommmmm
Abstract
This paper studies the problem of active causal discovery when the experiments can be done based on multi-fidelity oracles, where higher fidelity experiments are more precise and expensive, while the lower ones are cheaper but less accurate. In this paper, we formally define the task of multi-fidelity active causal discovery, and design a probabilistic model for solving this problem. specific, we first introduce nutuul-informationn based acquisition function to determine which variable should be intervened which fidelity, and then cascading model is proposed to capture the correlations between different fidelity oracles. Beyond the above basic framework, we also extend to the batch intervention scenario. We find that the theoretical foundations behind the widely used and efficient greedy method do not hold in our problem. To solve this problem, we introduce a new concept called e-submodular, and design constraint based fidelity model to theoretically validate the greedy method. conduct extensive experiments to demonstrate the effectiveness of our model.
1
Introduction
Causal discovery aims to learn the causal structure set of variables. which is fundamental for many real-world applications, including health caring [[], education [2] drug discovery [3] and protein synthesis [4. In general, causal structure learning is an NP-hard problem [5) and purely based on the observational datasets, one cannot identify the unique causal structure, where the best
result discovering Markov equivalence class
To more accurately identify the unique causal structure, promising direction is active causal discovery (ACD), where the model is allowed actively intervene the causal structure query key information for orienting the causal relations between different variables. For example, to study the causal relations between the drugs and diseases, one can conduct clinical tests via selectively administering the medicines to the patients. The key of active causal discovery is how to design effective experiments when the total cost (e.g., the number experiments) is limited. To achieve this goal, recent years have witnessed many promising models. For example, Agrawal et al. [7] proposes intervene on the variables which can orient as many as possible undirected edges. Tigas et al.
"Co-first authors.
I "Coresponning author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

designs a mutual information based method to determine the variables and values to be intervened, and study both single and batch intervention scenarios.
While the above models have achieved remarkable successes, they only allow to query a single oracle (e.g., the real causal structure) for the experiments However, in many real-world applications, the experiments can be done via different methods. For example, to investigate the drug-disease causal relations, in addition the clinical tests, one can also build simulators to obtain the medicine effects on the patients [9]. Usually, each experimental method corresponds to a unique oracle, and different oracles have various fidelites. Higher-fidelity experiments are more accurate but expensive, for example, administering drugs to the real patients. floer-fidelit, experiments are cheaper but inacourate, for example, using patient simulators. These different fidelity oracles may offer better cost-benett trade-offs, which, however, cannot be handled by existing active causal discovery models.
To bridge the above gap, in this paper, formally define the task of active causal discovery with multi-fidelity oracles, where the model has to actively select which variables and values to intervene which fidelities. This task is non-trivial due to the following reasons: to begin with, because of the introduction of multi-fidelity oracles, the model has to strategically choose the lower-cost and informative enough experiments to uncover the real causal structure, which needs our special designs. Then, given the experiment results with different fidelities, how to infer the real causal structure is also not easy, since the experiment results can be not produced from the oracle corresponding to the real causal structure. In addition, in practice, an efficient experiment should allow simultaneously intervening multiple variables [0]. However, how to extend our model the batch intervention scenario is still not clear.
To overcome the above challenges, we design a Bayesian active causal discovery model, which is composed of two components. The first one is a mutual information (MI) based acquisition function. It aims select the interventional variables, values and fidelities which are more informative for the real causal structure, The second one is a cascading fidelity model. In specific, we first regard the highest fidelity oracle as the real causal structure, and then a cascading model is built to correlate different fidelity oracles, so that the experiment results at one fidelity can be leveraged to infer the oracle another fidelity. To achieve more efficient experiments, we also extend our model to the batch intervention scenario. Previously, the greedy method is demonstrated to be an efficient and effective strategy for batch intervention { However, we found that, by allowing multi-fidell -idelityy oracles, the theoretical foundations behind the greedy method do not hold. For alleviating this problem, we introduce a new concept called c-submodular, and design a constraint-ban fidelity model to theoretically validate the greedy method.
The main contributions of this paper are summarized as follows: (1) we formally define the task of active causal discovery with multi-fidelity oracles, which, to our knowledge, is the first time in the field of causal discovery. (2) To solve the above task, we propose a Bayesian framework, which is composed of mutual information based acquisition function and a cascading fidelity model. (3) To extend our framework to the batch intervention scenario, we introduce a constraint- fidelity model, which provides theoretical guarantees for the efficient greedy method. (4) We conduct extensive experiments to demonstrate the effectiveness of our model.
2 Preliminaries
2.1 Structure Causal Model
Structure causal model (SCM) an effective language for describing and learning the causal relations between different random variables [T] Usually, SCM is composed of a causal graph and set of structure equation models (SEM).
For the causal graph, we denote it by G = (V, E) where V is the node set, and E is the adjacency matrix. Each node in V corresponds to a variable. Suppose there are variables in our studied problem, then we use Xv X2....... Xal to denote the variable set. The adjacency matrix E E {0, 1] 1]axd describes the causal relations between different variables. E.j = 1 means that X; is
'In the following, we may interchangeably use "oracle", "causal structure" and "structure causal model" to represent the underlying model for generating the results of the experiments.

---

parent of X,, and there exists an edge from X; to Xj, while Egg 0 indicates that there is no edge between X; and X,.
For the structure equation models, we denote them by g = {gr, 92, 9a). where each Ii quan- titatively describes the relation between X; and its parents. Formally, we implement F with the commonly used additive noise models (ANM) [2], that is:
X; g: + ei, 6 ~00,00,
where 7/ is the parameter set of gr. and the noise term follows the Gaussian distribution with of as the variance. We denote the complete parameter set g as {r, o., where y = {ri, 72,.... 7d and a = 02,,02....... Ja). Noted that, given the above equation, we can easily derive the distribution of Xv, that is, p(Xv) = I (pa( 7)).
Based on the above formulation, given an observational dataset D = fa* ~ p(Xv), causal discovery aims learn the adjacency matrix E, or more generally, simultaneously identify E and the SEM parameter 0. In this paper, we focus on the general case, and denote & (0, E).
2.2 Active Causal Discovery
Previous work has demonstrated that, purely based on the observational dataset, the real causal graph only be identifiable to its Markov equivalence class. Active causal discovery holds the promise of identifying more accurate causal graph via designing interventional experiments.
Formally, an interventional experiment is represented by {(i,)) e)). which means cutting all the causal relations pointing to X,, and fixing the value of X, as v. causal learning, the experiment can also be represented by do(X, v). Obviously, the distribution of Xy is changed after the experiment e, and we denote the experiment-induce cannot access the implementation of p. but only observe the experiment result sampled from p(Xv [X, p(Xy|do(X, , = v)). The key of active causal discovery is to design a series of experiments within
distribution by (Xv]do(XX = v)). In practice,
limited budgets, such that the results can be better leveraged to identify Ø
2.3 Multi-Fidelity Active Causal Discovery
Existing ACD models mostly obtain the experiment results via interacting with the real SCM. However, practice, the experiments can be done in different ways (e.g., real clinical tests or patient simulators). Each type of experiment corresponds to an underlying oracle, which produces the results of the experiments. Different oracles may provide better cost-henefit trade-offs for the experiment designs, which are failed to be considered by the previous work.
Formally, suppose have M oracles, and the parameters of the ith oracle is denoted by Φ.. Let the experiment cost of the ith oracle be X; and without loss of generality, we assume A < A2 < S. Av.. Intuitively, if an oracle is more accurate (i.e., has higher fidelity), then it should be more expensive We regard the real SCM as the most accurate oracle, thus we set ΦM = sb We denote all the oracle parameters and costs as (Φ1, $2.... ..., Φas) } and A (A1, A2 ..., da respectively.
Due to the introduction of multi fidelity oracles, the experiment in traditional active causal discovery extended be triplet
{(j, m)), where in addition the intervention pair u), the fidelity m should also be considered in the experiment design. We define the dataset for model training as D {ee, æ; ) )= where e, {(j v), m) indicates the distribution for generating æ;. In general, {(j. v), m) means that I, is generated from Pm (X |do (X, = v)). which is induced by Φm and the intervention (j, v). If (j, v) 0. then 2, an observational data, which is sampled from &n without any intervention. Finally, we define the task of
active causal discovery as follows:
Definition 1 AMulti-Fidelity Active Causal Discovery (MFACD)). Given M oracles with different costs A, we need to design model f. which can strategically determine the intervention pair (j. v) and fidelity m to achieve better cost-benefit trade-off in terms of identifying the real SCM dM--
'Noted that if some oracle costs more than another one with higher fidelity, then this oracle is useless, since may always choose query the higher tidelity oracle.
3

---

3 The Licence Model
For solving the task of MFACD, design Bayesian framework called Licence (Multi- -idilit active learning for causal discovery), which is composed of two components. The first one is an acquisition function based on mutual information, which is responsible for designing the experiments. The second one is a cascaded fidelity model, which is designed to capture the correlation between different Φ s. The experiment results obtained from the first component are leveraged to update the fidelity model, and Φv in the fidelity model is the finally predicted result. At last, we introduce how extend our model to the batch intervention scenario.
3.1 MI-based Acquisition Function
Intuitively, a better experiment should leverage little cost to reveal as much possible information about the real SCM ΦM. Thus, we design the following acquisition function:
f(i, v, m) T(x; Φar D)
Am
where e = [(i, v), m) is the experiment to be designed. is the dataset already collected, which will be enlarged after each experiment. / [xx; ΦM e, D) is the mutual information, which indicates that if we conduct experiment e, then how much information the experiment result may share with the target parameter Φas. Obviously, we should select which can lead larger (xx; Øarle, D). Am is the cost of e. By dividing [x; ΦMrle le. D) with Am, we trade-off the experiment informativeness and cost. To determine (j, v, m), we derive an estimator for fjj v, m) following the idea of Bayesian Active Learning by Disagreement (BALD) [33,, that is:
(.j.,m) [1 D) H( e,
Am (p(alee Φm)l] Ept D)) [ppaeee log p(zle, oM)))
(2)
-Ep( llog Ep( (p(xzl,
Am
where p(zle, D) and p( e, D) correspond to the distributions of x and Φm. after observing D under the intervention (j, v). p( D) is the posterior of dM after observing D. p(ælle, @m) the
probability of æ based on @.m intervened by (j, v). We approximate the expectation operator based Monte Carlo sampling. The detailed derivation process can be seen in the appendix.
Obviously, determining the best (j, m) equals to solving the following problem:
{i", "", m²) arg max u.). (j.m..)
In our task, the intervention value e is continuous. Following the previous work, we firstly learn the optimal for each (j. m) pair based on Bayesian optimization (BO). Then, compare all the results, and select the solution which can lead to the largest f(j. m.) We present the detailed Bayesian optimization process the appendix. It should be noted that one can also leverage more advanced BO methods for jointly learning (j, e, m) [4]. However, we do not find significant performance improvements by these models.
3.2 Cascaded Fidelity Model
Intuitively, the experiment results from different oracles may share common information. The samples at one fidelity may help to infer the oracles at other fidelities. To capture the correlations between different oracles, we build a cascaded probabilistic model (see Figure [ where the oracles with different fidelities are successively connected, and the observed samples only determined by their corresponding oracles.
Formally, to achieve more robust optimization, we first regard the discrete adjacency matrix E as the samples from Bernoulli distribution. In specific, we let E,y Bernoulli (STT..))): where S. T kxd xd (K & d) are two continuous matrices. S., and T. are the ith and jth columns of S and T, respectively. By replacing E with S and T, we revise the parameter set & as (0, S, T).
For each fidelity m, assign a prior distribution of om given Φm-1 as follows:
p(()) (.).). I), p( p(0m)mm-1) x
(4)
3..((..TTT.)) 3.
m 2 2,

---

Low-fidelity
High-fidelity
Intervention variable and value
(j.v
m
Fidelity-level
input
Oracle +
Oracle Φ
Oracle Φb
Oracle
Oracle
output
output
output
output
output
X
Figure I: The cascaded probabilistic model is shown above. Different fidelity oracles are successively connected, and the observed samples are only determined by their corresponding oracles.
where we add subscript m to indicate different fidelities. f(Sm. Tm) = EpjEs.... T.. (trace (ct) d) X2 [[|]]] is regularizer encourage E be sparse and directed acyclic graph [15]. X1 A2. a, b and o² are syper-parametenn.
Since the real SCM Φas, we need to infer the posterior p(ΦM) D). where D the initially observational dataset or experiment results. Directly computing p( D) is not easy, since the dataset D may contain samples from different fidelity oracles. obtain the joint distribution p( 4 D)) where & { 02,..,,... ΦM)) } is the collection of all the oracle parameters. Then we derive p(@ D) by marginalizing out {Φ) 02 ..., DM-1).
solve this problem, we first
To efficiently compute and sample from p( D), we introduce a variational approximator (()).
which is specified follows:
( ~ (0,,1),
am (#m xm ~ -N(mmmm-1 dm o 1n m N 2,
(5)
(( = (()1) II @m)m))m-1),
m=2
where eim {cmm dm J is the set of learnable parameters. and we denote Y = fm JMM m2 According the theory of variational inference [16], we maximize the following evidence lower bound (ELBO) [16] to learn Y
ELBO = Ex [log p(D)( (L(g()) (+))
(6)
Eg Φ) P( (b) log g( ()) + log ())
where the likelihood p(D(@) can be casily obtained based on equation More detailed derivation
on the ELBO can be seen in the appendix. Once have learned V the posterior p( D) is approximated by q( ()), and further, we have the following theory:
Theorem 1. pf $ D) zs ()), for any variable sets A, C T, P(ABB,)) = (). As special case p( Φ D) = g(
The proof of this theory is immediate, since
p(A|B, D) P(A, B(D) S A-B D) -AAR
A-B S 4 g( A-B
P(BID)
f p(
S 1 d
B)
((B) ((AlB)
Let A ΦN and 0. we have p( D) g(dM). Based on this theory, leverage & to replace in for easy sampling.
Remark. According to the specification of qu.,, ((ml/mm1)) 1). can easily demonstrate that q( () follows Gaussian distribution. Such property enables us use the reparameterization trick [17 to relate & with V. Different from traditional variational models, in our objective, the adjacency matrix
5

---

E f(Sm. Tm) is sampled from a discrete distribution, which cuts down the back-propagee propagation signal. To solve this problem, we leverage gumbel-softmatt Since & can be further represented by V,, all the variables in (6) can be reparameterized by V which enables us to optimize it an end to end manner. We present the detailed derivation process and the complete learning algorithm in the appendix.
further associate E with (S... Tm) E Φ.
3.3 Extension to Batch Intervention
In practice, simultaneously intervening multiple variables can be more efficient due to the lower frequency on interacting with the oracles. However, under the setting of batch intervention. the candidate intervention space exponentially increases with respect the number of targets. Suppose we need to select e out of d variables for intervention, then the size of the candidate space is d' Previous work found that the greedy strategy is a both efficient and effective method for the batch intervention scenario []. From the efficiency perspective, the greedy method only need to search in ed-sized candidate space. From the effectiveness perspective, people demonstrate that the mutual information obtained by the greedy strategy is not worse than that of the optimal solution multiplied by (1 - 2) [8] In the following, we introduce how to extend our model to the batch intervention scenario, and leverage the greedy strategy to solve our task.
Objective for batch intervention in MFACD. We use the following objective for batch intervention
scenario:
arg max ((2;)1=1 Φ :e) (
EASC,
where e; (Li, 0z), m; } is an experiment, 2; the observed sample from the experiment e;, that is, 24 ~ Pm: (Xv id((X (X,; = ez))- X is the cost of experiment e;. This objective aims to design a series of experiments with budget C, which can reveal the information about ΦM much as possible. The number of intervention targets n. is not a fixed value, which is constrained by the total budget.
The greedy method for MFACD. The greedy method designs each experiment independently. and each step, selects the experiment which can maximize the average information gain. Following [9 the kith experiment is determined based the following objective:
T( M=l U Jæy @M U ex, D) T((x) owll)) MMl D))
arg max 6%
Am
(8)
--1
s.t.
Σ m=11
A
where {e: is the previously designed experiments, and is fixed when learning ek-
What's wrong with the greedy method. The theoretical foundations of the greedy method is demonstrated by the previous work [19] as follows:
Theorem 2. [(x; ΦM D) is (1) submodular and (2) non-decreasing. then
(9)
where {e pm is the solution obtained the greedy method, and {e; He is the optimal solution.
However, due to the introduction of multi-fidelity, in our model, [(x; &M le, D) is actually not submodular. More specifically, in the proof of submodular, two samples x, and H; has to be independent given & (e.g., see B.4 in (8). In the single-fidelity setting, this requirement naturally holds, since Φ is exactly the parameter used to sample x, and z;- However, when we introduce multi-fidelity æ, and æ; may not independent given ΦM, since they are only directly influenced by their own oracles (see Figure T].
An improved greedy method tailored to MFACD. To alleviate the above problem, in this section we design an improved greedy method tailored to our task. In specific, we first define several new concepts, and then build theories based on these concepts inspire our model designs.

---

Definition 2 (c-independent) For random variables A, B and C, if their mutual information satisfy [AA; B(C') S e, then we say A and B are e-independent given C.
Definition 3 (e-submodular), Suppose f(-) is set function on Ω. If for any A S B and B. f(Au f(A) 2 f(B (z]) f(B) e, then we say f is e-submodular on Q
Based on the above two definitions, we have:
Theorem 3. For any two experiments and if the corresponding samples and xt are e-independent given ΦM, {ex, ex} and D, then I(: D) e-submodular.
X Theorem 4. If 1(: D) is e-submodular on X and non-decrease, for any i, A S B2, then
I((= 11=13 ΦM ( (c]) D) (1 -1-e-- + " (M D) B, where B = = Ei=1 Ban )
Following this theory, we improve objective (6) to a constaint-based ELBO follows to capture the degree of independence between different experiment results:
max E Ee [logp(D) log q( -loog (.)
s.t. (e.,er)
Σ
I 1/ fe,, D) c.
(10)
The proofs of the above theories are presented the appendix, and similar to objective 22 we use Monte Carlo method to approximate the mutual information in objective (TO).
4 Related Works
Bayesian Active Causal Discovery. Causal discovery 20 refers to recovering causality in set of variables, especially trying to find directed acyelic graph (DAG) that represent the relationship between variables in a system. Active causal discovery was first proposed in [23 24]
with the assumption that the data is discrete-valued. In active causal discovery, the experimenter attempts to intervene on the variables in the system at each step, utilizes the interventional data to
recover causal relation, and finally identifies the entire causal structure with minimal cost [25] Many methods have been discussed in different settings over the past decades, including continuous linear Bayesian networks 26 227, non-linear causal models (28]. and large-scale causal models 8
Multi-fidelity Settings. The fidelity commonly refers to how accurate the model or environment can be when providing information. Higher-fidelity models ones are less accurate but cheaper. order to combine the strength of each model, multi- fullii-iieeltyy models are proposed achieve accurate model with lower costs. Multi-fidelity models can be divided into two main categories [29]: Multi-fidelity Surrogate Models (MFSM) (30-32] and
accuracy but cost much, while lower
Multi-fidelity Hierarchical Models (MFHM) 83 33. In addition optimization, multi-fidelity
models can also be used for uncertainty propagation (36) and statistical inference [37]. Recent works
also start study the multi-fidelity settings when conducting Bayesian experiments [38 and adopt deep learning frameworks to solve corresponding problems.
Our paper makes first step towards multi-fidelity active causal discovery, and solve the non-trivial challenges when combining the above two fields, which, to the best of our knowledge, the first time in the causal inference domain.
Experiments
In this section, we conduct experiments to demonstrate the effectiveness of our model, where we focus on the following problems: (1) whether our model can achieve better performance than the previous ACD methods? (2) Whether the constraint in objective (10) in necessary? (3) How the DAG regularization coefficient influence the model performance? In the following, we first introduce the
experiment setup, and then present and analyze the results.
7

---

ER Graph(single)
SF Graph(single)
Graph(batch)
SF Graph(batch)
30
30
25
SHD
20
20
10
20
10
50
30
40
20
budget
budget
budget
budget
ER Graph(single)
SF Graph(singie)
Graph(batch)
SF Graph(batch)
0.4
0.4
AUPRC
AUPRC
AURRC
AUPRC
0.3
0.2
0.2
0.2
0.2
10
20
30
50
10
20
40
50
30
40
50
10
20
30
50
total budget
total budget
total budget
total budget
ER Graph(single)
SF Graph(single)
ER Graph(batch)
SF Graph(batch)
0.06
0.05
0.05
0.035
RM5B
RMSE
RM55
0.030
RMSE
0.00
0.04
0.025
0.03
0.03
0.020
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
total budget
total budget
total budget
total budget
AIT-REAL
AIT-RANDOM
CBED-REAL
CBED-RANDOM
Licence
Figure 2: Results of the overall performance on different datasets and budgets. Lower SHD, RMSE or larger AUPRC indicate better performances. We conduct each experiment for ten times, and report the average performances and error bars.
5.1 Experimental Setup
We experiment with three commonly used causal discovery datasets, including Erdös-Rényí graph (ER) [41], Scale-Free graph (SF) [42 and DREAM [43]. To demonstrate the effectiveness of our model, we compare with AIT 44 and CBED 8 which are the recent
models in this field. Since they cannot select different fidelities, we design two variants for each of the baseline, that is, X-REAL and X-RANDOM which means that the model always interacts with the ground truth oracle ΦM randomly select the oracles. Here "X" is AIT or CBED. For the evaluation metrics, we use SHD [45), AUPRC [46) and RMSE to evaluate different models. The first two metrics aim to evaluate the accuracy of the learned topological structure, and the last one measures the performance of functional relations. For single intervention, we first generate several observational samples to initialize the model. Then, indicate total intervention budget, and let the model interact with the causal graph with different fidelities until the budget runs out. For each interaction, the model will provide intervention, and correspondingly obtain sample from the oracles, which is used to update the model. Finally, the model outputs the estimated causal graph, which is leveraged to evaluate the performance. For batch intervention, indicate the total budget for each intervention step, and the model determines n interventions simultaneously. which are delivered to the oracles to obtain the samples. We present more detailed settings in the appendix.
5.2 Overall Performance
In this experiment, we evaluate the models under different total budgets, and we present the results on ER and SF datasets with 10 graph nodes. The experiments on DREAM and more nodes are presented in the appendix. From the results shown in Figure 2 can see: the total budget becomes larger, the performances of all the models tend to increase on both datasets. This is not surprising, since more experiment budgets enable us to conduct more interventions or query more accurate oracles, which can reveal more information about the ground truth and facilitate more accurate causal discovery. In most cases, our model can perform better than the baselines across different datasets, evaluation metrics and intervention budgets. These results demonstrate the effectiveness of our model. In specific, on the metrics of SHD, AUPRC and RMSE, our model can on average improve performance of the best baseline by about 27. 27.44, 82.35% and 22.74% on ER, and 17.69%, 60.27% and 21.62%

---

(a) Batch Intervention with Budget
(b) Influence
Model
SHD -
AUPRC [
RMSE (%)
0.35
ATT-REAL
20.10 + 2.26
17.74 + 0.32
3.11 + 0.00
AIT-RANDOM
19.92 +
17.57
+ 0.43
3,53 - 0.01
0.01
AUPRC
0.30
CBED-REAL
20.12 2.17
20.25 +: 0.19
3.51 0.00
CBED-RANDOM
19.34 + 1.72
24.32 + 0.47
3.74+0.00
0.25
Licence (w/o reg)
17.10 3.71
32.55
+ 1.81
3.12
Licence
16.75 + 5.68
37.94
+ 1.90
2.66 + 0.01
0.2
0.4
0.6
0.8
1.0
values of
Figure 3: (a) The results of experiments on ER graph with 10 graph nodes under the batch intervention scenario. The average performance and error bars are provided. (b) The results of Licence model with different DAG regulation coefficient ("s." The experiment is conducted based on ER graph with 10 graph nodes.
on SF, respectively. If we look more carefully, we find that, for both AIT and CBED, randomly querying the oracles sometimes perform better than always interacting with the ground truth oracles. This result suggests that lower fidelity oracles can be helpful to trade-off the performance and cost. However, the random method is still suboptimal, and designing more principled and tailored strategies to select the fidelities is necessary, which is evidenced by the lowered performance of "X-RANDOM" than our model.
5.3 Necessity of the Mutual Information Constraint in Objective
10
In our model, the mutual information constraint in objective (10) aims to make the greedy method validate. this section, we study whether is necessary by experiments, To achieve this goal, introduce variant of model Licence (w/o reg), where remove the mutual information constraint. We evaluate the models based on the dataset of ER with 10 graph nodes, and the total budget is set as 30. The results are presented in Figure a). We can see, in some cases, "X-RANDOM, performs better than "X-REAL". which is consistent with the above experiments, and demonstrates that always querying the ground truth oracle may not lead better performance under limited budget. By comparing our model with the variant Licence (w/o reg), we find that our model can consistently achieve better performances performance of Licence (w/o reg) by about 2.05%, 16.57% and 14.92% on SHD, AUPRC and RMSE respectively. These results demonstrate that the mutual information constraint is necessary in our model, which empirically verifies the theories proposed in section 3.3
all the evaluation metrics. In specific, Licence can improve the
5.4 Influence of the DAG regularization coefficient
In this section, we analysis the influence of the DAG regulation coefficient 33 in equation (4). The results are reported based on AUPRC. The coefficient, 3 reflects the importance of DAG regulation when updating the model. As 3 increases, the DAGness is more emphasized for the causal graph. In this subsection, we conduct experiments for various B, ranging from 0.0 1.0, and the results are shown in Figure B.b). We can see as 3 increases, the performance of AUPRC improves as well, and peaks at B 0.8. That is probably because lower | will decrease the acyelic property of causal graphs, which is incompatible with the prior knowledge of true causal graphs. However, higher coefficient may also impact the optimization process, which leads to sub-optimal results. We think trade-off between mild constraint for easy optimization and solid constraint for DAG property is supposed to take into consider for different tasks and settings.
Conclusion
This paper formally defines the task of active causal discovery with multi-fidelity oracles, which, to our knowledge, the first time in the causal discovery domain. solve this task, we propose Bayesian framework, which is composed of mutual information based acquisition function and a cascading fidelity model. We also extend our framework to the batch intervention scenario, and propose constraint- based fidelity model to validate the efficient greedy method.

---

This paper actually makes an initial step toward considering different oracles in active causal discovery. There is much room left for improvement. intervention strategies, which can bypass the greedy method and does not need to introduce the mutual information constraint the fidelity model. In addition, since the experiments in active causal discovery are conducted sequentially, and the former experiment results may influence the latter ones, it is interesting to consider the experiment designs as Markov decision process, and leverage reinforcement learning to optimize the total information gains of all the experiments in more principled manner.
begin with, one can design more advanced batch
ACKNOWLEDGMENTS
This work is supported part by National Key R&D Program of China (2022ZD0120103) National Natural Science Foundation of China (No. 62102420), Beijing Outstanding Young Scientist Program NO. BIJWZYJHO1 2019000009900020099 19000200098 Intelligent Social Governance Platform, Major Innovation & Planning Interdisciplinary Platform for the "Double-First Class" Initiative, Renmin University of China, Public Computing Cloud, Renmin University of China, fund for building world-class universities (disciplines) of Renmin University of China, Intelligent Social Governance Platform.
References
[1] Jeffrey S Levin. Religion and health: Is there an association, it valid, and is it causal? Social Science & Medicine, (8(11):1475-11 1471-1482222 1482, 1994.
[2] Eric Eide and Mark H Showalter. Methods matter: Improving causal inference in educational and social science research: A review article. Economics of Education Review, 31(5):744-744 2012.
[3] Bengt Muthén and Hendricks C Brown. Estimating drug effects in the presence of placebo response: causal inference using growth mixture modeling. Statistics in medicine, 28(27): 3363 3085, 2009.
[4] Karen Sachs, Omar Perez, Dana Pe'er, Douglas A Lauffenburger, and Garry P Nolan. Causal protein-signaling networks derived from multiparameter single-cell data. Science, 308(5721): 523- 529, 2005.
[5] David Maxwell Chickering. Learning bayesian networks is np-complete. Learning from data: Artificial intelligence and statistics pages 121-130, 1996.
[6] Thomas S Verma and Judea Pearl. Equivalence and synthesis of causal models. In Probabilistic and Causal Inference: The Works of Judea Pearl, pages 221-236. 2022.
[7] Raj Agrawal, Chandler Squires, Karren Yang, Karthikeyan Shanmugam, and Caroline Uhler. Abed- Abed-strategs strategy: Budgeted experimental design for targeted causal structure discovery. In The 22nd International Conference on Artificial Intelligence and Slatistics, pages 3400-3409, PMLR, 2019.
[8] Panagiotis Tigas, Yashas Annadani, Andrew Jesson, Bernhard Schölkopf, Yarin Gal, and Stefan Bauer. Interventions, where and how? experimental design for causal models at scale. In Advances in Neural Information Processing Systems.
[9] Azam Peyvandipour, Nafiseh Saberian, Adib Shafi, Michele Donato, and Sorin Draghici. A novel computational approach for drug repurposing using systems biology. Bioinformatics, 34 (16):2817-2825. 2018.
[10] Yashas Annadani, Panagiotis Tigas, Desi R Ivanova, Andrew Jesson, Yarin Gal, Adam Foster, and Stefan Bauer. Differentiable multi-target causal bayesian experimental design. arXiv preprint 21Xiv:2302.10607, 2023.
[11] Judea Pearl and Dana Mackenzie. The book of why: the new science of cause and effect. Basic books, 2018
10

---

[12] Patrik Hoyer, Dominik Janzing, Joris M Mooij. Jonas Peters, and Bernhard Schölkopf. Nonlinear causal discovery with additive noise models, Advances in neural information processing systems, 21, 2008.
[13] Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprini arXiv: /11.5744, 2011.
[14] Samuel Daulton, Xingchen Wan, David Eriksson, Maximilian Balandat, Michael A Oshorne, and Eytan Bakshy. Bayesian optimization over discrete and mixed spaces via probabilistic reparameterization. arXiv preprint arXiv:2210.10199 2022.
[15] Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. Dags with no tears: Continuous optimization for structure learning. Advances in neural information processing systems, 31, 2018.
[16] David M Blei, Alp Kucukelbir, and Jon McAuliffe. Variational inference: review for statisticians. Journal of the American statistical Association, 112(518):55--877 -877 2017.
[17] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint 1rXiv:1312.61 14, 2013.
[18) Andreas Krause and Carlos E Guestrin. Near-optimal nonmyopic value of information in graphical models. arXiv preprint arXiv: 120. 2012.
[19] Shibo Li, Jeff M Phillips, Xin Yu, Robert Kirby, and Shandian Zhe. Batch multi-fidelity active learning with budget constraints. Advances in Neural Information Processing Systems, 35: 995-1007, 2022.
[20] Christina Heeinee-emml Marloes H Maathuis, and Nicolai Meinshausen. Causal structure learning. Annual Review of Statistics and Its Application, 5:371-391, 2018.
[21] Clark Glymour, Kun Zhang, and Peter Spirtes. Review of causal discovery methods based on graphical models. Frontiers in genetics, 10:524, 2019.
[22] Matthew Vowels, Necati Cihan Camgoz, and Richard Bowden. D'ya like dags? a survey on structure learning and causal discovery. ACM Computing Surveys, 55(4):1-36, 2022.
[23] Kevin P Murphy. Active learning of causal bayes net structure. Technical report, technical report, UC Berkeley, 2001.
[24] Simon Tong and Daphne Koller. Active learning for structure in bayesian networks. In International joint conference on artificial intelligence, volume 17, pages 863-869, Citeseer 2001.
[25] Yang-Bo He and Zhi Geng. Active learning of causal networks with intervention experiments and optimal designs. Journal of Machine Learning Research, 9NNov) 2547, 2008
[26] Hyunghoon Cho, Bonnie Berger, and Jian Peng. Reconstructing causal biological networks through active learning. PloS one, 11(3)=01506611 2016.
[27] Robert Osazuwa Ness, Karen Sachs, Parag Mallick, and Olga Vitek. A bayesian active learning experimental design for inferring signaling networks. In Research in Computational Molecular Biology: 2 Annual International Conference, RECOMB 2017, Hong Kong, China, May 3-7, 2017, Proceedings 21, pages 134-156. Springer, 2017.
[28] Julius von Kügelgen, Paul K Rubenstein, Bernhard Schölkopf, and Adrian Weller. Optimal experimental design via bayesian optimization: active causal structure learning for gaussian process networks. arXiv preprint arXiv: 19X00:991000362. 2019.
[29] M Giselle Fernández-Godinn, Chanyoung Park, Nam-Ho Kim, and Raphael Haftka, Review of multi-fidelit models. arXiv preprint 3XXv:1669.00796 2016.
11

---

[30] Alexander IJ Forrester. Neil W Bressloff, and Andy Keane. Optimization using surrogate models and partially converged computational fluid dynamics simulations. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 462(2071):2177-2224 2006.
[31] TD Robinson, Michael S Eldred, Karen Willcox, and Robert Haimes. Surrogate-based optimization using multifidelity models with variable parameterization and corrected space mapping. AIAA journal, 46(1 (6(11))2814-288) 14-2822, 2008.
[32] Christopher C Fischer, Ramana V Grandhi, and Philip S Beran. Bayesian low-fidelity correction approach to multi-fidelity aerospace design. In 58th A/AA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, page 0133, 2017.
[33] SA Burton and P Hajela. A cettbble-cmplexiin approach to second-order reliability-based optimization. Structural and Multidisscilinnar Optimization, 25( 2003.
[34] Akil Narayan, Claude Gittelson, and Dongbin Xiu. A stochastic collocation algorithm with multifidelity models. SIAM Journal on Scientific Computing, 36(2):4455-A52,, 2014.
[35] Benjamin Peherstorfer, Tiangang Cui, Youssef Marzouk, and Karen Willcox. Multifidelity importance sampling. Computer Methods in Applied Mechanics and Engineering, 300:490-509, 2016.
[36] Leo Wai-Tsun Ng and Michael Eldred. Multifidelity uncertainty quantification using non- intrusive polynomial chaos and stochastic collocation. In 53rd aica//ssme/accelahsssse struc- tures, structural dynamics and materials conference 20th aiaa/asme/ahs adaptive structures conference 14th aiaa, page 1852, 2012.
[37] Tiangang Cui, Youssef M Marzouk, and Karen E Willcox. Data-driven model reduction for the bayesian solution of inverse problems. International Journal for Numerical Methods in Engineering, 10265))966-990, 2015.
[38] Shibo Li, Wei Xing, Robert Kirby, and Shandian Zhe. Multi-fidelity bayesian optimization via deep neural networks. Advances in Neural Information Processing Systems, 33:8521-8531, 2020.
[39] Shibo Li, Robert Kirby, and Shandian Zhe. Batch multi-fidelity bayesian optimization with deep nuto-regressive networks. Advances in Neural Information Processing Systems, 34:25463- 25475, 2021.
[40] Shibo Li, Jeff M Phillips, Xin Yu, Robert Kirby, and Shandian Zhe. Batch multi multi-fidelity active learning with budget constraints. Advances in Neural Information Processing Systems, 35: 995-1007, 2022.
[41] Paul Erdös, Alfréd Rényi, al. On the evolution of random graphs. Publ, Math. Inst. Hung. Acad. Sci, 5(1):17-60, 1960.
[42] Lun Li, David Alderson, John C Doyle, and Walter Willinger. Towards theory of scale-free graphs: Definition, properties, and implications. Internet Mathematics, 2(4):431-523, 2005,
[43] Thomas Schaffter, Daniel Marbach, and Dario Florcano. Genenetweaver: in silico benchmark generation and performance profiling of network inference methods. Bioinformatics, 27(16): 2263-22700 201
[44] Yashas Annadani, Jonas Rothfuss, Alexandre Lacoste, Nino Scherrer, Anirudh Goyal, Yoshua Bengio, and Stefan Bauer. Variational causal networks: Approximate bayesian inference over causal structures. arXiv preprint arXiv:2
2021.
[45] Ioannis Tsaaaardioos, Laura E Brown, and Constantin Aliferis. The max-min hill-climbing bayesian network structure learning algorithm. Machine learning, 65:31-78, 2006.
[46] Jesse Davis and Mark Goadrich. The relationship between precision-recall and roc curves. In Proceedings of the 23rd international conference on Machine learning, pages 233-240, 2006.
12

---

[47] Harold J Kushner. A new method of locating the maximum point of an arbitrary multipeak curve the presence of noise. 1964.
[48] Carl Edward Rasmussen. Gaussian processes in machine learning. In Advanced Lectures on Machine Leurning: ML Summer Schools 2003, Canberra, Australia, February 2-14, 2003, Tübingen, Germany, August 4-16, 2003, Revised Lectures, pages 63-71. Springer, 2004.
[49] Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. Gaussian pro- cess optimization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:0912.3995, 2009.
[50] Reuven Y Rubinstein and Dirk P Kroese. Simulation and the Monte Carlo method. John Wiley & Sons, 2016.
[51] Chris Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv: 161116111.. 2016.
[52] Dimitri P Bertsekas. Constrained optimization and Lagrange multiplier methods. Academic press, 2014.
[53] Lars Lorch, Jonas Rothfuss, Bernhard Schölkopf, and Andreas Krause. Dibs: Differentiable bayesian structure learning. Advances in Neural Information Processing Systems, 34:241] 1- 24123, 2021.
13

---

Appendix
Monte Carlo Approximation for f(j, v. m)
15
A.1
Derivation Process for [(j, v. m)
15
A.2
Sampling from pl p(@ml
16
A.3
Sampling from p( mm Ø
19
4
Calculation of p e.0m)
20
Bayesian Optimization for Determining (j" o², m
21
Detailed Training Process of ELBO
22
Derivation Process of ELBO
22
Estimation of ELBO
22
Gaussian Reparameterization Trick
24
Gumbel-softmax Reparameterization Trick
24
Optimization of ELBO
25
Training Process of Constraint based ELBO
25
Proof of Theory 3
26
Proof of Theory
26
Algorithm
28
More Experiments
28
Experimental Settings
28
H.I.J Datasets
28
Baselines
28
H.1.3 Metrics
29
H.2
Simulation of Oracles with Different Fidelities
30
H.3
Details of Configurations and Computation
30
H.4
Experiments on DREAM Dataset
30
H.5
Experiments on More Nodes
30
H.6
Supplementary Experiments MAE Metric
31
Supplementary Experiments on Different Oracle Settings
................................
8
Supplementary Experiments on Regularization Coefficient X
32
H.9
Supplementary Experiments Ablation Studies
32
Potentially Negative Social Impact
32
Limitations
34
14

---

A Monte Carlo Approximation for f(j, v, m)
A.1 Derivation Process for f(i, k.))
Considering that the mutual information is not directly tractable, we approximate f(i, v, m) by:
C₁
f(j. v. m)
log
p( (( ( [e)
Am K1 L1
===1
La
(llaa) (((olaal (Aaa) .e).
log
Am K2 L2 C2
where (0, m is the experiment to be designed, & ((1) (k)) pl D). xm (k141) (H2dd2)
pm
p(alom (11) e), / (ka) p() Φm (ka,la) p p(@mlo (ka), D) and am (2.)) ~ p( e).
We present the detailed approximation process follows:
f(j m)
Am
[H(ee, D) (alo,,
1
D) [logp(zle, D) + El) [p(eeesee) (M))))))
Am
1
Eptelee. [log Ee D) [p(a)le, Φm]]]]
Am
+
Ep(axiD)
(ppa log llgp( [l]]]]
Am
F
For part E, we can estimate it by
E Am Ki L1
1
log
1
Σ
p( de) (1) (obo))) e)
C
k==11=1
where for the first expectation p(@. (e, D), we first sample (ki) from om (k1) Kj times, and then for each @m (kt)) we sample am (k1.1) from am (k).4) p(æ @ e) for Lı times.
p(@mle, D) for
(ki)
For the second expectation p( D), we sample din))) p(@mle, D) for C₁ times. For part F. have
Eppcar D) [Eplalomee) (ogp(allarre))l
Am
/ p(xlos (lgpllx.e) dx
Am Ep(due)
[ p(xl@mm ()p(mm ΦM) dΦn logp(z) e) dæ
Ep(du(D)
Am
1
[ p( e)p(@ (M)) log e) dx dom
E E
Am
/ Eplcelomee) [p( ( log pog e)] dop...
E E (De))
Xm
Epposs D.e) Ep(siioo [ptaee plogp(x
can be estimated by
K₂
T-2
C2
Σ
log p(aolh (kg. Φ e)
Am K2 L2 C2
kz=1
1==11=1epp 1
15

---

where for the expectation on p(dM D, e), we sample (ka) For the expectation on p( for each @ M
p(omle, D) for K2 times. we sample @m (k2d2)) from @m ((aail)) ~ pl pm Φ (ka))
from & (kx)
(Az))
for L2 times. For the expectation on p(æ) @m e), for each @ (kz) and (12222) we sample a (22.)2.02) from Zm Therefore, conclude that [j, v, m) can be estimated by
(K2./2.02) p pm e. for C2 times.
(R2,)2)
C₁ p(
(0.u,)) Am Kn I1
log
C
1==1/=1
-1
K2
Lg
log ((arlkada (p .e)l
(122)
Am K2 L2 C2
k2=1 12=1
where Oin (11)
(k1)
p(@ml))) D), am (kid1) p(x .e), (12) p(oN) pm (
(kad2200)) pl p(al (1a.d2) e). p( PM D) and am
(kz)
Obviously, the above approximation of f(j, v, m) only depends on p(ó D pl 2m Φ D) and p(x|om: e). In the next, we show how to sample from them in Section A.2 A.3 and A.4 respectively.
A.2 Sampling from p(@m!))
Basically, sampling from the posterior of "'( (D) easy. To solve this problem, mentioned in the main paper. we introduce a variational probability "q" to approximate "p". . In specific, in order to sample from p( pm D), we first obtain a sample Φi from Φi ~ No, I), and then get Φm from the distribution ((0m (1).
Since
(( Φ1)
I(0m, Pm- dØ ... dØ2.
pm--
and
q( Q 1........ 22 (1)
II (
(=2
( NCcob + o,
we have (()ml(1) is Gaussian distribution, which is easy for sampling.
In our model, c, and of I are diagonal matrices, which means that the dimensions in Φ are indepen- dent with each other. We denote Φ; [dv,1, 1........idl where dijj is the jth element of &; Then, we have
a(pm: om 1, 22) II (m Pm 1ji--.. 02.1))1.))
So our target can be converted to calculate the probability gl0m, 0m-1,f (2)1015) for all dimensions V1 $ j s d. Let Cr.s di, and o. be the jth element of c, d, and of respectvely. We assume that di.j / //-1, + +100-11 (i 4) and Øs,j = σ2.j where is the papeeeparameter. Suppose Pi.j is the mean of the Gaussian distribution for ( (eb.sl@i-1.5) (0)) that is,
Mijj = G + di.j (i 2 2), then, the approximated joint distribution be represented
a(Omis 6m 1,6 .,, (2.5) = II ag( (o)-1)))
m
2ot (di., -pi
2
16

---

Then, we integrate $2.5, Ps.j..... 9m-1. sequentially to obtain a(om.)l01.))
First of all, integrate d2,j for the joint distribution, where we have:
a(Om. 4m 1.jh- $3. x1.5))
I q( Pm 1m---.t-- i..j g ()) diba.s
e 2r, (di,) -p
26255
1
20 (dr. pp,s
1
/
e202, (1s,2
+
V2TOss
V2
(d2., (w2d1, )l
2,5
di22...
Denote 22 C22 and d2.j d2.g. and because of 03. 02j. we have
g(0ms: Pm- (
1
1
209
-uis)
/
c c=03,
-1 10s,, (63,
V2T02
2
V2Tos,j
1=1
(62 2r3.,
+da.))
e
d62.
-i.5)
V2T023
V2TOOS
im4
[03.]
(c3, + d)
[02.5
(62,01,5 + d2
2,3
51
dΦ2.5
For S1, we have
81 = 103, (caj023 d3,;)1² + |025 (02J015 + d2.s))
= -. + 22 + b 23 - 203, j22. 22.1 - 2d3 + 02
+ 2 + dz, + 02 - 012 2d3.5
= (ch +1).
27
c3. c3 03,1 Ō2, d2.s) +
02,
d2.
22 +
+1
Cg +
(dg
03,03,
01
d2
+ +03 + & 0i + d2 + 02.
2d3,
22
+1
2
C3, + 22,01 do. - d3
- +1
22
22 +1
23 + tz
2 + 2d3
22 - 22.
3,j +1
2 + E + bb + 2dz, 02,00
2d3,503.1 -
22 222 da, 035
232 11
- -
- 203,
- 2d3,
o +1
17

---

Then we have
ca. + 02 + d2. - dayees
Si - + 02,
S +1
03.
21,2, + d25c3s d3.)(a.s
+
3. +1
(2 + d2 + d3.s) - Ly (0 +1)
C +1
C3 02, + d2.5 deyces
=(33, +1)
22,
G.3 +1
1
[ (ca, + d2 + ds.s]} dz.s.
+ c/3
+
Therefore we have
(m Om- @3.s (01,))
1
1
20
V2T02.5
2
+1
- +1)
3,,0,
t 11 tdgt - -33 )²
203,
92.3
o +1
d02,
v /2T03..
S2
2 (6, 11) (a, +d.
1
20
[ +1
The S2 part is the integration form of d2.s ~ (( ,o equal 1, so we have
da
- +1
0
which is
q(mmis &m l.f...
e 2o,, (dis ((is)=
-1 e 203, +
( +d..))
v 2TT22
v 2noss
i=4
5.5
C
+1
13
We denote č3.j = C2, and da,j = d2. + da.j. and denote r2. =
203,
so we have
g(0m. Qm 1.f...... 23.5 23.))
m
(oh.. -m
203, 103,, 11) los,, l0s. (es. +ds.))
e 20?
.e
r22
V2T023
1=1 V2mossj
m
(di.s -
ioh (cids, +d]]]² 20
201
V2T02J
2
V2TOA
6=5
(da.,
(2a, +ds. 1|
305 (0, +1)
T2.j
m 22
(dit.s -
1
c 20 s0 ( +d,]l²
V
V
i=5
2oर (oa.) (Pa. )))
e
T2.
18

---

Similarly, then we integrate 03.j
(m Im 1..... Φa (11))
= = ((m jo Pm l.j..... $3.5 (11.)) do3.j
12.5
xo (ow -po))
1
20
los 1 (csdssstt))
e
V2TO2.
V2T04
1=5
20
[da.
(Cs.sh +
e
d02.j.
The formulation is similar to the previous one, so we can utilize the process above to integrate succesively, and we finally obtain
1=2 rij.
e artiis (cn (omm. (rm dmis))
which indicates
11 m- Pij
omm (Bmthe tdmms)
p D)
3o mis mo m -15
V2T022
where we have the iterative calculation by
11
Tijj = e 2
V / +
Cjj = C-Ci-1jj) 1,0 (i2 3),
di = d di + di (i 2 3).
A.3 Sampling from p(pm.o., D)
To sample from the distribution pm @ D), we first obtain sample Φi from the prior distribution (i.e,, Φ1 ~ ~O, I)), then get Φm from a consecutive sampling process:
ΦM-1 1 $1, D), 0M-2~ ~ 2 D),
o.. pm @m+1 Φ1, D),
because of the Markov property in our cascaded model. So our target is obtaining the distributions p(o;-1l0i, (1, D). For certain p( Φ1, D), according to the Bayes rule, we have
p (. 11 D) p(0: 1(01-1)01.1)
16 $1 D)
p( D)
Similarly with the last section, we use non-bold symbols to represent one dimension of the multi- dimension parameters, where they are able to transfer independently, and finally construct the eventual parameters by concatenating, that is,
p(o:-1l0: D) = II plo.. 10-11. D).
So our target can be converted to calculate the probability p( p(0i.1l0 10 Φ1.. D) for all dimensions v1 < j < d. According to the Markov property and the transportation probability, we have
p(disloe 1. &b..j D) R g( 10
1.j 21.)
1
(Bi. - te: tda.)l e 2
v 2
19

---

According to the previous section, we have
IT Tis Tigj
-1
(or,
1
p(ot,lo1.;: D) a(01,101.g)
2202 16,,
20
V2T022
2
11
lo: 15-18 worstt 1.1]
p(01-11/61.1 D) g(0r-1=1011)
V2T02J
Then we have
( 01,5) 1(0i-151011,
p(o: p(0i-15101, (1,, D) ~
q( ((01)11)1))
11-15 (6--1,9 1,901, 4-11
(1.
20 (
6
V 2 Ti-1
-207
20?
[a ( (119-1111 1dhall
(o (4 ( 1d
Io- ( -202
Gr +1
20/41,3
2033
2T
e
207 +
Gij +1
203
2 4d
[ois (caDi-1s + dr 10--15 - (0-15011 + +
20
Then we calculate the part C as
= [Rij - (ci
+ 10--1- + d+)1))
= oi + (ci.0i-1)
2(c,, +
Φi-1, + +(W,-101, + d. )² 2,
d1-1,5)
oig + oi-1 di + 2d. 20, 01- 1 2d,,, Ri,; + o7-1.
+ 0 + Rhis + 20, 1 11011 - 26, 1 11 - 2.1--118--115
2
Φ1-15 G.giu + 1- 01 + d di.
= (w +1)
B,
c +1
where B does not include thi which indicates
0 01 D) N Gistis + 8- + di di di
of
2 +1
A.4
Calculation of p(x)e,
In this section, we will show how to calculate the graph probability p(zle, Φm). Remember the graph parameters &n. [0m: Sm: Tm. so we have
p(z e, @n) p(xle, P.. E) P(Ee, Sm. Tm, dE
JE
-Eg vnd ((xaee, Om. E)
According to Monte Carlo sampling, have
K
p(zle, Φm)) = K
Σ p E.).
where [y ~ Bernoull(a(ST..1 DT)). In order to conduct intervention process, we change the jth column of E; to zeros, and represent it with E, Moreover, we replace the jth element of æ
20

---

with v, and get the result Σ. We change the jth element of Cm with zero, and get the result Em Then according the definition of causal graphs, we have
E p Φm) = K
1
Ni f(E; Ed, Ym), Em);
1=1
where f is the causal function that depends the parameter 7m-
B Bayesian Optimization for Determining (j", 2², m²)
We intend to find the best tuple for acquisition, that is,
( v² m²) arg max fj, v, m). (...))
We define the best interventional value a under interventional node j and fidelity m as (j, m) = arg max f(i, v, m)
-arg (1).
where (u) is rewritten from f(j. v, m) under given j. m. Therefore, our task is calculat- ing v" (j.m) m) for Vj E /dl.m E [M] with Bayesian optimization [47] We utilize a Gaus- sian Process (GP) 48 model surrogate function distributions for each v" m). We denote if ~ (P U2)), and K(u,,, v2) is the kernel of GP. We sequentially find and calculate f3. (we) direct the process. According to GP, the previous functions and the function are multivariate Gaussian distribution,
F1a fi+1
K,
ke+1 (((4))))1)))))
N
0.
h
where we define
F [fi, f2..,.... fel.
k441 = [K(M+4 UI), K((++1, 02),... K( ((+1))
K((1,01)
K( m1)
K, =
(11)
K((,,v) v1) K(vi, vt)
Given previous steps, we have the posterior probability is P(f++1l((1,, fy. (v:)) v(41) (f
with the non- non-parametric means and variances
Me K (K +1) 'FI.. 1.1.
(12)
(1(41) = KC(u+11 U/+1) k (K +1) +1) -
(13)
We acquire the next 0441 with GP-UCB [49 function
at+1 (v) = (v) + Bae oo((v),
0++1 = arg max (v)
where Br is hyper-parameted Suppose the maximum of steps is T, the final output of function 0* (j.m)
(j, m) = arg max (v)..
Then we choose the best interventional node j and fidelity m by their best values under O(d M)
m jm.m" = arg max o" (j.m).
J.
2 v " m3)
21

---

Detailed Training Process of ELBO
C.1 Derivation Process of ELBO
Because the distribution g( to approximate the distribution p(0m), then we intend to minimize the distance between these two distributions optimize the parameters g(0m). where we utilize KL divergence measure the distance, that is,
V
arg min
V
According to the variational inference, we have
KL (0 ( |pp( D)J
(( log (()) D)
d4
/ q log 8 ④ dΦ
(($) log p( d$
p( ( D)
E ) olog q( [)]
9 ( log
P(D) d$
/ (( ( log pl 4 D) d +
E (+)) [log 1( [)]
(($) log DD) d④
(+)) olog (( ④
E. 4 v 4
log 1 D)]
q log D) de
E (+)) (log (( ④b) E logr( 4 D) + l(g
Because log D) is not related to V, minimizing KL [ is equivalent to maximizing the ELBO part, and we have
ELBO
( [log p( d,, D)] Ex [log q( ())
E ) [log [ + Ex Φ) [log Φ)] ( [log q( [b)
E [log (D) log q( 4b) log ((b)
Above all, we can conclude that
T arg min (L (
equivalent maximize evidence lower bound
V arg max ELBO
arg max E (0)) (D(@) logg + logp(())
C.2 Estimation of ELBO
We represent the equation of ELBO as
ELBO
( [log p( D)
log ( + log p($)
E (log p D)
E Eqp [log ( log +
For the part A, we have
N
E Ex)ng(()) log II
p(a (000 ( (0mm m0 ..)
22

---

where N is the current number of samples in buffer. Then we have
N
A (
log
II
( v10) m()) +)
iml
N
E.p-ol +
log p( (8) 11(00) 2000) _m! ))
p(zl) (1(0) ml() x))
Es m0 ro m())
(ogp(zz0))11 v(0) ml pmcr)
Using Monte Carlo sampling
50
we calculate the expectation by Ng samples for each point.
N Ns
log p( (1)00 (0) ( (6) m
A
where we sample of (
mls) ~ (()m)))) with size Ns.
Then denote the distribution (( ND Eav), and similarly, have p(Φ) I B= NNuaa, Extt.. Both the parameter Aau, Eau can be represented by the parameters in V while Pau and Eall are constant. Then we calculate part B
=EE ( [log q( logp($9 (())
(iaall Eail)
N (ian, Dau) log
db
Im=1 e 8-1(8m, Tm,) -Nualll Eau)
Ninnu Eau)
dD [ N(pult, Dall) log
(iant Eau) log
dΦ
N(pall Dail)
II --..((mm.Tm)
M
KL[N(Pall: Eau)l Eau)] + Nipau Eau) log
II m=1
B f... dΦ
4
According to KL divergence of Gaussian distribution, we can calculate C' in close-form.
C CKILIN (Malt Eau)lW (pallt Eau)]
Eaull d tr(E (E-EISaa Dau) + (faall (all) End --1 (faall Mall))
2 log
Eaull
Then we calculate D by the following steps:
M
D =
NHuall Ear) log
II m=l
8. (Sm. de
M
N(Mal Eai)
Σ
log 8 (S T.,) d
m=l
/
N( Eall)
Σ
B f(Sm.
Tm) dΦ
M
3 Eg Eai)
Σ
S(S m Tm
m=1
23

---

Using Monte Carlo sampling, we can calculate the expectation by ND samples for each point.
D =
f(S
T
(=1 m==1
No
3
E ME
T
[] [tr (c d] + X2 ||
1=1
where we samples ((() N (pau, Ean) with size Nn. Using Monte Carlo sampling again, we can calculate the expectation by Ng samples.
ND M Ng Ai
D= B
Σ>
(c d + A2 [|||
j=1
where we samples E Finally, we obtain the estimation
p(E|S m
(3)
T with size NE.
N
No
ELBO
log pgr
v00) m
2i m(n))
i=
j=1
Σ Eatll
log
+ 1 Eall) ) + (Mall mall) Eall I
(Fall paal))
Σ Eatlll
Np
3
[A
[tr (c") d + >2 -E
C.3 Gaussian Reparameterization Trick
In the last section, we derive the objection function for optimizing the model parameters, where can use methods of the gradient decent solve However, a significant problem rises due to the sampling process, because the gradient of model parameters can not pass backward from the naive sampling process(i.e., untraceable). Therefore, we use Gaussian reparameterization trick to make the Gaussian sampling process traceable.
In specific, we will demonstrate the traceable calculation of & by Gaussian reparameterization trick. In order to sample & ~ (H,, we first sample 5 ~ I). instead, and then obtain Φ = μl 8 @ E. Therefore, the gradient can be traced from μu and E. In specific, both μa and Σ can be represented with the function of learnable parameter V
C.4 Gumbel-softmax Reparameterizatiio Trick
Besides of the Gaussian sampling process, the Bernoulli sampling in our equation not traceable either, so we utilize Gumbel- Gumbel-softmax reparameteriization trick to make it traceable.
We demonstrate the traceable calculation of E p S, T) by Gumbel-max reparameterizztiio trick. According to Gumbel-max [5T], we have
Bernoullipp) > [G, + logp > Go + log(1 p)]
Go, G1 Gumbel 1).
Instead of using unit step function, we utilize sigmoid function
oGG logp Go + +log(1 pp)).
Therefore, we have
Ei,- = a (L.. + ST. TT);
where Lij ~ L(O, 1). Therefore, we sample Lij ~ L(O, 1) instead, where L(0, 1) logistic distribution, and calculate E,, = (L,, Sf Ty) to trace gradients. Specifically, both S, and T, can be represented with the function of learnable parameter V
24

---

C.5
Optimization of ELBO
With the estimation and reparameterization trick, we are able to conduct gradient descent methods to optimize our parameters with the objection function
arg max ELBO.
The format of stochastic gradient descent (SGD) is
ŌELBO
-
V 4
0v
where y is the learning rate.
D Training Process of Constraint based ELBO
We intend optimize our parameter with
V arg max (x))( (lgg(( -log + logp(#)),
s.t.
T. (e,, D) <c
(e..e.)
However, the objection has a constraint, which is hard to optimize with gradient descent methods. So we utilize Lagrange multiplier [52 convert it to constraint-free method:
Vr* arg max E g( [logp(D)) Φ) log q( + log p( ④ + A
1(x 1 te,, eit. D). (e,.ec)
Σ
where À is the Lagrange multiplier. Then, we intend to calculate the constraint part.
First of all, have
1( (e., D) =11(0.|oM (e,, e,), D) + (z
H(x, 2.lOM, (e,, e:), =H( e, D) + H( e, D) H(,,, as f ec), D),
For the term H (xx M e.. D), we have
D), H (z, 1oM Cs, D) = | eo)
e., D) dx,
p,,, pm.... [logp(æ, Φrs e., D)]
and we have We use Monte Carlo sampling to estimate H scc, es, D).
Ka
H M C.,, D) Ki K₂
log (kr Az)) (b))
where sample graphs t a(Ømle", ΦM)), and obtain samples x (kr. p(zle" s1a). Similarly, we can calculate
K log pl x (ki, Φ15),
H(I.OMM et. D) K1 K2 ki=1
12=1
where we sample graphs th glmmlle ΦM), and obtain samples (M)) ~ p(z)ee, (.1).
And have
Ki
Ka
K2
H (xx,, x, b0. (e,,er)
log pl (k) (ki Ab)) (k),)))) ((e,, @M))
Kg K2 Kg
k
25

---

where we sample graphs t ~ g( eet, ΦMr, obtain samples x (((1.1)) ~ p(ø/e", 2 and obtain samples (k) p(zle', ).
Therefore, we add constraint on the original loss function to obtained the estimation of constraint based ELBO, that is,
ELBO
log p(x (10), m
ob mits)
i=1
Eail
log
+ tr( Eall) + (Hall - mau)" "Eal (Pall - Mau))
2
Eaul
No
Ng
[] [ir ( -d 12 [||] + [T(a,; et), D))
E Proof of Theory 3
Proof. To begin with, we introduce two anchor variables a, e, indicating existing samples and experiments in the system, which are independent with the following experiments. Since a. Ze e-independent given ΦM, {e,, et} and D, we have:
[ w fes, e)) = I(x a.l. a, eu fe,, ei). Se ex), D) + H( OM a, e (e,, et), D)
-H a. (e,, e D) E,
Since
[[x Hu x, e (e,, e,), D) = e.}. D) I(x; on eu e,), D) eu D) Z, e,), D)
x, zue D)
We have:
(e,, [( Auciooll (e,, ee, (tch (es, ee,, D)
=H(x, eu e) D) + H(
-Ha,, e. a fe,, ex), D)
D)
_H, x e e,), D) H x. x, el fe,, e,), D)
=I(x, mi ou x e {e,, e,), c
According to the basic mutual information property ((B:C
I [(A; CJB), have:
I(x u dw u ex), D) (ee.
+ +1( U wi ow eulleffen et), D) - [( (e,
> 21 U (a,, @M (e., e:), D) - 1(: (e,, et): D)
Thus, we have:
I(x u.; do (e,, ei), D) + I( U o e D) >1 U (a,, J @Mlee U (e., e, D) + 1(x; @M/ (e,, e:), D)
Since different experiments are independent, we have:
1 (xx Jzi > (ar U (æ,, );) ) & (e,, ex), D) 1( &M le, D) c.
D) + 1(x 1(x u (ee))
Thus, (M. D) is e-submodular.
F Proof of Theory 4
For clear presentation, we denote g((e)))))) = ( D), then we need to solve the following problem:
arg max g(e.) {c
(14)
26

---

Suppose S = {e; Ji=1 is the optimal solution for objective [4). and the results of the greedy
method is S i= where the experiments are sequentially determined from e, to e,, We
denote S1.j = {e; ) and A(e)S15)) - g(S1. Ue) g(S1.)) ). according to the greedy method, we have:
A(e(S1:))
ej+1 arg max
Ae
where Ae is the cost of experiment e.
Based on all the above notations, we have:
g(5*) s(5 U11
= ((Sig) + 1 Uei) - ((Sig)
+ (Sin Uei UeS) 9(1ng Uej)
+...
+ tSSn U (ei,.... e g(Xini U ter e.
+ = 9((i))
[0(S U fer e.)) - g(xn eej
k=1
5g((in) +
La(S U (ett) g(Su.) +d
=(((g) +
Σ [A((E)))Sng) +e],
where the first inequality holds because the non-decreasing property, and the second inequality holds because of the e-submodular property.
Since ej+1
arg maxe
A(c(S1.) we have A(e(S)))
Am
An
A(epa1SSu) for any e, thus A(e(Smn)) s
Aayt1
Ae_ A(e)+1|S1s) S B, By bringing this result into the above equation, we have: Anst1
n
((8)) +
[((e)))) +c]
(0((1)) +
B (e + c]
k=1 ((( + nB B ne
Let T, = g(S")= - g(S1.)), we have:
T; - T3+1 = -(S(g(t)) = ((ej+) Tn ne
nB
Then
1
1
Ta < (1
(T
+
< [(1
1| T
+
nB
nB
nBx
B
Bx
1
1
< < [11
T) + [[1
n +... B
nB>
nB,
Bx
Let B = [1 1 = +
+ Bg = B, Σ
[1
1 ) and considering that [(1
nBA
nB,
nB )1n Bx. have:
(( -((1mm) SE- th g(8") + B
Thus, we have ((S1.n) 2 (1 - xx )g(5)) B.
27

---

Algorithm 1: Algorithm of Licence for Single Intervention Scenario
Input:
Variable set Xv, number of oracles M, cost of oracles A, observational data D total budget C, and learning rate 17.
Output: Causal graph Φv-
Initialize the model parameter V
2 Optimize with the training process ELBO under D
Initialize D V.
while Budget C' does not out do Initialize ,"m" , and let n* = -00.
for (j, m.) in (1,2....,dh) d) x (1,2..... M) do
Calculate 1² (j. m) with BO.
if f v² (j, (m)mm) > -5 then
Update j f- J. m t m and 8" + u (j.m). Update C +- f(j, 2 f)m)
10
11
end end
12
13
Subtract the budget with C' C'C
14
Acquire (j", 2²" .m") towards the true causal graph to obtain x Pm(AV/do(X, =v)).
15
Update D' < DJ U {xx).
Optimize V with training process of ELBO under D U
17 end
18
Sample ΦM from p( p(@arlD return Causal graph ΦM.
19
G Algorithm
The algorithm for Licence method for single interventiion scenario is shown in Algorithm Moreover, the algorithm for Licence method for batch interventiion scenario shown in Algorithm
H More Experiments
H.1 Experimental Settings
H.1.1 Datasets
The details of our experimental datasets presented follows:
Erdis-Rényi (ER) [41] graph is random graph introduced by Paul Erdös and Alfréd Rényi. For ER graph, a graph with vertices is generated by connecting each pair of vertices with a probability p.
Scale-Free (SF) graph is type of random graph that has a degree distribution following power law. A small number of vertices in SF graph own large number of edges, while the vast majority of vertices have relatively few edges.
DREAM 4 is the abbreviation for Dialogue for Reverse Engineering Assessments and Methods, which can estimate the reverse quality that causal discovery methods perform. Specifically, we use biological graph generator GeneNetWeaver for experiments, which is real-word public dataset.
H.1.2 Baselines
The details experimental baselines are demonstrated as follows. We utilize DiBS [53 as our basic graph representation component. For acquisition methods, use AIT and CBED and obtain the query tuples node and value.
AIT 44 is an active learning method that utilize f-score to select intervention queries.
28

---

Subtract the budget with C & C C- Cm Update B' < B' 1 ((""", ,0'",," v", m))} Algorithm 2: Algorithm of Licence for Batch Intervention Scenario
Input:
Variable set Xv, number of oracles M, cost of oracles A, observational data D total batch experiment step T, total budget C, and learning rate 7).
Output: Causal graph ΦM- Initialize the model parameter Vr
2 Optimize with training process of constraint based ELBO under D
Initialize B¹ =
for t in 1.2...., do
while Budget C does not run oul and let 5* -00
v² Initialize ,"mm", =m",
for (j. m) in (1.2.....d d) x f1.2,.... ..,...,,,,.,............ M. do
Calculate u² (i, f(j. va (j. (j.m). > then
[j.m) m) with BO.
Update j + J. mm f- and v* t 0r m).
11
Update x* < f(j, m).
12
end end
13
14
15
16
end
Acquire B' towards the true causal graph to obtain fxr Pm (X (Xvldo(X
17
u)))u..m)BBB.
18
Update D < D' U {>* (...).
Optimize I with training process of constraint based ELBO under DO U D
19
20 end
Sample dM from pl ((MM)D return Causal graph ΦM--
22
based the calculation of mutual information (MI), which intend select intervention queries with maximal MI scores after obtaining new samples under current queries.
CBED g
For the batch intervention scenario, extend above methods with greedy strategy, which can promise an lower bound for approximation with submodular property. For choosing the fidelities to query, we use cireumstances, i.e., REAL and RANDOM.
REAL fidelity means the model always choose the highest fidelity to conduct experiments. This strategy is aligned with classic causal discovery under active learning paradigm without multi-fidelity settings, which can just choose the most accurate samples to conduct discovery process.
RANDOM fidelity means the model choose different fidelities randomly with uniform probability.
H.1.3
Metrics
The details of experimental metrics are demonstrated as follows. We utilize SHD and AUPRC to reflect the topological structure discovering performance, and design RMSE to reflex the predicting performance of functional relations.
SHD 45 is the abbreviation for Structural Hamming Distance, and it estimate the topological
structure by counting the number of different edges on adjacency matrix. We calculate the expectation of SHD under multiple graph samplings.
AUPRC [46] is the area under precision-recall curve, where consider entities the adjacency matrix as binary classification problem. The AUPRC also under the expectation for multiple graph sampling.
RMSE designed for estimating the performance grasping functional relations. We obtain several samples from the true causal graph, and let our model and the true causal function to conduct forward process respectively, then calculate the RMSE between the two results. We calculate RMSE by sampling graphs for multiple times.
29

---

Table 1: The left table demonstrate the details of the configuration of device and platform. The right
table shows the details of time cost on computation.
Name CPU GPU Memory Python
Details
Model
Time (secs)
Intel Xcon Platinum 8350C 2.60GHz
AIT-REAL
7.686
RTX A5000 (24GB) 42GB RAM
AIT-RANDOM
7.451
CBED-REAT
7.998
Version 3.8
CBED BEED-RANDOM
7.989
Java
Version 1.8.0 (Necessary for DREAM)
Licence
8.320
Table The details of experimental settings.
Name budget
Explanation
Value
The total budget for interventional experiments, (i.e., C).
10/20/30/40/50
oracle number oracle cost oracle noise observation number expect edge number additive noise
The number oracles, (i.e., M)
The cost for each oracle, (i.e., A)
2, 8, 32 0.04, 0.02, 0.00
The extra additive noise for each oracle. The number of observational samples.
1000
The number of expect edges. The value of additive noise during data generations.
2 0.01
H.2 Simulation of Oracles with Different Fidelities
For given intervention v), suppose have M oracles {dh 02, -.. then the experiment results (Iy,e,1, Rij,, } are specified as follows:
Jj.,m = Ij..M + 8ms
5m (0,
where 2 M is the ground truth, which can be directly obtained from the datasets. Since ag,.m is correlated with T by the first line, their underlying oracles om and ΦMs are correlated in our simulation. In experiment, we set 8 82 ... dM O. Suppose the cost of om as Ams then we set A1 < X2 < ... AM-
H.3 Details of Configurations and Computation
The details of the configurations of device and platform demonstrate in Table (left). We will show the details of the time cost on computation. We measure the time cost on the generation each intervention per fidelity for all models, and the results are shown in Figure rright). We find that our method cost a little more than the baselines, which is probably due to the more complex sampling process in our model.
We also show the details experimental settings for our overall experiments in Table 2 We carefully tune the typer-parammter for baselines and our model, and the final values can be obtained in the configuration file our codes.
H.4 Experiments on DREAM Dataset
We conduct experiments real-world biological dataset, called DREAM. Note that, DREAM does not support the calculation of RMSE, because of the lack of interface in this real-world dataset. We use two sub-datasets Ecoli and Yeast our true causal graphs. The results are shown in Figure We find that our model outperforms that other baselines on both Ecoli and Yeast, and both single and batch intervention scenario.
H.S Experiments on More Nodes
In this section, we conduct further experiments on datasets with more nodes. We extend the number of nodes from 10 20, and experiment on the ER graph. The results are shown in Figure We find that our model is still effective on the scenario of more nodes, and is better than baselines.
30

---

Ecolittingle)
Veast(single)
Ecoli(batch)
Yeast(batch)
40
30
20
23
budget Ecoli(single)
budget
budget Yeast/batch)
Yeastlsingle)
Ecolifbatch)
0.28
0.30
AUPRC
0.24
0,250
AUPRC
0.26
0.28
AUPRC
0.25
0.22
0.225
0.20
0.200
budget
total
budget
Figure 4: The performance among models DREAM datasets with different datasets and budgets. Lower SHD, RMSE indicate better performances. We conduct each experiment for ten times, and report the average performances and error bars.
Table 3: SHD results of 20 nodes graphs on different budgets. Lower SHD indicates better perfor- mances. We conduct each experiment for ten times, and report average performances and error bars.
Model AIT-REAL
B1dget(10)
Budget(20)
B3dget(30)
Budget(40)
Budget(50)
63.36:14.89
64.36:15.18
64.5316.83
63.28:1.86 868
64.35:5.19
AIT-RANDOM
63.62:14.61
62.16::5.75
64.60115.23
66.87-1.47
63.53:5.27
DiBS-REAL
63.58:16.35
61.50:17.69
63.50:1.866
63.56:16.34
61.4547.69
DiBS-RANDON
63.68::6.77
65.07=1.44
63.91#7.14
63.9914.46
63.86+3.00
Licence
49.67=11.64
49.61
55.6818.63
51.34#11.24
51.3619.11
H.6 Supplementary Experiments on MAE Metric
We further compare model with the baselines based the Mean Absolute Error (MAE) metric. The experiments are conducted based on ER with different total budgets. The results are presented in Table
Table 4: Results of the metric MAE (%).
Model
Budget(10)
Budget(20)
Budget(30)
Budget(40)
Budget(50)
ATT-REAL
3.46#0.01
2.43+0.01
2.6340.01
3.4620.01
2.42+0.01
AIT-RANDOM
3.71#0.02
2.42+0.01
2.82+0.01
2.68+0.00
2.54+0.01
CBED-REAL
3.73+0.01
2.56+0.00
2.54+0.00
3.6920.01
2.52+0.00
CBED-RANDOM
4.00=0.02
2.53+0.00
2.88+0.00
3.14+0.01
2.70+0.01
Licence
2.06#0.00
2.20+0.01
1.70=0.00
1.77+0.00
2.07+0.00
The results indicate that our model surpasses the baselines in terms of MAE. This further provides evidence that the superior performance of our model is general conclusion.
H.7 Supplementary Experiments on Different Oracle Settings
To demonstrate that model is generally effective for different cost- and noisy-levels. We conduct experiments based on different sets of oracles. In specific, the experiments are conducted based on the following settings in Table The results presented in Figure
From the results, we can see that our model can always perform better than the baselines on different sets of oracles.
31

---

Table 5: Settings for different cost and noise levels.
Setting
Setting
Oracle (m)
cost (Am)
noise (am)
Oracle (m)
cost (Am)
noise (am)
1
0.04
0.04
2
8
0.02
8
0.02
3
32
16
0
Setting 3
Setting 4
Oracle (m)
cost (Am)
noise (am)
Oracle (m)
cost (Am)
noise (am)
1
2
0.04
2
0.04
2
0.02
8
0.02
3
32
32
0
Setting 5
Setting
Oracle (m)
cost (Am)
noise (am)
Oracle (m)
cost (Am)
noise (am)
1
0.08
0.04
2
8
0.02
8
0.03
3
32
32
0
H.8 Supplementary Experiments Regularization Coefficient À
In our model, the c-independent constraint in Equation 10 is an important contribution. In the optimization process, convert it the objective. We study the influence of the coefficient by tuning it in the range of [10-5,10 1ee.-11..00000 .100-, 1e11111000$____....
.100$,100000000 $0009,.. The results are presented in Figure 6
From the results, we can see the performances of our model varies as set different X's. In most cases, the best performance is achieved when is moderated (not too large or too small).
H.9 Supplementary Experiments on Ablation Studies
To study whether the correlation modeling between different oracles are necessary, we first build variant of our model by regarding different oracles as independent components, that is, removing the links between different d's in Figure I and then compare our model with such variant. The results are presented Table
Table 6: Results of the Licence and Licence without the cascaded relation.
Metries
Model
Budget(10)
Budget(20)
Budget(30)
Budget(40)
Budget(50)
Licence (w/o rel)
14.61=2.30
15.25=2.74
15.47+3.74
15.1744.30
18.52+4.65
SHD 4.
Licence
14.67+2.98
14.73+2.04
14.29&3.20
14.83+3.24
15.0243.01
Licence (w/o rel)
28.96:1.28
35.05=1.35
31.7421.50
37.8542.03
33.7014.36
AUPRC (%) f
Licence
35.75+2.13
25.12+2.01
41.79x1.71
40.89+2.92
41.76+3.19
Licence (w/o rel)
3.34=0.0M
3.10=0.00
2.46=00.00
2.61#0.00
2.83+0.00
RMSE (%)
Licence
2.82+0.01
2.75+0.00
2.40=0.00
2.68+0.01
2.69=0.01
We can see, in most cases, our model can achieve better performance than its variant without modeling the correlations between different oracles.
I Potentially Negative Social Impact
Causal discovery focuses understanding causal relationships between variables. While causal discovery has the potential to bring about positive social impacts, it is important to consider both the positive and negative implications of its applications. In this response, will focus on the negative impact of causal discovery.
Reductionism and Oversimplification. Causal discovery techniques often aim to identify simple cause-and-effect relationships. However, complex social phenomena often involve multitude of interconnected factors, making it difficult to capture the full complexity of the system. Relying solely on causal discovery may lead to oversimplification and reductionism, neglecting the nuanced interactions between variables.
32

---

Setting
Setting
Setting
30
30
20
SH
54
10
10
20
total budget
total budget
total budget
Setting 1
Setting 2
Setting
AUPRC
0.4
AUPRC
0.4
AUPRC
0.4
0.2
0.2
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
total budget
total budget
total budget
Setting 1
Setting 2
Setting
0.06
RMSE
RMSE
RMSE
0.04
0.04
0.04
10
20
30
40
50
10
20
30
40
10
20
30
40
50
budget
total budget
budget
ATT-REAL
AIT-RANDOM
CBED-REAL
CBED-RANDOM
Licence
Setting
Setting
Setting
30
30
20
SHD
10
50
10
20
20
total budget
total budget
total budget
Setting 4
Setting 5
Setting 6
0.4
AUPRC
AUPRC
0.4
ALIPCC
0.3
0.2
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
total budget
total budget
total budget
Setting 4
Setting 5
Setting
0.06
RMSE
RMSE
RMSE
0.04
0.04
0.04
10
20
30
40
50
10
20
30
40
10
20
30
40
50
budget
total budget
budget
AIT-REAL
AT--RANOMM
CBED-REAL
CBED-RANDOM
Licence
Figure 5: Results of different settings of oracles in terms of cost and noise.
Budget
Total Budget
Total Budget
Total Budget
Total Budget 50
20
20
20
g 20
10
10
10
10
20
10
10
10
Total Budget
10
Total Budget = 20
Total Budget 30
Total Budget - 40
Total Budget 50
0.6
0.6
AUPRC
0.2
0.2
10-4
10
20
10th
10-1
10-4 10t4 10 13 10-3
10
10th
10t2
10 10
10-2
10-2
10- 10r4 10
10-9
10
10-1
10t4
10-5
Budget=
Budget
Budget
Budget
Budget
RMSE
RMSSE
RMISE
10055
10
10
Figure 6: Results of extensive experiments on regularization A.
33

---

Ethical Concerns. Causal discovery can involve analyzing sensitive data, such as personal information or medical records. If not handled carefully, the use of this data can raise significant ethical concerns related to privacy, consent, and potential discrimination. Improper handling of data could lead to violations of privacy and unfair treatment of individuals or groups.
Overreliance on Correlation. Causal discovery often relies on identifying statistical correlations between variables. However, correlation does not imply causation, and there is risk of mistakenly inferring causal relationships based solely on correlation. Overreliance on such methods can lead to erroneous conclusions, leading to misguided decision maiiiinnmmainng and ineffective interventions.
Social Bias and Inequality. Causal discovery relies on the data used for analysis, which can reflect existing biases and inequalities present in society. If the data used is biased, the causal relationships discovered may perpetuate or exacerbate existing social inequalities. Causal discovery methods need to be sensitive to potential biases and strive for fairness and inclusivity in both data collection and analysis.
In conclusion, while causal discovery holds promise in understanding complex systems, it is crucial to consider its potential negative impacts. Ovvesimplification, ethical concerns, overreliance on correlation, and social bias are all factors that need to be addressed to ensure responsible and beneficial applications of causal discovery. It is essential to approach this field with caution and incorporate broader societal considerations mitigate the negative impacts and harness its potential for positive social change.
J Limitations
In this section, analyze the limitations of our work, including sub-optimum of greedy method, estimation of mutual information, and scale of causal graph.
Sub-optimum of greedy method. For the whole process of active causal discovery, the interven- tional data will be acquired successively in the greedy manner. Therefore, even if the strategy for acquisition is the optimal for each current step, the entire trajectory of causal discovery is sub-optimal. possible solution is finding the best acquisition trajectory by reinforcement learning.
Estimation of mutual information. For different circumstances, the costs, accuracy and data scale can be various. Therefore, the scale of mutual information can be affected as well. So is important to adjust hyper-parameters accordingly.
Scale of causal graph. It is classic problem for causal discovery that most existing methods suffer from the difficulty in extending to large-scale graphs. The efficiency and effectiveness are supposed be further improved, and we will optimize our model as well.
References
[1] Jeffrey S Levin. Religion and health: Is there association, is it valid, and is it causal? Social Science & Medicine, 38(11) 1475:1:475- 1994.
[2] Eric Eide and Mark H Showalter. Methods matter: Improving causal inference in educational and social science research: A review article. Economics of Education Review, 3145)7744-748 2012.
[3] Bengt Muthén and Hendricks C Brown. Estimating drug effects in the presence of placebo response: causal inference using growth mixture modeling. Statistics in medicine, 28(27): 3363-332 3863-3385,22 2009.
[4] Karen Sachs, Omar Perez, Dana Pe'er, Douglas A Lauffenburger, and Garry P Nolan. Causal protein-signalinn networks derived from multiparameter single-cell data. Science, 308(5721): 229,529, 2005.
[5] David Maxwell Chickering. Learning bayesian networks is np-complete. Learning from data: Artificial intelligence and statistics V. pages 121-130, 1996.
[6] Thomas S Verma and Judea Pearl. Equivalence and synthesis of causal models. In Probabilistic and Causal Inference: The Works of Judea Pearl, pages 221-236. 2022.
34

---

[7] Raj Agrawal, Chandler Squires, Karren Yang, Karthikeyan Shanmugam, and Caroline Uhler. Abed-stratess Abed-strategy: Budgeted experimental design for targeted causal structure discovery. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3400-3409. PMLR, 2019.
[8] Panagiotis Tigas, Yashas Annadani, Andrew Jesson, Bernhard Schölkopf, Yarin Gal, and Stefan Bauer. Interventions, where and how? experimental design for causal models at scale. In Advances in Neural Information Processing Systems.
[9] Azam Peyvandipour, Nafiseh Saberian, Adib Shafi, Michele Donato, and Sorin Draghici. A novel computational approach for drug repurposing using systems biology. Bioinformatics, 34 (16):2817-2825, 2018.
[10] Yashas Annadani, Panagiotis Tigas, Desi Ivanova, Andrew Jesson, Yarin Gal, Adam Foster, and Stefan Bauer. Differentiable multi-target causal bayesian experimental design. arXiv preprint arXiv:2302.10607, 2023.
[11] Judea Pearl and Dana Mackenzie. The book of why: the new science of cause and effect. Basic books, 2018.
[12] Patrik Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlinear causal discovery with additive noise models. Advances in neural information processing systems, 21, 2008.
[13] Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv: 111.5745, 2011.
[14] Samuel Daulton, Xingchen Wan, David Eriksson, Maximilian Balandat, Michael A Osborne, and Eytan Bakshy. Bayesian optimization over discrete and mixed spaces via probabilistic reparameterization. arXiv preprint arXiv:2210. 101199,, 2022.
[15] Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. Dags with no tears: Continuous optimization for structure learning. Advances in neural information processing systems, 31, 2018
[16] David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: review for statisticians. Journal of the American statistical Association, 112(518)859--87, 2017.
[17] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 14, 2013.
[18] Andreas Krause and Carlos Guestrin. Near-optimal nonmyopic value of information in graphical models. arXiv preprint arXiv: 2012.
[19] Shibo Li, Jeff M Phillips, Xin Yu, Robert Kirby, and Shandian Zhe. Batch multi-fidelity active learning with budget constraints. Advances in Neural Information Processing Systems, 35: 995-1007, 2022.
[20] Christina Heennee-eemi Marloes Maathuis, and Nicolai Meinshausen. Causal structure learning. Annual Review of Statistics and Its Application, 5:371-391, 2018.
[21] Clark Glymour, Kun Zhang, and Peter Spirtes. Review of causal discovery methods based on graphical models. Frontiers in genetics, 10:524, 2019.
[22] Matthew Vowels, Necati Cihan Camgoz, and Richard Bowden. D'ya like dags? survey on structure learning and causal discovery. ACM Computing Surveys, 55(4):1-36, 2022.
[23] Kevin P Murphy. Active learning of causal bayes net structure. Technical report, technical report, UC Berkeley, 2001
[24] Simon Tong and Daphne Koller. Active learning for structure in bayesian networks. In International joint conference on artificial intelligence, volume 17, pages 863-869. Citeseer, 2001.
35

---

[25] Yang-Bo He and Zhi Geng. Active learning of causal networks with intervention experiments and optimal designs. Journal of Machine Learning Research, 9(Nov):2523-2544 -547,, 2008
[26] Hyunghoon Cho, Bonnie Berger, and Jian Peng. Reconstructing causal biological networks through active learning. PloS one, 11(3):e01 5061 2016.
[27] Robert Osazuwa Ness, Karen Sachs, Parag Mallick, and Olga Vitek. A bayesian active learning experimental design for inferring signaling networks. In Research in Computational Molecular Biology: s Annual International Conference, RECOMB 2017, Hong Kong, China, May 3-7, 2017, Proceedings 21, pages 134-156. Springer, 2017.
[28] Julius von Kügelgen, Paul K Rubenstein, Bernhard Schölkopf, and Adrian Weller. Optimal experimental design via bayesian optimization: active causal structure learning for gaussian process networks. arXiv preprint arXiv: 191000991, 2019.
[29] M Giselle Geddnd, Chanyoung Park. Nam-Ho Kim, and Raphael T Haftka. Review of multi-fidelity models. arXiv preprint arXiv: 160.0016 2016.
[30] Alexander IJ Forrester. Neil W Bressloff, and Andy Keane. Optimization using surrogate models and partially converged computational fluid dynamics simulations. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 462/2071):2117722244, 2006.
[31] TD Robinson, Michael S Eldred, Karen E Willcox, and Robert Haimes. Surrogate-based optimization using multifidelity models with variable parameterization and corrected space mapping. AIAA journal, 46 1):2814-2822, 2008
[32] Christopher C Fischer, Ramana V Grandhi, and Philip S Beran. Bayesian low-fidelity correction approach to multi-fidelity aerospace design. In 58th A/AA/SCC/AAS/ASC Structures, Structural Dynamics, and Materials Conference, page 0133, 2017.
[33] SA Burton and P Hajela. A variable-complextty approach to second-order reliability- optimization. Structural and Multidisciplinary Optimization, 25(4):237 2003.
[34] Akil Narayan, Claude Gittelson, and Dongbin Xiu. A stochastic collocation algorithm with multifidelity models. SIAM Journal on Scientific Computing. 36(2),A495-A51, 2014.
[35] Benjamin Peherstorfer, Tiangang Cui, Youssef Marzouk, and Karen Willcox. Multifidelity importance sampling. Computer Methods in Applied Mechanics and Engineering, 300:490-509, 2016.
[36] Leo Wai-Tsun Ng and Michael Eldred. Multifidelity uncertainty quantification using non- intrusive polynomial chaos and stochastic collocation. In 53rd ciaa/osme/ssc//ah/hae struc- tures, structural dynamics and materials conference 20th aiaa/asme/ahs adaptive structures conference 14th aiaa, page 1852, 2012.
[37] Tiangang Cui, Youssef M Marzouk, and Karen E Willcox. Data-driven model reduction for the bayesian solution of inverse problems. International Journal for Numerical Methods in Engineering. 102(5):966-99) 2015.
[38] Shibo Li, Wei Xing. Robert Kirby, and Shandian Zhe. Multi-fidelity bayesian optimization via deep neural networks. Advances in Neural Information Processing Systems, 33:8521-8531, 2020.
[39] Shibo Li, Robert Kirby, and Shandian Zhe. Batch multi-fidelity bayesian optimization with deep auto-regressive networks. Advances in Neural Information Processing Systems, 34:25463- 25475, 2021.
[40] Shibo Li, Jeff M Phillips, Xin Yu, Robert Kirby, and Shandian Zhe. Batch multi-fidelity active learning with budget constraints. Advances in Neural Information Processing Systems, 35: 995- 1007, 2022.
[41] Paul Erdös, Alfréd Rényi, et al. On the evolution of random graphs. Publ, Math. Inst. Hung. Acad, Sci, 5(1):17-60, 1960.
36

---

[42] Lun Li, David Alderson, John C Doyle, and Walter Willinger. Towards theory of scale-free graphs: Definition, properties, and implications. Internet Mathematics, 2(4):431-523, 2005.
[43] Thomas Schaffter, Daniel Marbach, and Dario Floreano. Genenetweaver: silico benchmark generation and performance profiling of network inference methods. Bioinformatics, 27(16):
2263 2270, 2011.
[44] Yashas Annadani, Jonas Rothfuss, Alexandre Lacoste, Nino Scherrer, Anirudh Goyal, Yoshua Bengio, and Stefan Bauer. Variational causal networks: Approximate bayesian inference causal structures. arXiv preprint arXiv:2 2021.
[45] Ioannis Tsamardinos, Laura E Brown, and Constantin F Aliferis. The max-min hill-climbing bayesian network structure learning algorithm. Machine learning, 65:31-78, 2006.
[46] Jesse Davis and Mark Goadrich. The relationship between precision-recall and roc curves. In Proceedings of the 23rd international conference on Machine learning, pages 233-240 2006.
[47] Harold J Kushner. A new method of locating the maximum point of arbitrary multipeak curve in the presence of noise. 1964.
[48] Carl Edward Rasmussen. Gaussian processes in machine learning. In Advanced Lectures on Machine Learning: ML Summer Schools 2003, Canberra, Australia, February 2-14, 2003, Tübingen, Germuny, August 4-16, 2003, Revised Lectures, pages 63-71. Springer, 2004.
[49] Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. Gaussian pro- cess optimization in the bandit setting: No regret and experimental design. arXXiv preprint arXiv:0912.3995, 2009.
[50] Reuven Y Rubinstein and Dirk P Kroese. Simulation and the Monte Carlo method. John Wiley Sons, 2016.
[51] Chris Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00711 2016.
[52] Dimitri P Bertsekas. Constrained optimization and Lagrange multiplier methods. Academic press, 2014.
[53] Lars Lorch, Jonas Rothfuss, Bernhard Schölkopf, and Andreas Krause. Dibs: Differentiable bayesian structure learning, Advances in Neural Information Processing Systems, 34:2411 24123, 2021.
37


Reviews:
[{"review_id": "WmkNoTzFhR", "reviewer": "Reviewer_Xcc7", "summary": "This paper addresses the problem of active causal discovery with multi-fidelity oracles, where experiments can be done based on different costs, precisions, and reliabilities. The paper formally defines the task of multi-fidelity active causal discovery and proposes a Bayesian framework consisting of a mutual information-based acquisition function and a cascading fidelity model. The paper also extends the framework to the multi-target intervention scenario and introduces a constraint-based fidelity model to validate the greedy method. The effectiveness of the proposed model is demonstrated through extensive experiments.\n", "questions": "Have you considered other evaluation metrics besides SHD, AUPRC, and MSE? How does the proposed model compare to other state-of-the-art methods in terms of these metrics?", "limitations": "No potential negative societal impact of their work.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "(1) The paper addresses an important and practical problem of active causal discovery with multi-fidelity oracles, which is more realistic than previous single-fidelity settings.\n(2) The proposed Bayesian framework, including the mutual information-based acquisition function and the cascading fidelity model, provides a novel and practical solution to the multi-fidelity active causal discovery problem.\n(3) The extension to the multi-target intervention scenario and the introduction of the constraint-based fidelity model further enhance the applicability of the proposed model.\n", "weaknesses": "(1) The evaluation and ablation studies for the proposed method are limited, and more comprehensive experiments could be conducted to strengthen the empirical findings.\n(2) Have you considered other evaluation metrics besides SHD, AUPRC, and MSE? How does the proposed model compare to other state-of-the-art methods in terms of these metrics?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "For reviewer Xcc7:\n\nThanks for your overall positive comments on our paper. We try to alleviate your concerns as follows:\n\n> Concerns: More comprehensive experiments could be conducted to strengthen the empirical findings. Have you considered other evaluation metrics besides SHD, AUPRC, and MSE? How does the proposed model compare to other state-of-the-art methods in terms of these metrics?\n\nThe reasons why we choose SHD, AUPRC, and MSE as our evaluation metrics:\n\n(1) These metrics are widely used in previous studies in the field of active causal discovery, particularly for the datasets used in our experiments [1-3].\n\n(2) These metrics provide a comprehensive evaluation of the model's performance from different perspectives. The first two metrics assess the accuracy of the learned topological structure, while the last one measures the performance of functional relations.\n\nTo alleviate your concerns, we further conduct the following experiments:\n\n1. More experiments on additional evaluation metrics (see Table 2 of the submitted one-page pdf).\n\n2. More experiments on the influence of different oracle settings (see Table 1 and Figure 3 of the submitted one-page pdf). For a given intervention $(j,v)$, suppose we have $M$ oracles $\\\\{\\phi_{1},\\phi_{2},...,\\phi_{M}\\\\}$, then the experiment results $\\\\{x_{j,v,1},x_{j,v,2},...,x_{j,v,M}\\\\}$ are specified as follows:\n> $$\n> x_{j,v,m} = x_{j,v,M} + \\delta_m ,\n> $$\n > $$\\delta_m \\sim N(0,\\sigma_m) ,$$\n\nwhere $x_{j,v,M}$ is the ground truth, which can be directly obtained from the datasets.  Since $x_{j,v,m}$ is correlated with $x_{j,v,M}$ by the first line, their underlying oracles $\\phi_{m}$ and $\\phi_{M}$ are correlated in our simulation.  In our experiment, we set $\\sigma_1 > \\sigma_2 > ... >\\sigma_M = 0$. Suppose the cost of $\\phi_{m}$ as $\\lambda_m$, then we set $\\lambda_1 < \\lambda_2 <...<\\lambda_M$. \n\nTo demonstrate that our model is generally effective for different cost- and noisy-levels. We conduct experiments based on different sets of oracles.\n\nIn specific, the experiments are conducted based on the settings presented in Table 1 of the submitted one-page pdf. The results are presented in Figure 3 of the submitted one-page pdf. From the results, we can see that our model can always perform better than the baselines on different sets of oracles. \n\n\n3. More experiments on the regularization coefficient $\\lambda$ under line 172 in the appendix (see Figure 2 of the submitted one-page pdf).\n\n4. More experiments on ablation studies (see Table 3 of the submitted one-page pdf).\n\n\nWe sincerely thank you for your time to review our paper and give positive comments on it. We are glad to answer further questions, which, we believe, can definitely improve our paper.\n\n\n**References**\n\n[1] Tigas, P., Annadani, Y., Jesson, A., Schölkopf, B., Gal, Y., & Bauer, S. (2022). Interventions, where and how? experimental design for causal models at scale. *Advances in Neural Information Processing Systems*, *35*, 24130-24143.\n\n[2] Scherrer, N., Bilaniuk, O., Annadani, Y., Goyal, A., Schwab, P., Schölkopf, B., ... & Ke, N. R. (2021). Learning neural causal models with active interventions. *arXiv preprint arXiv:2109.02429*.\n\n[3] Zheng, X., Aragam, B., Ravikumar, P. K., & Xing, E. P. (2018). Dags with no tears: Continuous optimization for structure learning. *Advances in neural information processing systems*, *31*."}, {"review_id": "oNenciAd50", "reviewer": "Reviewer_HBUA", "summary": "This paper proposes an approach for Bayesian active causal discovery with multi-fidelity observations. This approach has two main components: (1) a cascade probabilistic model handling the correlation between fidelity levels, and (2) a cost-aware information-theoretic acquisition function, which quantifies the mutual information (per unit of cost) between the causal graph and an observation at a given input location and fidelity level. An extension to the multi-target setting, where multiple nodes can be intervened simultaneously, is considered. In such a setting, a seemingly natural choice is to select the nodes to be intervened in a greedy fashion by iteratively maximizing the mutual information. However, submodularity does not hold, so the classical approximation guarantee is not obtained. To alleviate this issue, the notions of $\\epsilon$-independence and $\\epsilon$-submodularity. The high-level idea is to select points with low mutual information so that submodularity holds approximately. The proposed approach is shown to significantly outperform various baselines across three test problems.", "questions": "1. How is $\\epsilon$ chosen? More generally, how were all the algorithm hyperparameters chosen?\n\n2. I believe some of the datasets used are, in principle, not multi-fidelity. How was this addressed?\n\n3. It would be helpful to include figures showing the causal graph structure of the test problems considered.\n\n4. In the context of causal optimization, the term \"target\" is often used to denote the node to be optimized. Thus, I believe the term \"multi-target\" is misleading in its use here. Perhaps \"batch\" would be more appropriate.", "limitations": "The potential negative societal impact was adequately discussed (in the supplement). The authors also mention some interesting ways to improve their method. However, I believe two important limitations were not addressed:\n\n1. Computational cost of the proposed approach vs. the standard approach.\n2. Robustness to the choice of the algorithm hyperparameters.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The problem considered by this paper is of significant practical relevance.\n2. The proposed approach is technically sound.\n3. This paper is very well written overall.", "weaknesses": "My only significant concern about this paper is its empirical evaluation, as there are several details I could not figure out. See questions 1 and 2 below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "For reviewer HBUA: \n\nThanks for your comments. In the following, we try to alleviate your concern one by one:\n\n>  Question 1: How is episilon chosen? More generally, how were all the algorithm hyper parameters chosen?\n\nAs can be seen in Appendix D, when training our model, the constraint involving $\\epsilon$ is converted to a regularization to the objective. Actually, $\\epsilon$ play similar roles as the regularization coefficient, that is, $\\lambda$ in the equation below line 184. For $\\lambda$ as well as all the other hyper-parameters, we determine them by grid search based on the validation set.\n\n> Question 2: I believe some of the datasets used are, in principle, not multi-fidelity. How was this addressed?\n\n1. In our experiments, we regard the original ground truth as the highest fidelity oracle. Then we add Gaussian noises on the highest fidelity oracle to simulate the other oracles with different fidelities. We add more uncertain noises on the lower fidelity oracles to make it less accurate. At last, we manually set the costs of the oracles with different fidelities to ensure that \"higher fidelity oracles cost more than the lower ones\".\n\n2. The specific formulations of the above experiment settings are presented as follows:\n\nIn our experiments, we follow the common practice to simulate the oracles with different fidelities as follows:\n\nFor a given intervention $(j,v)$, suppose we have $M$ oracles $\\\\{\\phi_{1},\\phi_{2},...,\\phi_{M}\\\\}$, then the experiment results $\\\\{x_{j,v,1},x_{j,v,2},...,x_{j,v,M}\\\\}$ are specified as follows:\n$$\nx_{j,v,m} = x_{j,v,M} + \\delta_m,\n$$\n$$\\delta_m \\sim N(0,\\sigma_m),$$\nwhere $x_{j,v,M}$ is the ground truth, which can be directly obtained from the datasets.  Since $x_{j,v,m}$ is correlated with $x_{j,v,M}$ by the first line, their underlying oracles $\\phi_{m}$ and $\\phi_{M}$ are correlated in our simulation.  In our experiment, we set $\\sigma_1 > \\sigma_2 > ... >\\sigma_M = 0$. Suppose the cost of $\\phi_{m}$ as $\\lambda_m$, then we set $\\lambda_1 < \\lambda_2 <...<\\lambda_M$. \n\n\n3. To demonstrate that our model is generally effective for different cost- and noisy-levels. We conduct experiments based on different sets of oracles.\n   In specific, the experiments are conducted based on the settings presented in Table 1 of the submitted one-page pdf. The results are presented in Figure 3 of the submitted one-page pdf. From the results, we can see that our model can always perform better than the baselines on different sets of oracles. \n\n\n4. The above experiment settings to simulate the oracles with different fidelities are very common in the field of multi-fidelity optimization [1-3].\n\n\nWe will definitely add the above experiment settings in the final paper.\n\n\n> Question 3: It would be helpful to include figures showing the causal graph structure of the test problems considered.\n\nFollowing your advice, we have added many examples of the test causal graph structures in Figure 4 of the submitted one-page pdf.\n\n\n>  Question 4: In the context of causal optimization, the term \"target\" is often used to denote the node to be optimized. Thus, I believe the term \"multi-target\" is misleading in its use here. Perhaps \"batch\" would be more appropriate.\n\nThank you for your comments. In the final version, we will revise these inappropriate terminations.\n\n> Limitation 1: Computational cost of the proposed approach vs. the standard approach should be addressed.\n\nActually, we have already compared the computation costs between our model and the baselines in Appendix H.2. \n\n\n> Limitation 2: Robustness to the choice of the algorithm hyper-parameters.\n\nWe have conducted many experiments on the influence of the hyper-parameters, for example, the regularization coefficient $\\lambda$ (see Figure 2 of the submitted one-page pdf), the oracle cost- and noisy-levels (see Figure 3 of the submitted one-page pdf), the DAG regularization coefficient $\\beta$ (see Figure 3(b) in the main paper). From the results, we find that the model performance is robust to some parameters like $\\lambda$, but may be sensitive to the other hyper-parameters like $\\beta$. \n\n\n\n\n**References**\n\n[1] Li, S., Kirby, R., & Zhe, S. (2021). Batch Multi-Fidelity Bayesian Optimization with Deep Auto-Regressive Networks. *Advances in Neural Information Processing Systems*, *34*, 25463-25475.\n\n[2] Li, S., Phillips, J. M., Yu, X., Kirby, R., & Zhe, S. (2022). Batch Multi-Fidelity Active Learning with Budget Constraints. *Advances in Neural Information Processing Systems*, *35*, 995-1007.\n\n[3] Li, S., Kirby, R. M., & Zhe, S. (2020). Deep multi-fidelity active learning of high-dimensional outputs. *arXiv preprint arXiv:2012.00901*."}, {"review_id": "1e5GzNxntS", "reviewer": "Reviewer_am1k", "summary": "In this manuscript, the authors propose a novel method for active causal discovery (ACD) in a multi-fidelity setting, where experiments with different cost and accuracy can be designed and performed for network intervention for the purpose of accurately learning the causal structure.\nFor this multi-fidelity ACD (MFACD), the manuscript proposes License - a Bayesian framework for \"Multi-fidelity\nactive learning for causal discovery\".\nLicense adopts an information-theoretic acquisition function motivated by the popular Bayesian Active Learning by Disagreement (BALD) for the prediction of the best experiment, which consists of selecting the node for intervention, its value, and the fidelity.\nFurthermore, a cascaded fidelity model is proposed to capture the correlations between the experimental outcomes across different fidelities.\n", "questions": "Please see the questions and concerns in the above section.", "limitations": "The broader impacts of the current study and its limitations are not discussed in the current manuscript.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem of ACD is widely studied in a number of fields, especially so in life science to uncover the regulatory relations among genes.\nNetwork interventions are routinely performed in labs to unveil the causal structure of the network, hence designing experiments that can accurately uncover the causal relations among nodes and minimizing the overall cost of designing and carrying out the experiments is an important problem.\nMulti-fidelity experiments may often be considered in practical settings to maximize the value of information acquired by the experiments based on a given experimental budget, and the MFACD problem tackled in this manuscript is therefore of practical importance as well as theoretical interest.\n\nThe proposed acquisition function for predicting the best experiment is reasonable, as its acquisition function is motivated by the widely popular active learning scheme - BALD - which is normalized by the experimental cost to assess the cost-normalized value of information that may be attained by a given experiment.\n\nThe evaluation results in the manuscript shows that the proposed License method may have potential advantages over some alternative schemes.\n", "weaknesses": "While the problem being tackled in this current study is very interesting and is both of practical importance as well as of theoretical significance, there are several major concerns regarding the current manuscript as summarized below.\n\nAlthough the concept of designing multi-fidelity experiments to maximize the accuracy of ACD while minimizing the overall experimental cost is important, the type of multi-fidelity experiments considered in this work is not well motivated.\n\nFor example, the authors assume that there are multiple SCMs with different fidelities.\nHowever, it is unclear how the accuracy of the SCM at a specific fidelity is defined and what parameters govern this accuracy.\n\nAnd in practice, how does one construct or obtain multiple SCMs at different fidelities?\nWhere does the different experimental cost arise, if all SCMs are of the same form with different fidelities?\nSince the manuscript doesn't give any practical example of how such multi-fidelity models may be constructed or accessed and how (and why) they may differ in terms of the acquisition cost (of the experimental outcomes), it is unclear whether the proposed method and the derived results may be applicable in any practical setting.\n\nFor example, in drug discovery, it is typical to assess the efficacy of a drug candidate against a specific target using multi-fidelity models, starting from a fast ML surrogate model, and then to a high-throughput docking model, then use molecular dynamics simulations to assess the binding affinity between the drug candidate and the target.\nNaturally, there are differences in terms of the computational cost as well as the accuracy, hence decisions need to be made as to which model should be incurred to maximize the value of information on the computational investment made.\nUnfortunately, the current manuscript does not provide any real example to motivate the proposed method and its problem setting, making it difficult to see how the proposed approach might be applied.\n\nFurthermore, it is natural that multi-fidelity experimental outcomes will be correlated to each other (at least to a certain extent) and it is a good idea to explore this property when learning/constructing the multi-fidelity models and utilizing them in MFACD experimental design.\nHowever, in the current study, it is unclear how leveraging such correlations across different fidelities actually translates to the efficacy of the designed experiments and/or savings in terms of the experimental cost.\n\nThe description of \"extension to multi-target intervention\" is also somewhat confusing.\nBased on the technical descriptions in 3.3. and the equations (7)-(9), it appears that while the experiments (or interventions) are designed in sequence (hence referred to as \"a series of experiments\"), the interventions are being applied to the network \"simultaneously\".\nHowever, as the actual \"multi-target intervention\" setting being investigated in the current study is not clearly described (or described in a potentially misleading manner), there is some ambiguity in the experimental setting which needs to be clarified.\n\nTheorem 3 states that \"For any two experiments es and et, if the corresponding samples xs and xt are\nε-independent given φM, {es, et} and D, then I(·; φM |·, D) is ε-submodular.\"\nHowever, how can one actually guarantee whether a given pair of samples xs and xt are indeed ε-independent, considering the unknown and complex causal structure underlying these samples?\nThis needs to be elaborated more clearly.\n\nFinally, the experimental results shown in the current study are very limited, and far from convincing to demonstrate the general applicability of the proposed License method to general MFACD problems.\nFurthermore, the multi-fidelity SCMs used in these evaluations, the number of fidelities considered, and the respective experimental/intervention costs are not clearly described, which makes it difficult to understand how the overall evaluations have been carried out.\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "For reviewer am1k: \n\n> Concerns on the multiple SCMs.\n\n1. For the real-world scenarios, multi-fidelity oracles are very common, for example (as mentioned in the paper), to investigate the drug-disease causal relations, one can either conduct clinical tests (high cost but more accurate) or build simulators to obtain the medicine effects on the patients. \n\n2. In our study, it is very hard to find public datasets, which can be used for causal discovery and simultaneously contain multi-fidelity oracles. Thus, we firstly find the commonly used causal discovery datasets. And then regard the ground truths in these datasets as the highest-fidelity oracle, and simulate multiple low-fidelity oracles by adding Gaussian noises on the ground truths. \n\n3. The specific formulation of the above experiment settings are presented as follows: for a given intervention $(j,v)$, suppose we have $M$ oracles $\\\\{\\phi_{1},\\phi_{2},...,\\phi_{M}\\\\}$, then the experiment results $\\\\{x_{j,v,1},x_{j,v,2},...,x_{j,v,M}\\\\}$ are specified as follows:\n>$$x_{j,v,m} = x_{j,v,M} + \\delta_m,$$\n>$$\\delta_m \\sim N(0,\\sigma_m),$$\n\nwhere $x_{j,v,M}$ is the ground truth, which can be directly obtained from the datasets.  Since $x_{j,v,m}$ is correlated with $x_{j,v,M}$ by the first line, their underlying oracles $\\phi_{m}$ and $\\phi_{M}$ are correlated in our simulation.  In our experiment, we set $\\sigma_1 > \\sigma_2 > ... >\\sigma_M = 0$. Suppose the cost of $\\phi_{m}$ as $\\lambda_m$, then we set $\\lambda_1 < \\lambda_2 <...<\\lambda_M$. \n\n4. To demonstrate that our model is generally effective for different cost- and noisy-levels. We conduct experiments based on different sets of oracles.\nIn specific, the experiments are conducted based on the settings presented in Table 1 of the submitted one-page pdf. The results are presented in Figure 3 of the submitted one-page pdf. From the results, we can see that our model can always perform better than the baselines on different sets of oracles. \n\n5. The above experiment settings to simulate the oracles with different fidelities are very common in the field of multi-fidelity optimization [1-3]. We exactly follow the common practice in the multi-fidelity domain. We believe this is understandable, since it is hard to find public datasets with multi-fidelity oracles, but this research direction is meaningful and need to be improved.\n\n> Concerns on the effectiveness of modeling oracle correlations.\n\nTo study whether the correlation modeling between different oracles are necessary, we first build a variant of our model by regarding different oracles as independent components, that is, removing the links between different $\\phi$'s in Figure 1, and then compare our model with such variant. The results are presented in Table 3 of the submitted pdf. We can see, in most cases, our model can achieve better performance than its variant without modeling the correlations between different oracles. \n\n> Concerns on the description of \"extension to multi-target intervention\". \n\n1. Multi-target intervention aims to (1) firstly determine a set of intervention variables, and then (2) leverage them to simultaneously query the oracles to obtain the experiments results. \n\n2. For (1), we leverage equation (7) and (8) to determine the variables one by one within a budget $C$. This step only aims to determine a batch of variables. We still do not leverage them to conduct experiments. We use the greedy method (*i.e.,* determine the variables in sequence), because it has solid theoretical grantees and has been widely used before.\n\n3. For (2), once the batch of variables has been determined, we use them to simultaneously intervene the oracles and obtain the experiment results.\n\n> Concerns on Theorem 3.\n\n1. In the previous single-fidelity studies, the theoretical foundation behind the greedy method is \"T1: if $x_s$ and $x_t$ are independent given $\\Phi_M$, then the greedy method can be bounded.\" (see [4-6]).\n\n2. In our multi-fidelity study, we find that, \"$x_s$ and $x_t$ are actually not independent given $\\phi_M$\", which means that the theoretical foundation of the greedy method fails.\n\n3. To extend T1 to multi-fidelity settings, we introduce the concepts of $\\epsilon$-independent and $\\epsilon$-submodular, and propose Theorem 3.\n\n4. $\\epsilon$-independent is a concept describing the data independent character. When $\\epsilon \\rightarrow 0$, $\\epsilon$-independent equals to independent. When $\\epsilon \\rightarrow \\infty$, xs and xt can be highly correlated. For any dataset, there should be always an $\\epsilon$ which ensures that \"$x_s$ and $x_t$ $\\epsilon$-independent\" (although $\\epsilon$ can be a very large value).  \n\n5. Actually, theorem 3 is a general version of Theorem B.2 in [4]. When $\\epsilon = 0$, our theorem reduces to Theorem B.2 in [4] as well.\n\n6. The constraint in equation (10) aims to introduce inductive bias on the independent characters of the real data. \n\n> Concerns on the experiments. \n\nTo improve our experiments, we conduct a large amount of further experiments on the additional evaluation metrics, different oracle settings, influence of the regularization coefficient $\\lambda$ under line 172 in the appendix and ablation studies (see the submitted pdf).\n\n> The broader impacts of the current study and its limitations are not discussed in the current manuscript.\n\nActually, we have discussed the impacts of the current study in the Appendix. For the limitations, we will add them in the final version.\n\n**References**\n\n[1] Batch Multi-Fidelity Bayesian Optimization with Deep Auto-Regressive Networks.\n\n[2] Batch Multi-Fidelity Active Learning with Budget Constraints.\n\n[3] Deep multi-fidelity active learning of high-dimensional outputs. \n\n[4] Interventions, where and how? experimental design for causal models at scale. \n\n[5] Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. \n\n[6] Abcd-strategy: Budgeted experimental design for targeted causal structure discovery. "}, {"review_id": "TQBe74I4G7", "reviewer": "Reviewer_yEiY", "summary": "This paper presents a method for active causal discovery with \"multi-fidelity\" oracles. \nHere multi-fidelity refers to the option to request outcome labels for a given experiment from a set of oracles with different quality levels. \nThe method extends causal experimental design methods where experiments are defined by a given variable and value to this (to my knowledge) novel problem setting. \nAs such, their method views an experiment as consisting of a triple: (variable, value, fidelity) and defines a mutual information objective for experiment selection.\nThe method also includes a \"cascading fidelity model\" accounting for information shared between fidelity models.\nThey propose an \"$\\epsilon-$submodular\" method for multiple-intervention experiments.\nThey empirically validate their results.", "questions": "What issues would arise if the cost $\\lambda$ and the mutual information $I$ are not on similar scales?\n\nCan you know the scale of the mutual information? Is your mutual information estimator reliable?\n", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper makes novel non-obvious contributions, which I have summarized above. To my knowledge this is the first exploration of multi-fidelity active causal discovery, and they have provided a clever and principled solution. I cannot speak to the correctness of the improved greedy method, but the rest of the paper does not seem to have major flaws.", "weaknesses": "## Minor Concerns\n\nI really only have minor concerns, which I hope will help to improve an already excellent paper.\n\n1. **Writing Quality**. As an example, \"which is fundamental for many real-world applications, ranging from health caring, education, to drug discovery, and protein synthesis,\" is a little awkward and could be improved as, \"which is fundamental for many real-world applications, including health care, education, drug discovery, and protein synthesis.\" Another example would be  that \"In specific\" can be replaced by \"Specifically\" or \"In particular.\" Another would be starting a sentence with a citation should use the \\citet{} function rather than \\cite{} or \\citep{}, such as on line 34. There are other examples and I would suggest using a tool like Grammarly. I do not find this to be a deal breaker, though, and some sections are written very well (e.g., 2.1).\n\n1. **Motivating Example**. The example of a high-fidelity experiment as an actual patient outcome vs. a low-fidelity experiment as a  simulated patient outcome is a little weak. At least need to explain why the simulator could be lower-fidelity. If the simulator has domain knowledge, why not incorporate that knowledge into the causal model rather than try to infer that knowledge? Is it a black box? As alternatives, maybe consider spacial or temporally measured outcomes as examples of low vs. high-fidelity. For example, perhaps a lower spatial or temporal resolution measurement vs. a higher spatial or temporal measurement? A satellite image at 10-mile resolution vs. 1-mile resolution. An MRI from a 1T machine vs. a 7T machine? Maybe alternatively, a radiologist's assessment of malignancy vs. a biopsy result?\n\n1. **line 91**. covariance -> variance\n\n1. **f**. You use $f$ to define the SCM functions and the acquisistion function, which is a little confusing. \n\n1. **Active Causal Discovery**. Since you work on identifying the adjacency matrix and the  SEM parameters, I'd use causal experimental design. Only because causal discovery has focused more on just identifying the adjacency matrix. But, active causal discovery does have a nice ring to it.\n\n1. **Duplicated references**. There are references with multiple but distinct entries.\n\n1. **Licence** is a bit arbitrary as a name.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "For reviewer yEiY: \n\nThanks for your detailed comments.\n\n> Reviews: As an example, \"which is fundamental for many real-world applications, ranging from health caring, education, to drug discovery, and protein synthesis,\" is a little awkward and could be improved as, \"which is fundamental for many real-world applications, including health care, education, drug discovery, and protein synthesis.\" Another example would be that \"In specific\" can be replaced by \"Specifically\" or \"In particular.\" Another would be starting a sentence with a citation should use the \\citet{} function rather than \\cite{} or \\citep{}, such as on line 34. There are other examples and I would suggest using a tool like Grammarly. I do not find this to be a deal breaker, though, and some sections are written very well (e.g., 2.1).\n\n\nThanks for pointing out the writing problems, In the final version, we will carefully revise them according to your comments.\n\n\n\n> Reviews: Motivating Example. The example of a high-fidelity experiment as an actual patient outcome vs. a low-fidelity experiment as a simulated patient outcome is a little weak. At least need to explain why the simulator could be lower-fidelity. If the simulator has domain knowledge, why not incorporate that knowledge into the causal model rather than try to infer that knowledge? Is it a black box? As alternatives, maybe consider spacial or temporally measured outcomes as examples of low vs. high-fidelity. For example, perhaps a lower spatial or temporal resolution measurement vs. a higher spatial or temporal measurement? A satellite image at 10-mile resolution vs. 1-mile resolution. An MRI from a 1T machine vs. a 7T machine? Maybe alternatively, a radiologist's assessment of malignancy vs. a biopsy result?\n\nThanks for this comments. In the patient example, we believe the simulator can be less accurate, since the models used for simulation can be not perfect, it may contain different approximation errors. However, the simulation method may cost little, since we do not have to make experiments on the real people. \n\nWe believe the motivating example of spacial or temporally measured outcomes is also very interesting, we will definitely add it into our final paper.\n\n\n\n> Reviews: line 91. covariance -> variance. You use f to define the SCM functions and the acquisistion function, which is a little confusing.\n\nIn the final version, we will revise these inappropriate aspects accordingly. \n\n\n\n> Reviews: Active Causal Discovery. Since you work on identifying the adjacency matrix and the SEM parameters, I'd use causal experimental design. Only because causal discovery has focused more on just identifying the adjacency matrix. But, active causal discovery does have a nice ring to it. Duplicated references. There are references with multiple but distinct entries. Licence is a bit arbitrary as a name.\n\nThanks for this comment.  We will change ACD to causal experimental design in the final version. For the references, will double check them to make the paper more clear. For the model name Licence, we will change it in the final version.\n\n\n\n> Reviews: What issues would arise if the cost and the mutual information are not on similar scales?\n\nFor the same oracle, we just need to obtain the ranking of the mutual information for different intervention variable and value pairs. The absolute value may be not that important.\nActually, in our experiments, we find that the mutual information term is always in [0.05, 0.2], which does not differ too much from the cost.\n\n\n\n> Reviews: Can you know the scale of the mutual information? Is your mutual information estimator reliable?\n\nThe scale of the mutual information term is always in [0.05, 0.2]. We can not ensure that the estimator is completely correct. The reliability of the estimator comes from the Law of Large Numbers."}, {"review_id": "sDhWCLtc4j", "reviewer": "Reviewer_ug4Y", "summary": "The authors introduce the task of active causal discovery with multi-fidelity oracles which they go onto show is superior to many state-of-the-art methods for active causal discovery. They demonstrate their method in multiple settings and compare it alongside the aforementioned methods, to show superior performance almost across the board.", "questions": "## Abstract\n\n- A very good abstract which I enjoyed reading but it is too long, much too long. The abstract is merely meant to summarise what you are doing in the paper, and pick out some main results and then leave the rest for the paper. I suggest about half the length of what you currently have.\n\n## Introduction\n\n- Typo: Markov equivalence class (not ‘equivalent’) - line 26.\n- The paragraph starting on line 37 is excellent, very interesting. I would be interested to know if the authors have more information on this particular topic discussed in the paragraph i.e. do clinicians actually take these considerations intom account? References to that effect would be very informative.\n- Do you know if there are more sources on multi-objective (multi-target), line 54, studies which you mention? This is an open problem for much of causal inference as you rightly know. It would be effectual to add such sources if you have them.\n\n## Preliminary\n\n- Typo / language: the section tile should be ‘Preliminaries’ as it is plural as you are describing multiple ideas here.\n- Reference Pearl when you first introduce the SCM on line 78. It was his idea after all and he deserves credit whenever it is raised in academic context.\n- You will need to give more information on the SCM. It is a strict definition which consists of a four-tuple. See Def 7.1.1 of Pearl’s book on causality. At present you are not giving the correct definition as there are pieces missing w.r.t. your description of the SCM.\n- Your language for describing the adjacency matrix is a bit confusing. If i and j are denoted by a 1 in the matrix that means that there is an arc from i to j in the causal diagram or that there is an edge which exists i and points to j. Phrase it in this relation instead of saying a ‘causal relation’ as you have done, as you are fundamentally describing a DAG.\n- Line 102: you cannot say ‘using causal language’ to define the do-operator because the do-operator is not unique for modelling interventions, there are many other frameworks such as the potential outcome framework. Please revise and provide adequate references.\n- You have not discussed identifiability in section 2.2 which seems somewhat important because at present you are assuming that all your interventional distributions can be calculated from the observational distribution which is not always the case.\n\n## The license model\n\n- Explain what $H$ is in equation 2.\n- Figure 1 does not make sense. What does the dashed arrow represent? Why does it point to another arrow? You are going to have to redo that figure with much more detail and clarification because this reviewer cannot make heads or tails of it. It looks like a first-order Markov model but I am sure that that has nothing to do with the problem you are studying but equally I do not quite understand what you are trying to show with that figure.\n\n## Experiments\n\n- Line 276: you cannot say “state-of-arts”. That phrasing is used as so: ‘the state-of-the-art model’ - it is an adjective that refers to a noun, in this case ‘model’. \n- This section could be improved by adding the experimental topologies that are being targeted by the different causal discovery methods, to give the reader an idea of the complexity involved.", "limitations": "Please see the Questions section for the main review. I would be happy to increase my score but my concerns above would have to adequately addressed in order to do so. This is not nitpicking (which it surely will be construed as from the authors) but the paper is genuinely very difficult to follow for the reasons listed but could be much improved with some simple proof-reading and revisions.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "Please see the Questions section for the main review.", "weaknesses": "Please see the Questions section for the main review.\n\n- The paper is very hard to follow for a number of reasons. First, the language is lacking in a lot of places and complex sentence construction makes following the arguments presented, somewhat harder still (this sentence for example on line 256: \"Higher-fidelity models are more accuracy but cost much\" should presumably be 'higher-fidelity models are more accurate but they cost more'). To help the authors I recommend Grammarly which is a great free tool that can pickup many of the language mistakes that are present in the manuscript. Second, the authors jump a lot in their narrative thus making quite difficult to understand the point they are trying to make. The manuscript would do well to be proof-read a few more times by different people. Most all the manuscript lacks a red thread, taking the reader through the paper one coherent argument at time, thus building a narrative as it goes along. At the end of section three, I still do not quite understand where the authors are heading with their method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "For reviewer ug4Y:\n\nThanks so much for your detailed comments. We will try our best to alleviate your concerns in the following.\n\n> Reviews: **Abstract**\n>\n> A very good abstract which I enjoyed reading but it is too long, much too long. The abstract is merely meant to summarise what you are doing in the paper, and pick out some main results and then leave the rest for the paper. I suggest about half the length of what you currently have.\n\nThanks for the comments. According to your comments, we plan to change our abstract by removing the discussions on the differences between single- and multi-fidelity oracles. In addition, we will also remove the challenges that we may face for multi-fidelity ACD. We believe the removed contents can be cleared delivered in the introduction.\n\n> Concerns on the **Introduction**\n\n1. For the typo \"Markov equivalence class\": we will correct it in the final version.\n\n2. For the topic discussed in the paragraph starting on line 37: we believe this example is very practical, and there are indeed many studies on simulation based drug-disease relation discovery. For example, [1-9] are all about this topic. We will add these references into the final paper.\n\n3. For the termination of \"multi-objective (multi-target)\": there are indeed many papers on multi-target intervention, for example, [10-12]. We will add them into the final version.\n\n> Concerns on the **Preliminary**\n\n1. For the typo and language problems: we will revise them carefully according to your suggestions.\n\n2. For the reference of Pearl: we are sorry for this point. In the final version, we will definitely cite Pearl's paper.\n\n3. For the definition of SCM: in the final version, we will present the correct definition according to Def 7.1.1 of Pearl’s book on causality.\n\n4. For the adjacency matrix: thanks so much for your correction, in the final version, we will change \"causal relation from $X_i$ to $X_j$\" to \"if $E_{ij} =1$, then there is an edge which exists from $i$ to $j$\".\n\n5. For the do-operator: we will remove the term \"using causal language\", and add Pearl's papers as references for the do-operator.\n\n6. For the identifiability problem, we will add more discussions on the assumptions usually leveraged to derive identifiability.\n\n\n\n> Concerns on the **The license model**\n\n\n1. For $H$: it is the notation of entropy, that is, $H(x) = E_x[log x]$.\n\n2. For Figure 1: the dashed arrow means that $x$ is determined by $\\phi$ and $(j,v)$ jointly. It should point to $x$. In the additional submitted one-page pdf, we have replotted this figure according to your comments in **Figure 1**.\n\n\n\n> Concerns on the **Experiments**\n\nThanks for these comments. \n\n1. For the term \"state-of-arts\": we will revise it in the final version.\n\n2. For the topologies: in the submitted one-page pdf, we have added the structures of the real causal graph in **Figure 4**.\n\n\n\n**References**\n\n[1] Li, J., & Lu, Z. (2013). Pathway-based drug repositioning using causal inference. BMC bioinformatics, 14(16), 1-10.\n\n[2] Yang, J., Li, Z., Fan, X., & Cheng, Y. (2014). Drug–disease association and drug-repositioning predictions in complex diseases using causal inference–probabilistic matrix factorization. Journal of chemical information and modeling, 54(9), 2562-2569.\n\n[3] Peyvandipour, A., Saberian, N., Shafi, A., Donato, M., & Draghici, S. (2018). A novel computational approach for drug repurposing using systems biology. Bioinformatics, 34(16), 2817-2825.\n\n[4] Kelly, J., Berzuini, C., Keavney, B., Tomaszewski, M., & Guo, H. (2022). A review of causal discovery methods for molecular network analysis. Molecular Genetics & Genomic Medicine, 10(10), e2055.\n\n[5] Domingo-Fernández, D., Gadiya, Y., Patel, A., Mubeen, S., Rivas-Barragan, D., Diana, C. W., ... & Colluru, V. (2022). Causal reasoning over knowledge graphs leveraging drug-perturbed and disease-specific transcriptomic signatures for drug discovery. PLoS computational biology, 18(2), e1009909.\n\n[6] Chen, H., Zhang, Z., & Peng, W. (2017). miRDDCR: a miRNA-based method to comprehensively infer drug-disease causal relationships. Scientific reports, 7(1), 15921.\n\n[7] Subpaiboonkit, S., Li, X., Zhao, X., Scells, H., & Zuccon, G. (2019, November). Causality discovery with domain knowledge for drug-drug interactions discovery. In International Conference on Advanced Data Mining and Applications (pp. 632-647). Cham: Springer International Publishing.\n\n[8] Subpaiboonkit, S., Li, X., Zhao, X., & Zuccon, G. (2022, November). Causality Discovery Based on Combined Causes and Multiple Causes in Drug-Drug Interaction. In International Conference on Advanced Data Mining and Applications (pp. 53-66). Cham: Springer Nature Switzerland.\n\n[9] Tian, X. Y., & Liu, L. (2012). Drug discovery enters a new era with multi-target intervention strategy. *Chinese journal of integrative medicine*, *18*(7), 539-542.\n\n[10] Tigas, P., Annadani, Y., Ivanova, D. R., Jesson, A., Gal, Y., Foster, A., & Bauer, S. (2023, May). Differentiable Multi-Target Causal Bayesian Experimental Design. In *ICLR 2023-Machine Learning for Drug Discovery workshop*.\n\n[11] Guerrero, L. R., Ho, J., Christie, C., Harwood, E., Pfund, C., Seeman, T., ... & Wallace, S. P. (2017, December). Using collaborative approaches with a multi-method, multi-site, multi-target intervention: evaluating the National Research Mentoring Network. In *BMC proceedings* (Vol. 11, No. 12, pp. 193-200). BioMed Central.\n\n[12] Somvanshi, R. K., Zou, S., Kadhim, S., Padania, S., Hsu, E., & Kumar, U. (2022). Cannabinol modulates neuroprotection and intraocular pressure: A potential multi-target therapeutic intervention for glaucoma. *Biochimica et Biophysica Acta (BBA)-Molecular Basis of Disease*, *1868*(3), 166325."}]

Author rebuttal:
Dear reviewers:

   Thanks for your detailed reviews.  Additional tables and figures that mentioned in rebuttals are shown in the submitted one-page pdf. 

Meta-review: The authors consider the problem of causal discovery when the experiments may be performed at multiple fidelities. The reviewers agree that the problem is important and the solution proposed would be valuable to the NeurIPS community. The paper was also improved through the rebuttal process and the authors are strongly encouraged to incorporate this feedback into the final version.
Decision: Accept (poster)