SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $f_{j,m}(v)$  is rewritten from  $f(j,v,m)$  under given  $j,m$ . Therefore, our task is calculating  $v^{*}(j,m)$  for  $\forall j\in [d],m\in [M]$  with Bayesian optimization [47]. We utilize a Gaussian Process (GP) [48] to model surrogate function distributions for each  $v^{*}(j,m)$ . We denote  $f\sim \mathcal{GP}(\mathbf{0},\mathcal{K}(v_i,v_j))$ , and  $\mathcal{K}(v_i,v_j)$  is the kernel of GP. We sequentially find  $v_{t}$  and calculate  $f_{j,m}(v_t)$  to direct the process. According to GP, the previous  $t$  functions and the  $t + 1$  function are multivariate Gaussian distribution,
$$
\left[ \begin{array}{c} \mathbf {F} _ {1: t} \\ f _ {t + 1} \end{array} \right] \sim \mathcal {N} \left(\mathbf {0}, \left[ \begin{array}{c c} \mathbf {K} _ {t} & \boldsymbol {k} _ {t + 1} \\ \boldsymbol {k} _ {t + 1} ^ {T} & \mathcal {K} (v _ {t + 1}, v _ {t + 1}) \end{array} \right]\right),
$$
$$
\mathbf {F} _ {1: t} = \left[ f _ {1}, f _ {2}, \dots , f _ {t} \right],
$$
$$
\boldsymbol {k} _ {t + 1} = \left[ \mathcal {K} \left(v _ {t + 1}, v _ {1}\right), \mathcal {K} \left(v _ {t + 1}, v _ {2}\right), \dots , \mathcal {K} \left(v _ {t + 1}, v _ {t + 1}\right) \right] ^ {T},
$$
$$
\mathbf {K} _ {t} = \left[ \begin{array}{c c c} \mathcal {K} \left(v _ {1}, v _ {1}\right) & \dots & \mathcal {K} \left(v _ {t}, v _ {1}\right) \\ \vdots & \ddots & \vdots \\ \mathcal {K} \left(v _ {t}, v _ {1}\right) & \dots & \mathcal {K} \left(v _ {t}, v _ {t}\right) \end{array} \right]. \tag {11}
$$
Given previous  $t$  steps, we have the posterior probability is
$$
p \left(f _ {t + 1} \mid \left\{\left(v _ {i}, f _ {j, m} \left(v _ {i}\right)\right) \right\} _ {i = 1} ^ {t}, v _ {t + 1}\right) = \mathcal {N} \left(\mu_ {t} \left(v _ {t + 1}\right), \sigma_ {t} ^ {2} \left(v _ {t + 1}\right)\right),
$$
with the non-parametric means and variances
$$
\mu_ {t} \left(v _ {t + 1}\right) = \boldsymbol {k} _ {t + 1} ^ {T} \left(\mathbf {K} + \mathbf {I}\right) ^ {- 1} \mathbf {F} _ {1: t}, \tag {12}
$$
$$
\sigma_ {t} ^ {2} \left(v _ {t + 1}\right) = \mathcal {K} \left(v _ {v + 1}, v _ {t + 1}\right) - \boldsymbol {k} _ {t + 1} ^ {T} (\mathbf {K} + \mathbf {I}) ^ {- 1} \boldsymbol {k} _ {t + 1}. \tag {13}
$$
We acquire the next  $v_{t+1}$  with GP-UCB [49] function
$$
a _ {t + 1} (v) = \mu_ {t} (v) + \beta_ {a c} \cdot \sqrt {\sigma_ {t} ^ {2} (v)},
$$
$$
v _ {t + 1} = \underset {v} {\arg \max } a _ {t + 1} (v).
$$
where  $\beta_{ac}$  is a hyper-parameter. Suppose the maximum of steps is  $T$ , the final output of function  $v^{*}(j,m)$  is
$$
v ^ {*} (j, m) = \underset {v} {\arg \max } \mu_ {T} (v).
$$
Then we choose the best interventional node  $j$  and fidelity  $m$  by their best values under  $\mathcal{O}(d\cdot M)$
$$
j ^ {*}, m ^ {*} = \underset {j, m} {\arg \max } v ^ {*} (j, m),
$$
$$
v ^ {*} = v ^ {*} (j ^ {*}, m ^ {*}).
$$
C Detailed Training Process of ELBO
C.1 Derivation Process of ELBO
Because we use the distribution  $q(\phi_m)$  to approximate the distribution  $p(\phi_m)$ , then we intend to minimize the distance between these two distributions. To optimize the parameters of  $q(\phi_m)$ , where we utilize KL divergence to measure the distance, that is,
$$
\Psi^ {*} = \underset {\Psi} {\arg \min } \mathrm {K L} [ q (\Phi | | p (\Phi | D) ].
$$
According to the variational inference, we have
$$
\begin{array}{l} \operatorname {K L} [ q (\Phi) | | p (\Phi | D) ] \\ = \int q (\Phi) \log \frac {q (\Phi)}{p (\Phi | D)} d \Phi \\ = \int q (\boldsymbol {\Phi}) \log q (\boldsymbol {\Phi}) d \boldsymbol {\Phi} - \int q (\boldsymbol {\Phi}) \log p (\boldsymbol {\Phi} | D) d \boldsymbol {\Phi} \\ = \mathbb {E} _ {\Phi \sim q (\Phi)} [ \log q (\Phi) ] - \int q (\Phi) \log \frac {p (\Phi , D)}{p (D)} d \Phi \\ = \mathbb {E} _ {\Phi \sim q (\Phi)} \left[ \log q (\Phi) \right] - \int q (\Phi) \log p (\Phi , D) d \Phi + \int q (\Phi) \log p (D) d \Phi \\ = \mathbb {E} _ {\boldsymbol {\Phi} \sim q (\boldsymbol {\Phi})} [ \log q (\boldsymbol {\Phi}) ] - \mathbb {E} _ {\boldsymbol {\Phi} \sim q (\boldsymbol {\Phi})} [ \log p (\boldsymbol {\Phi}, D) ] + \int q (\boldsymbol {\Phi}) \log p (D) d \boldsymbol {\Phi} \\ = \underbrace {\mathbb {E} _ {\Phi \sim q (\Phi)} [ \log q (\Phi) ] - \mathbb {E} _ {\Phi \sim q (\Phi)} [ \log p (\Phi , D) ]} _ {- \mathrm {E L B O}} + \log p (D). \\ \end{array}
$$
Because  $\log p(D)$  is not related to  $\Psi$ , minimizing KL  $[q(\Phi)||p(\Phi|D)]$  is equivalent to maximizing the ELBO part, and we have
$$
\begin{array}{l} \operatorname {E L B O} = \mathbb {E} _ {\boldsymbol {\Phi} \sim q (\boldsymbol {\Phi})} [ \log p (\boldsymbol {\Phi}, D) ] - \mathbb {E} _ {\boldsymbol {\Phi} \sim q (\boldsymbol {\Phi})} [ \log q (\boldsymbol {\Phi}) ] \\ = \mathbb {E} _ {\boldsymbol {\Phi} \sim q (\boldsymbol {\Phi})} [ \log p (D | \boldsymbol {\Phi}) ] + \mathbb {E} _ {\boldsymbol {\Phi} \sim q (\boldsymbol {\Phi})} [ \log p (\boldsymbol {\Phi}) ] - \mathbb {E} _ {\boldsymbol {\Phi} \sim q (\boldsymbol {\Phi})} [ \log q (\boldsymbol {\Phi}) ] \\ = \mathbb {E} _ {\boldsymbol {\Phi} \sim q (\boldsymbol {\Phi})} [ \log p (D | \boldsymbol {\Phi}) - \log q (\boldsymbol {\Phi}) + \log p (\boldsymbol {\Phi}) ] \\ \end{array}
$$
Above all, we can conclude that
$$
\Psi^ {*} = \underset {\Psi} {\arg \min } \mathrm {K L} [ q (\Phi | | p (\Phi | D) ]
$$
is equivalent to maximize evidence lower bound
$$
\begin{array}{l} \Psi^{*} = \operatorname *{arg  max}_{\Psi}\mathrm{ELBO} \\ = \operatorname * {a r g   m a x} _ {\Psi} \mathbb {E} _ {\Phi \sim q (\Phi)} \left[ \log p (D | \Phi) - \log q (\Phi) + \log p (\Phi) \right]. \\ \end{array}
$$
C.2 Estimation of ELBO
We represent the equation of ELBO as
$$
\begin{array}{l} \operatorname {E L B O} = \mathbb {E} _ {\Phi \sim q (\Phi)} [ \log p (D | \Phi) - \log q (\Phi) + \log p (\Phi) ] \\ = \underbrace {\mathbb {E} _ {\boldsymbol {\Phi} \sim q (\boldsymbol {\Phi})} [ \log p (D | \boldsymbol {\Phi}) ]} _ {A} - \underbrace {\mathbb {E} _ {\boldsymbol {\Phi} \sim q (\boldsymbol {\Phi})} [ \log q (\boldsymbol {\Phi}) - \log p (\boldsymbol {\Phi}) ]} _ {B}. \\ \end{array}
$$
For the part  $A$ , we have
$$
A = \mathbb {E} _ {\boldsymbol {\Phi} \sim q (\boldsymbol {\Phi})} \left[ \log \prod_ {i = 1} ^ {N} p (\boldsymbol {x} ^ {(i)} | j ^ {(i)}, v ^ {(i)}, m ^ {(i)}, \boldsymbol {\Phi}) \right],
$$
where  $N$  is the current number of samples in buffer. Then we have
$$
\begin{array}{l} A = \mathbb {E} _ {\Phi \sim q (\Phi)} \left[ \log \prod_ {i = 1} ^ {N} p (\boldsymbol {x} ^ {(i)} | j ^ {(i)}, v ^ {(i)}, m ^ {(i)}, \Phi) \right] \\ = \mathbb {E} _ {\Phi \sim q (\Phi)} \left[ \sum_ {i = 1} ^ {N} \log p \left(\boldsymbol {x} ^ {(i)} \mid j ^ {(i)}, v ^ {(i)}, m ^ {(i)}, \Phi\right) \right] \\ = \sum_ {i = 1} ^ {N} \mathbb {E} _ {\Phi \sim q (\Phi)} \left[ \log p \left(\boldsymbol {x} ^ {(i)} \mid j ^ {(i)}, v ^ {(i)}, m ^ {(i)}, \Phi\right) \right] \\ = \sum_ {i = 1} ^ {N} \mathbb {E} _ {\boldsymbol {\phi} _ {m ^ {(i)}} \sim q (\boldsymbol {\phi} _ {m ^ {(i)}})} \left[ \log p (\boldsymbol {x} ^ {(i)} | j ^ {(i)}, v ^ {(i)}, m ^ {(i)}, \boldsymbol {\phi} _ {m ^ {(i)}}) \right]. \\ \end{array}
$$
Using Monte Carlo sampling [50], we can calculate the expectation by  $N_{S}$  samples for each point.
$$
A = \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {N _ {S}} \log p (\boldsymbol {x} ^ {(i)} | j ^ {(i)}, v ^ {(i)}, m ^ {(i)}, \phi_ {m ^ {(i)}} ^ {(j)}),
$$
where we sample  $\phi_{m^{(i)}}^{(j)}\sim q(\phi_{m^{(i)}})$  with size  $N_{S}$ .
Then we denote the distribution  $q(\Phi) = \mathcal{N}(\tilde{\mu}_{all},\tilde{\Sigma}_{all})$ , and similarly, we have  $p(\Phi) = \prod_{m=1}^{M} e^{-\beta \cdot f(\mathbf{S}_m,\mathbf{T}_m)} \cdot \mathcal{N}(\boldsymbol{\mu}_{all},\boldsymbol{\Sigma}_{all})$ . Both the parameter  $\tilde{\mu}_{all},\tilde{\Sigma}_{all}$  can be represented by the parameters in  $\Psi$ , while  $\boldsymbol{\mu}_{all}$  and  $\boldsymbol{\Sigma}_{all}$  are constant. Then we calculate part  $B$
$$
\begin{array}{l} B = \mathbb {E} _ {\Phi \sim q (\Phi)} [ \log q (\Phi) - \log p (\Phi) ] \\ = \int_ {\Phi} \mathcal {N} \left(\tilde {\boldsymbol {\mu}} _ {a l l}, \tilde {\boldsymbol {\Sigma}} _ {a l l}\right) \log \frac {\mathcal {N} \left(\tilde {\boldsymbol {\mu}} _ {a l l} , \tilde {\boldsymbol {\Sigma}} _ {a l l}\right)}{\prod_ {m = 1} ^ {M} e ^ {- \beta \cdot f \left(\mathbf {S} _ {m} , \mathbf {T} _ {m}\right)} \cdot \mathcal {N} \left(\boldsymbol {\mu} _ {a l l} , \boldsymbol {\Sigma} _ {a l l}\right)} d \Phi \\ = \int_ {\Phi} \mathcal {N} (\tilde {\boldsymbol {\mu}} _ {a l l}, \tilde {\boldsymbol {\Sigma}} _ {a l l}) \log \frac {\mathcal {N} (\tilde {\boldsymbol {\mu}} _ {a l l} , \tilde {\boldsymbol {\Sigma}} _ {a l l})}{\mathcal {N} (\boldsymbol {\mu} _ {a l l} , \boldsymbol {\Sigma} _ {a l l})} d \Phi + \int_ {\Phi} \mathcal {N} (\tilde {\boldsymbol {\mu}} _ {a l l}, \tilde {\boldsymbol {\Sigma}} _ {a l l}) \log \frac {1}{\prod_ {m = 1} ^ {M} e ^ {- \beta \cdot f (\mathbf {S} _ {m} , \mathbf {T} _ {m})}} d \Phi \\ = \underbrace {\operatorname {K L} [ \mathcal {N} (\tilde {\boldsymbol {\mu}} _ {a l l} , \tilde {\boldsymbol {\Sigma}} _ {a l l}) | | \mathcal {N} (\boldsymbol {\mu} _ {a l l} , \boldsymbol {\Sigma} _ {a l l}) ]} _ {C} + \underbrace {\int_ {\boldsymbol {\Phi}} \mathcal {N} (\tilde {\boldsymbol {\mu}} _ {a l l} , \tilde {\boldsymbol {\Sigma}} _ {a l l}) \log \prod_ {m = 1} ^ {M} e ^ {\beta \cdot f (\mathbf {S} _ {m} , \mathbf {T} _ {m})} d \boldsymbol {\Phi}} _ {D}. \\ \end{array}
$$
According to KL divergence of Gaussian distribution, we can calculate  $C$  in a close-form.
$$
\begin{array}{l} C = \mathrm {K L} [ \mathcal {N} (\tilde {\boldsymbol {\mu}} _ {a l l}, \bar {\boldsymbol {\Sigma}} _ {a l l}) | | \mathcal {N} (\boldsymbol {\mu} _ {a l l}, \boldsymbol {\Sigma} _ {a l l}) ] \\ = \frac {1}{2} \left[ \log \frac {| | \boldsymbol {\Sigma} _ {a l l} | |}{| | \tilde {\boldsymbol {\Sigma}} _ {a l l} | |} - d + \operatorname {t r} \left(\boldsymbol {\Sigma} _ {a l l} ^ {- 1} \tilde {\boldsymbol {\Sigma}} _ {a l l}\right) + \left(\tilde {\boldsymbol {\mu}} _ {a l l} - \boldsymbol {\mu} _ {a l l}\right) ^ {T} \boldsymbol {\Sigma} _ {a l l} ^ {- 1} \left(\tilde {\boldsymbol {\mu}} _ {a l l} - \boldsymbol {\mu} _ {a l l}\right) \right]. \\ \end{array}
$$
Then we calculate  $D$  by the following steps:
$$
\begin{array}{l} D = \int_ {\boldsymbol {\Phi}} \mathcal {N} (\boldsymbol {\mu} _ {a l l}, \boldsymbol {\Sigma} _ {a l l}) \log \prod_ {m = 1} ^ {M} e ^ {\beta \cdot f (\mathbf {S} _ {m}, \mathbf {T} _ {m})} d \boldsymbol {\Phi} \\ = \int_ {\Phi} \mathcal {N} \left(\boldsymbol {\mu} _ {a l l}, \boldsymbol {\Sigma} _ {a l l}\right) \sum_ {m = 1} ^ {M} \log e ^ {\beta \cdot f \left(\mathbf {S} _ {m}, \mathbf {T} _ {m}\right)} d \boldsymbol {\Phi} \\ = \cdot \int_ {\Phi} \mathcal {N} \left(\boldsymbol {\mu} _ {a l l}, \boldsymbol {\Sigma} _ {a l l}\right) \sum_ {m = 1} ^ {M} \beta \cdot f \left(\mathbf {S} _ {m}, \mathbf {T} _ {m}\right) d \boldsymbol {\Phi} \\ = \beta \cdot \mathbb {E} _ {\boldsymbol {\Phi} \sim \mathcal {N} (\boldsymbol {\mu} _ {a l l}, \boldsymbol {\Sigma} _ {a l l})} \left[ \sum_ {m = 1} ^ {M} f (\mathbf {S} _ {m}, \mathbf {T} _ {m}) \right]. \\ \end{array}
$$
Using Monte Carlo sampling, we can calculate the expectation by  $N_{D}$  samples for each point.
$$
\begin{array}{l} D = \beta \cdot \sum_ {i = 1} ^ {N _ {D}} \sum_ {m = 1} ^ {M} f \left(\mathbf {S} _ {m} ^ {(i)}, \mathbf {T} _ {m} ^ {(i)}\right). \\ = \beta \cdot \sum_ {i = 1} ^ {N _ {D}} \sum_ {m = 1} ^ {M} \mathbb {E} _ {p (\mathbf {E} | \mathbf {S} _ {m} ^ {(i)}, \mathbf {T} _ {m} ^ {(i)})} \left[ \lambda_ {1} \cdot \left[ \mathrm {t r} \left(e ^ {\mathbf {E}}\right) - d \right] + \lambda_ {2} \cdot | | \mathbf {E} | | \right], \\ \end{array}
$$
where we samples  $\Phi^{(i)}\sim \mathcal{N}(\pmb{\mu}_{all},\pmb{\Sigma}_{all})$  with size  $N_{D}$ . Using Monte Carlo sampling again, we can calculate the expectation by  $N_{E}$  samples.
$$
D = \beta \cdot \sum_ {i = 1} ^ {N _ {D}} \sum_ {m = 1} ^ {M} \sum_ {j = 1} ^ {N _ {E}} \left[ \lambda_ {1} \cdot \left[ \operatorname {t r} \left(e ^ {\mathbf {E}}\right) - d \right] + \lambda_ {2} \cdot | | \mathbf {E} | | \right],
$$
where we samples  $\mathbf{E}^{(j)}\sim p(\mathbf{E}|\mathbf{S}_m^{(i)},\mathbf{T}_m^{(i)})$  with size  $N_{E}$
Finally, we obtain the estimation
$$
\begin{array}{l} \mathrm {E L B O} = \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {N _ {S}} \log p (x ^ {(i)} | j ^ {(i)}, v ^ {(i)}, m ^ {(i)}, \phi_ {m ^ {(i)}} ^ {(j)}) \\ - \frac {1}{2} \left[ \log \frac {| | \boldsymbol {\Sigma} _ {a l l} | |}{| | \tilde {\boldsymbol {\Sigma}} _ {a l l} | |} - d + \operatorname {t r} (\boldsymbol {\Sigma} _ {a l l} ^ {- 1} \tilde {\boldsymbol {\Sigma}} _ {a l l}) + (\tilde {\boldsymbol {\mu}} _ {a l l} - \boldsymbol {\mu} _ {a l l}) ^ {T} \boldsymbol {\Sigma} _ {a l l} ^ {- 1} (\tilde {\boldsymbol {\mu}} _ {a l l} - \boldsymbol {\mu} _ {a l l}) \right] \\ - \beta \cdot \sum_ {i = 1} ^ {N _ {D}} \sum_ {m = 1} ^ {M} \sum_ {j = 1} ^ {N _ {E}} \left[ \lambda_ {1} \cdot \left[ \operatorname {t r} \left(e ^ {\mathbf {E}}\right) - d \right] + \lambda_ {2} \cdot | | \mathbf {E} | | \right]. \\ \end{array}
$$
C.3 Gaussian Reparameterization Trick
In the last section, we derive the objection function for optimizing the model parameters, where we can use methods of the gradient descent to solve it. However, a significant problem rises due to the sampling process, because the gradient of model parameters can not pass backward from the naive sampling process (i.e., untraceable). Therefore, we use Gaussian reparameterization trick to make the Gaussian sampling process traceable.
In specific, we will demonstrate the traceable calculation of  $\phi$  by Gaussian reparameterization trick. In order to sample  $\phi \sim \mathcal{N}(\pmb{\mu}, \pmb{\Sigma})$ , we first sample  $\delta \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$  instead, and then obtain  $\phi = \pmb{\mu} + \delta \odot \pmb{\Sigma}$ . Therefore, the gradient can be traced from  $\phi$  to  $\pmb{\mu}$  and  $\pmb{\Sigma}$ . In specific, both  $\pmb{\mu}$  and  $\pmb{\Sigma}$  can be represented with the function of learnable parameter  $\Psi$ .
C.4 Gumbel-softmax Reparameterization Trick
Besides of the Gaussian sampling process, the Bernoulli sampling in our equation is not traceable either, so we utilize Gumbel-softmax reparameterization trick to make it traceable.
We demonstrate the traceable calculation of  $\mathbf{E} \sim p(\mathbf{E}|\mathbf{S},\mathbf{T})$  by Gumbel-max reparameterization trick. According to Gumbel-max [51], we have
$$
\operatorname {B e r n o u l l i} (p) \iff \mathbf {1} \left[ G _ {1} + \log p > G _ {0} + \log (1 - p) \right], \quad G _ {0}, G _ {1} \sim \operatorname {G u m b e l} (0, 1).
$$
Instead of using unit step function, we utilize sigmoid function
$$
\sigma \left(G _ {1} + \log p > G _ {0} + \log (1 - p)\right).
$$
$$
\mathbf {E} _ {i, j} = \sigma (\mathbf {L} _ {i, j} + \mathbf {S} _ {i} ^ {T} \cdot \mathbf {T} _ {j}),
$$
where  $\mathbf{L}_{i,j} \sim L(0,1)$ . Therefore, we sample  $\mathbf{L}_{i,j} \sim L(0,1)$  instead, where  $L(0,1)$  is logistic distribution, and calculate  $\mathbf{E}_{i,j} = \sigma (\mathbf{L}_{i,j} + \mathbf{S}_i^T\cdot \mathbf{T}_j)$  to trace gradients. Specifically, both  $\mathbf{S}_i$  and  $\mathbf{T}_i$  can be represented with the function of learnable parameter  $\Psi$ .
C.5 Optimization of ELBO
With the estimation and reparameterization trick, we are able to conduct gradient descent methods to optimize our parameters with the objection function
$$
\Psi^ {*} = \underset {\Psi} {\arg \max } \text {E L B O}.
$$
The format of stochastic gradient descent (SGD) is
$$
\Psi \leftarrow \Psi + \gamma \cdot \frac {\partial \text {E L B O}}{\partial \Psi},
$$
where  $\gamma$  is the learning rate.
D Training Process of Constraint based ELBO
We intend to optimize our parameter with
$$
\begin{array}{l} \Psi^ {*} = \underset {\Psi} {\arg \max} \mathbb {E} _ {\Phi \sim q (\Phi)} \left[ \log p (D | \Phi) - \log q (\Phi) + \log p (\Phi) \right], \\ \text {s . t .} \sum_ {\left\{e _ {s}, e _ {t} \right\}} I \left(\boldsymbol {x} _ {s}; \boldsymbol {x} _ {t} \mid \phi_ {M}, \left\{\boldsymbol {e} _ {s}, \boldsymbol {e} _ {t} \right\}, D\right) \leq \epsilon . \\ \end{array}
$$
However, the objection has a constraint, which is hard to optimize with gradient descent methods. So we utilize Lagrange multiplier [52] to convert it to a constraint-free method:
$$
\Psi^{*} = \operatorname *{arg  max}_{\Psi}\mathbb{E}_{\Phi \sim q(\Phi)}\left[\log p(D|\Phi) - \log q(\Phi) + \log p(\Phi)\right] + \lambda \cdot \sum_{\{\boldsymbol {e}_{s},\boldsymbol{e}_{t}\}}I(\boldsymbol{x}_{s};\boldsymbol{x}_{t}|\phi_{M},\{\boldsymbol{e}_{s},\boldsymbol{e}_{t}\} ,D),
$$
where  $\lambda$  is the Lagrange multiplier. Then, we intend to calculate the constraint part.
First of all, we have
$$
\begin{array}{l} I \left(\boldsymbol {x} _ {s}; \boldsymbol {x} _ {t} \mid \phi_ {M}, \left\{\boldsymbol {e} _ {s}, \boldsymbol {e} _ {t} \right\}, D\right) \\ = H \left(\boldsymbol {x} _ {s} \mid \phi_ {M}, \left\{\boldsymbol {e} _ {s}, \boldsymbol {e} _ {t} \right\}, D\right) + H \left(\boldsymbol {x} _ {t} \mid \phi_ {M}, \left\{\boldsymbol {e} _ {s}, \boldsymbol {e} _ {t} \right\}, D\right) - H \left(\boldsymbol {x} _ {s}, \boldsymbol {x} _ {t} \mid \phi_ {M}, \left\{\boldsymbol {e} _ {s}, \boldsymbol {e} _ {t} \right\}, D\right) \\ = H \left(\boldsymbol {x} _ {s} \mid \phi_ {M}, \boldsymbol {e} _ {s}, D\right) + H \left(\boldsymbol {x} _ {t} \mid \phi_ {M}, \boldsymbol {e} _ {t}, D\right) - H \left(\boldsymbol {x} _ {s}, \boldsymbol {x} _ {t} \mid \phi_ {M}, \left\{\boldsymbol {e} _ {s}, \boldsymbol {e} _ {t} \right\}, D\right), \\ \end{array}
$$
For the term  $H(\pmb{x}_s|\phi_M, \pmb{e}_s, D)$ , we have
$$
\begin{array}{l} H \left(\boldsymbol {x} _ {s} \mid \phi_ {M}, \boldsymbol {e} _ {s}, D\right) = - \int p \left(\boldsymbol {x} _ {s} \mid \phi_ {M}, \boldsymbol {e} _ {s}, D\right) \log p \left(\boldsymbol {x} _ {s} \mid \phi_ {M}, \boldsymbol {e} _ {s}, D\right) d \boldsymbol {x} _ {s} \\ = - \mathbb {E} _ {p \left(\boldsymbol {x} _ {s} \mid \phi_ {M}, \boldsymbol {e} _ {s}, D\right)} \left[ \log p \left(\boldsymbol {x} _ {s} \mid \phi_ {M}, \boldsymbol {e} _ {s}, D\right) \right]. \\ \end{array}
$$
We use Monte Carlo sampling to estimate  $H(\pmb{x}_s|\phi_M, \pmb{e}_s, D)$ , and we have
$$
H (\boldsymbol {x} _ {s} | \phi_ {M}, \boldsymbol {e} _ {s}, D) \approx \frac {1}{K _ {1} \cdot K _ {2}} \sum_ {k _ {1} = 1} ^ {K _ {1}} \sum_ {k _ {2} = 1} ^ {K _ {2}} \log p (\boldsymbol {x} ^ {(k _ {1}, k _ {2})} | \boldsymbol {e} ^ {s}, \phi_ {M}),
$$
where we sample graphs  $\phi_m^{k_1} \sim q(\phi_m|e^s, \phi_M)$ , and obtain samples  $\mathbf{x}^{(k_1,k_2)} \sim p(\mathbf{x}|e^s, \phi_m^{k_1})$ . Similarly, we can calculate
$$
H (\boldsymbol {x} _ {t} | \phi_ {M}, \boldsymbol {e} _ {t}, D) \approx \frac {1}{K _ {1} \cdot K _ {2}} \sum_ {k _ {1} = 1} ^ {K _ {1}} \sum_ {k _ {2} = 1} ^ {K _ {2}} \log p (\boldsymbol {x} ^ {(k _ {1}, k _ {2})} | \boldsymbol {e} ^ {t}, \phi_ {M}),
$$
where we sample graphs  $\phi_m^{k_1} \sim q(\phi_m | e^t, \phi_M)$ , and obtain samples  $\pmb{x}^{(k_1, k_2)} \sim p(\pmb{x} | e^t, \phi_m^{k_1})$ .
$$
H (\pmb {x} _ {s}, \pmb {x} _ {t} | \phi_ {M}, \{\pmb {e} _ {s}, \pmb {e} _ {t} \}, D) \approx \frac {1}{K _ {1} \cdot K _ {2} \cdot K _ {3}} \sum_ {k _ {1} = 1} ^ {K _ {1}} \sum_ {k _ {2} ^ {1} = 1} ^ {K _ {2}} \sum_ {k _ {2} ^ {2} = 1} ^ {K _ {2}} \log p (\pmb {x} ^ {(k _ {1}, k _ {2} ^ {1})}, \pmb {x} ^ {(k _ {1}, k _ {2} ^ {2})} | \{\pmb {e} _ {s}, \pmb {e} _ {t} \}, \phi_ {M}),
$$
where we sample graphs  $\phi_{m}^{k_{1}}\sim q(\phi_{m}|\{e_{s},e_{t}\} ,\phi_{M})$  , obtain samples  $\pmb{x}^{(k_1,k_2^1)}\sim p(\pmb {x}|e^s,\phi_m^{k_1})$  , and obtain samples  $\pmb{x}^{(k_1,k_2^2)}\sim p(\pmb {x}|e^t,\phi_m^{k_1})$
Therefore, we add constraint on the original loss function to obtained the estimation of constraint based ELBO, that is,
$$
\begin{array}{l} \operatorname {ELBO} = \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {N _ {S}} \log p (x ^ {(i)} | j ^ {(i)}, v ^ {(i)}, m ^ {(i)}, \phi_ {m ^ {(i)}} ^ {(j)}) \\ - \frac {1}{2} \left[ \log \frac {| | \boldsymbol {\Sigma} _ {a l l} | |}{| | \tilde {\boldsymbol {\Sigma}} _ {a l l} | |} - d + \operatorname {t r} \left(\boldsymbol {\Sigma} _ {a l l} ^ {- 1} \tilde {\boldsymbol {\Sigma}} _ {a l l}\right) + \left(\tilde {\boldsymbol {\mu}} _ {a l l} - \boldsymbol {\mu} _ {a l l}\right) ^ {T} \boldsymbol {\Sigma} _ {a l l} ^ {- 1} \left(\tilde {\boldsymbol {\mu}} _ {a l l} - \boldsymbol {\mu} _ {a l l}\right) \right] \\ - \beta \cdot \sum_ {i = 1} ^ {N _ {D}} \sum_ {m = 1} ^ {M} \sum_ {j = 1} ^ {N _ {E}} \left[ \lambda_ {1} \cdot \left[ \operatorname {t r} \left(e ^ {\mathbf {E}}\right) - d \right] + \lambda_ {2} \cdot | | \mathbf {E} | | \right] + \lambda \cdot \left[ I (\boldsymbol {x} _ {s}; \boldsymbol {x} _ {t} | \phi_ {M}, \{\boldsymbol {e} _ {s}, \boldsymbol {e} _ {t} \}, D) \right]. \\ \end{array}
$$
E Proof of Theory 3
Proof. To begin with, we introduce two anchor variables  $\pmb{x}, \pmb{e}$ , indicating existing samples and experiments in the system, which are independent with the following experiments. Since  $\pmb{x}_s, \pmb{x}_t$  are  $\epsilon$ -independent given  $\phi_M, \{e_s, e_t\}$  and  $D$ , we have:
$$
\begin{array}{l} I \left(\boldsymbol {x} _ {s}; \boldsymbol {x} _ {t} \mid \phi_ {M}, \left\{\boldsymbol {e} _ {s}, \boldsymbol {e} _ {t} \right\}, D\right) = I \left(\boldsymbol {x} _ {s}; \boldsymbol {x} _ {t} \mid \phi_ {M}, \boldsymbol {x}, \boldsymbol {e} \cup \left\{\boldsymbol {e} _ {s}, \boldsymbol {e} _ {t} \right\}, D\right) \leq \epsilon \\ \Leftrightarrow H \left(\boldsymbol {x} _ {s} \mid \phi_ {M}, \boldsymbol {x}, e \cup \left\{\boldsymbol {e} _ {s}, \boldsymbol {e} _ {t} \right\}, D\right) + H \left(\boldsymbol {x} _ {t} \mid \phi_ {M}, \boldsymbol {x}, e \cup \left\{\boldsymbol {e} _ {s}, \boldsymbol {e} _ {t} \right\}, D\right) \\ - H \left(\boldsymbol {x} _ {s}, \boldsymbol {x} _ {t} \mid \phi_ {M}, \boldsymbol {x}, e \cup \left\{\boldsymbol {e} _ {s}, \boldsymbol {e} _ {t} \right\}, D\right) \leq \epsilon , \\ \end{array}
$$
$$
I \left(\boldsymbol {x} _ {s}; \phi_ {M} | \boldsymbol {x}, \boldsymbol {e} \cup \left\{\boldsymbol {e} _ {s}, \boldsymbol {e} _ {t} \right\}, D\right) = H \left(\boldsymbol {x} _ {s} | \boldsymbol {x}, \boldsymbol {e} \cup \left\{\boldsymbol {e} _ {s}, \boldsymbol {e} _ {t} \right\}, D\right) - H \left(\boldsymbol {x} _ {s} | \phi_ {M}, \boldsymbol {x}, \boldsymbol {e} \cup \left\{\boldsymbol {e} _ {s}, \boldsymbol {e} _ {t} \right\}, D\right)
$$
$$
I \left(\boldsymbol {x} _ {t}; \phi_ {M} | \boldsymbol {x}, \boldsymbol {e} \cup \left\{\boldsymbol {e} _ {s}, \boldsymbol {e} _ {t} \right\}, D\right) = H \left(\boldsymbol {x} _ {t} | \boldsymbol {x}, \boldsymbol {e} \cup \left\{\boldsymbol {e} _ {t}, \boldsymbol {e} _ {t} \right\}, D\right) - H \left(\boldsymbol {x} _ {t} | \phi_ {M}, \boldsymbol {x}, \boldsymbol {e} \cup \left\{\boldsymbol {e} _ {s}, \boldsymbol {e} _ {t} \right\}, D\right)
$$