SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where for the expectation on  $p(\phi_M|D,e)$ , we sample  $\phi_M^{(k_2)}$  from  $\phi_M^{(k_2)}\sim p(\phi_M|e,D)$  for  $K_{2}$  times. For the expectation on  $p(\phi_m|\phi_M)$ , for each  $\phi_M^{(k_2)}$ , we sample  $\phi_m^{(k_2,l_2)}$  from  $\phi_m^{(k_2,l_2)}\sim p(\phi_m|\phi_M^{(k_2)})$  for  $L_{2}$  times. For the expectation on  $p(\boldsymbol {x}|\phi_m,\boldsymbol {e})$ , for each  $\phi_M^{(k_2)}$  and  $\phi_m^{(k_2,l_2)}$ , we sample  $\boldsymbol{x}_{m}^{(k_{2},l_{2},c_{2})}$  from  $\boldsymbol{x}_m^{(k_2,l_2,c_2)}\sim p(\boldsymbol {x}|\phi_m^{(k_2,l_2)},\boldsymbol {e})$  for  $C_2$  times.
Therefore, we can conclude that  $f(j,v,m)$  can be estimated by
where  $\phi_{m}^{(c_{1})},\phi_{m}^{(k_{1})}\sim p(\phi_{m}|D),\pmb{x}_{m}^{(k_{1},l_{1})}\sim p(\pmb {x}|\phi_{m}^{(k_{1})},\pmb {e}),\pmb{\phi}_{M}^{(k_{2})}\sim p(\pmb {\phi}_{M}|D),\pmb{\phi}_{m}^{(k_{2},l_{2})}\sim$ $p(\phi_m|\phi_M^{(k_2)},D)$  and  $\pmb{x}_{m}^{(k_{2},l_{2},c_{2})}\sim p(\pmb {x}|\phi_m^{(k_2,l_2)},\pmb {e}).$
Obviously, the above approximation of  $f(j,v,m)$  only depends on  $p(\phi_m|D)$ ,  $p(\phi_m|\phi_M,D)$  and  $p(\pmb {x}|\phi_m,\pmb {e})$ . In the next, we show how to sample from them in Section A.2, A.3 and A.4, respectively.
A.2 Sampling from  $p(\phi_m|D)$
Basically, sampling from the posterior of  $p(\cdot |D)$  is not easy. To solve this problem, as mentioned in the main paper, we introduce a variational probability "  $q$  " to approximate "  $p$  ". In specific, in order to sample from  $p(\phi_m|D)$ , we first obtain a sample  $\phi_1$  from  $\phi_{1}\sim \mathcal{N}(\mathbf{0},\mathbf{I})$ , and then get  $\phi_{m}$  from the distribution  $q(\phi_m|\phi_1)$ .
$$
q \left(\phi_ {m} \mid \phi_ {1}\right) = \int_ {\phi_ {m - 1}} \dots \int_ {\phi_ {2}} q \left(\phi_ {m}, \phi_ {m - 1}, \dots , \phi_ {2} \mid \phi_ {1}\right) d \phi_ {m - 1} \dots d \phi_ {2}.
$$
$$
q \left(\phi_ {m}, \phi_ {m - 1}, \dots , \phi_ {2} \mid \phi_ {1}\right) = \prod_ {i = 2} ^ {m} q \left(\phi_ {i} \mid \phi_ {i - 1}\right)
$$
$$
q \left(\phi_ {i} \mid \phi_ {i - 1}\right) = \mathcal {N} \left(\boldsymbol {c} _ {i} \phi_ {i - 1} + \boldsymbol {d} _ {i}, \sigma_ {i} ^ {2} \boldsymbol {I}\right),
$$
we have  $q(\phi_m|\phi_1)$  is a Gaussian distribution, which is easy for sampling.
In our model,  $c_{i}$  and  $\sigma_{i}^{2}I$  are diagonal matrices, which means that the dimensions in  $\phi_{i}$  are independent with each other. We denote  $\phi_{i} = [\phi_{i,1},\phi_{i,2}\dots \phi_{i,d}]$ , where  $\phi_{i,j}$  is the  $j$ th element of  $\phi_{i}$ . Then, we have
$$
q \left(\phi_ {m}, \phi_ {m - 1}, \dots , \phi_ {2} \mid \phi_ {1}\right) = \prod_ {j = 1} ^ {d} q \left(\phi_ {m, j}, \phi_ {m - 1, j}, \dots , \phi_ {2, j} \mid \phi_ {1, j}\right).
$$
So our target can be converted to calculate the probability  $q(\phi_{m,j}, \phi_{m-1,j}, \ldots, \phi_{2,j} | \phi_{1,j})$  for all dimensions  $\forall 1 \leq j \leq d$ . Let  $c_{i,j}, d_{i,j}$  and  $\sigma_{i,j}^2$  be the  $j$ th element of  $c_i$ ,  $d_i$  and  $\sigma_i^2$ , respectively. We assume that  $\sigma_{i,j} = \sqrt{c_{i-1,j}^2 + 1} \cdot \sigma_{i-1,j} (i \geq 4)$  and  $\sigma_{3,j} = \sigma_{2,j} = e$ , where  $e$  is the hyper-parameter. Suppose  $\mu_{i,j}$  is the mean of the Gaussian distribution for  $q(\phi_{i,j} | \phi_{i-1,j})$ , that is,
$$
\mu_ {i, j} = c _ {i, j} \phi_ {i - 1, j} + d _ {i, j} (i \geq 2),
$$
then, the approximated joint distribution can be represented as
$$
\begin{array}{l} q \left(\phi_ {m, j}, \phi_ {m - 1, j}, \dots , \phi_ {2, j} \mid \phi_ {1, j}\right) = \prod_ {i = 2} ^ {m} q \left(\phi_ {i, j} \mid \phi_ {i - 1, j}\right) \\ = \prod_ {i = 2} ^ {m} \frac {1}{\sqrt {2 \pi} \sigma_ {i , j}} \cdot e ^ {\frac {- 1}{2 \sigma_ {i , j} ^ {2}} (\phi_ {i, j} - \mu_ {i, j}) ^ {2}}. \\ \end{array}
$$
Then, we integrate  $\phi_{2,j},\phi_{3,j},\ldots ,\phi_{m - 1,j}$  sequentially to obtain  $q(\phi_{m,j}|\phi_{1,j})$
First of all, we integrate  $\phi_{2,j}$  for the joint distribution, where we have:
$$
\begin{array}{l} q \left(\phi_ {m, j}, \phi_ {m - 1, j}, \dots , \phi_ {3, j} \mid \phi_ {1, j}\right) \\ = \int q (\phi_ {m, j}, \phi_ {m - 1, j}, \dots , \phi_ {3, j}, \phi_ {2, j} | \phi_ {1, j}) d \phi_ {2, j} \\ = \int \prod_ {i = 2} ^ {m} \frac {1}{\sqrt {2 \pi} \sigma_ {i , j}} \cdot e ^ {\frac {- 1}{2 \sigma_ {i , j} ^ {2}} (\phi_ {i, j} - \mu_ {i, j}) ^ {2}} d \phi_ {2, j} \\ = \prod_ {i = 4} ^ {m} \frac {1}{\sqrt {2 \pi} \sigma_ {i , j}} \cdot e ^ {\frac {- 1}{2 \sigma_ {i , j} ^ {2}} (\phi_ {i, j} - \mu_ {i, j}) ^ {2}} \cdot \frac {1}{\sqrt {2 \pi} \sigma_ {3 , j}} \cdot \frac {1}{\sqrt {2 \pi} \sigma_ {2 , j}} \cdot \int e ^ {\frac {- 1}{2 \sigma_ {3 , j} ^ {2}} [ \phi_ {3, j} - (c _ {3, j} \phi_ {2, j} + d _ {3, j}) ] ^ {2}}. \\ e ^ {\frac {- 1}{2 \sigma_ {2 , j} ^ {2}} [ \phi_ {2, j} - (w _ {2} \phi_ {1, j} + d _ {3, j}) ] ^ {2}} d \phi_ {2, j}. \\ \end{array}
$$
Denote  $\bar{c}_{2,j} = c_{2,j}$  and  $\bar{d}_{2,j} = d_{2,j}$ , and because of  $\sigma_{3,j} = \sigma_{2,j}$ , we have
$$
\begin{array}{l} q \left(\phi_ {m, j}, \phi_ {m - 1, j}, \dots , \phi_ {3, j} \mid \phi_ {1, j}\right) \\ = \frac {1}{\sqrt {2 \pi} \sigma_ {2 , j}} \cdot \prod_ {i = 4} ^ {m} \frac {1}{\sqrt {2 \pi} \sigma_ {i , j}} \cdot e ^ {\frac {- 1}{2 \sigma_ {i , j} ^ {2}} (\phi_ {i, j} - \mu_ {i, j}) ^ {2}} \cdot \frac {1}{\sqrt {2 \pi} \sigma_ {3 , j}} \int e ^ {\frac {- 1}{2 \sigma_ {3 , j} ^ {2}} [ \phi_ {3, j} - (c _ {3, j} \phi_ {2, j} + d _ {3, j}) ] ^ {2}}. \\ e ^ {\frac {- 1}{2 \sigma_ {3 , j} ^ {2}} \left[ \phi_ {2, j} - (\bar {c} _ {2, j} \phi_ {1, j} + \bar {d} _ {2, j}) \right] ^ {2}} d \phi_ {2, j} \\ = \frac {1}{\sqrt {2 \pi} \sigma_ {2 , j}} \cdot \prod_ {i = 4} ^ {m} \frac {1}{\sqrt {2 \pi} \sigma_ {i , j}} \cdot e ^ {\frac {- 1}{2 \sigma_ {i , j} ^ {2}} (\phi_ {i, j} - \mu_ {i, j}) ^ {2}} \cdot \frac {1}{\sqrt {2 \pi} \sigma_ {3 , j}} \int . \\ \begin{array}{l} \frac {- 1}{2 \sigma_ {3 , j} ^ {2}} \underbrace {\left\{\left[ \phi_ {3 , j} - \left(c _ {3 , j} \phi_ {2 , j} + d _ {3 , j}\right) \right] ^ {2} + \left[ \phi_ {2 , j} - \left(\bar {c} _ {2 , j} \phi_ {1 , j} + \bar {d} _ {2 , j}\right) \right] ^ {2} \right\}} _ {s _ {1}} \\ e \end{array} d \phi_ {2, j}. \\ \end{array}
$$
For  $S_{1}$ , we have
$$
\begin{array}{l} S _ {1} = \left[ \phi_ {3, j} - \left(c _ {3, j} \phi_ {2, j} + d _ {3, j}\right) \right] ^ {2} + \left[ \phi_ {2, j} - \left(\bar {c} _ {2, j} \phi_ {1, j} + \bar {d} _ {2, j}\right) \right] ^ {2} \\ = \phi_ {3, j} ^ {2} + c _ {3, j} ^ {2} \phi_ {2, j} ^ {2} + d _ {3, j} ^ {2} + 2 d _ {3, j} c _ {3, j} \phi_ {2, j} - 2 c _ {3, j} \phi_ {3, j} \phi_ {2, j} - 2 d _ {3, j} \phi_ {3, j} + \phi_ {2, j} ^ {2} \\ + \bar {c} _ {2, j} ^ {2} \phi_ {1, j} ^ {2} + \bar {d} _ {2, j} ^ {2} + 2 \bar {d} _ {2, j} \bar {c} _ {2, j} \phi_ {1, j} - 2 \bar {c} _ {2, j} \phi_ {1, j} - 2 d _ {3, j} \phi_ {3, j} \\ = \left(c _ {3, j} ^ {2} + 1\right) \cdot \\ \left[ \phi_ {2, j} ^ {2} + \frac {2 (d _ {3 , j} c _ {3 , j} - c _ {3 , j} \phi_ {3 , j} - \bar {c} _ {2 , j} \phi_ {1 , j} - \bar {d} _ {2 , j})}{c _ {3 , j} ^ {2} + 1} + \left(\frac {d _ {3 , j} c _ {3 , j} - c _ {3 , j} \phi_ {3 , j} - \bar {c} _ {2 , j} \phi_ {1 , j} - \bar {d} _ {2 , j}}{c _ {3 , j} ^ {2} + 1}\right) ^ {2} \right] \\ - \frac {\left(d _ {3 , j} c _ {3 , j} - c _ {3 , j} \phi_ {3 , j} - \bar {c} _ {2 , j} \phi_ {1 , j} - \bar {d} _ {2 , j}\right) ^ {2}}{c _ {3 , j} ^ {2} + 1} + \phi_ {3, j} ^ {2} + \bar {c} _ {2, j} ^ {2} \phi_ {1, j} ^ {2} + \bar {d} _ {2, j} ^ {2} + 2 \bar {d} _ {2, j} \bar {c} _ {2, j} \phi_ {1, j} - 2 d _ {3, j} \phi_ {3, j} \\ = \left(c _ {3, j} ^ {2} + 1\right) \cdot \left(\phi_ {2, j} - \frac {c _ {3 , j} \phi_ {3 , j} + \bar {c} _ {2 , j} \phi_ {1 , j} + \bar {d} _ {2 , j} - d _ {3 , j} c _ {3 , j}}{c _ {3 , j} ^ {2} + 1}\right) ^ {2} \\ + \frac {c _ {3 , j} ^ {2} \phi_ {3 , j} ^ {2} + \bar {c} _ {2 , j} ^ {2} c _ {3 , j} ^ {2} \phi_ {1 , j} ^ {2} + \bar {d} _ {2 , j} ^ {2} c _ {3 , j} ^ {2} + 2 d _ {3 , j} \bar {c} _ {2 , j} c _ {3 , j} \phi_ {1 , j} - \bar {c} _ {2 , j} ^ {2} \phi_ {1 , j} ^ {2} - \bar {d} _ {2 , j} ^ {2} - 2 \bar {d} _ {2 , j} \bar {c} _ {2 , j} \phi_ {1 , j}}{c _ {3 , j} ^ {2} + 1} \\ + \frac {\phi_ {3 , j} ^ {2} + \bar {c} _ {2 , j} ^ {2} \phi_ {1 , j} ^ {2} + \bar {d} _ {2 , j} ^ {2} + 2 \bar {d} _ {2 , j} \bar {c} _ {2 , j} \phi_ {1 , j} - 2 d _ {3 , j} \phi_ {3 , j} - c _ {3 , j} ^ {2} d _ {3 , j} ^ {2} - c _ {3 , j} ^ {2} \phi_ {3 , j} ^ {2} + 2 c _ {3 , j} ^ {2} d _ {3 , j} \phi_ {3 , j}}{c _ {3 , j} ^ {2} + 1} \\ + \frac {2 c _ {3 , j} \bar {d} _ {2 , j} d _ {3 , j} - 2 c _ {3 , j} \phi_ {3 , j} \bar {c} _ {2 , j} \phi_ {1 , j} - 2 c _ {3 , j} \phi_ {3 , j} \bar {d} _ {2 , j} + 2 \bar {d} _ {2 , j} \bar {c} _ {2 , j} c _ {3 , j} ^ {2} \phi_ {1 , j} - 2 d _ {3 , j} c _ {3 , j} ^ {2} \phi_ {3 , j}}{c _ {3 , j} ^ {2} + 1}. \\ \end{array}
$$
$$
\begin{array}{l} S _ {1} = \left(c _ {3, j} ^ {2} + 1\right) \cdot \left(\phi_ {2, j} - \frac {c _ {3 , j} \phi_ {3 , j} + \bar {c} _ {2 , j} \phi_ {1 , j} + \bar {d} _ {2 , j} - d _ {3 , j} c _ {3 , j}}{c _ {3 , j} ^ {2} + 1}\right) ^ {2} \\ + \frac {\phi_ {3 , j} ^ {2} - 2 (\bar {c} _ {2 , j} c _ {3 , j} \phi_ {1 , j} + \bar {d} _ {2 , j} c _ {3 , j} + d _ {3 , j}) \phi_ {3 , j}}{c _ {3 , j} ^ {2} + 1} \\ + \frac {\left(\bar {c} _ {2 , j} c _ {3 , j} \phi_ {1 , j} + \bar {d} _ {2 , j} c _ {3 , j} + d _ {3 , j}\right) ^ {2} - d _ {3 , j} ^ {2} \cdot \left(c _ {3 , j} ^ {2} + 1\right)}{c _ {3 , j} ^ {2} + 1} \\ = \left(c _ {3, j} ^ {2} + 1\right) \cdot \left(\phi_ {2, j} - \frac {c _ {3 , j} \phi_ {3 , j} + \bar {c} _ {2 , j} \phi_ {1 , j} + \bar {d} _ {2 , j} - d _ {3 , j} c _ {3 , j}}{c _ {3 , j} ^ {2} + 1}\right) ^ {2} \\ + \frac {1}{c _ {3 , j} ^ {2} + 1} \cdot \left[ \phi_ {3, j} - (c _ {3, j} \bar {c} _ {2, j} \phi_ {1, j} + \bar {d} _ {2, j} c _ {3, j} + d _ {3, j}) \right] ^ {2} - d _ {3, j} ^ {2}. \\ \end{array}
$$
Therefore we have
$$
\begin{array}{l} q \left(\phi_ {m, j}, \phi_ {m - 1, j}, \dots , \phi_ {3, j} \mid \phi_ {1, j}\right) \\ = \frac {1}{\sqrt {2 \pi} \sigma_ {2 , j}} \cdot \prod_ {i = 4} ^ {m} \frac {1}{\sqrt {2 \pi} \sigma_ {i , j}} \cdot e ^ {\frac {- 1}{2 \sigma_ {i , j} ^ {2}} (\phi_ {i, j} - \mu_ {i, j}) ^ {2}}. \\ \underbrace {\left[ \int \frac {\sqrt {c _ {3 , j} ^ {2} + 1}}{\sqrt {2 \pi} \sigma_ {3 , j}} e ^ {\frac {- (c _ {3 , j} ^ {2} + 1)}{2 \sigma_ {3 , j} ^ {2}}} \left(\phi_ {2 , j} - \frac {c _ {3 , j} \phi_ {3 , j} + \bar {c} _ {2 , j} \phi_ {1 , j} + \bar {d} _ {2 , j} - d _ {3 , j} c _ {3 , j}}{c _ {3 , j} ^ {2} + 1}\right) ^ {2} d \phi_ {2 , j} \right]} _ {S _ {2}}. \\ e ^ {\frac {- 1}{2 \sigma_ {3 , j} ^ {2} \cdot (c _ {3 , j} ^ {2} + 1)} \left[ \phi_ {3, j} - (c _ {3, j} \bar {c} _ {2, j} \phi_ {1, j} + \bar {d} _ {2, j} c _ {3, j} + d _ {3, j}) \right] ^ {2}} \cdot e ^ {\frac {d _ {3 , j} ^ {2}}{2 \sigma_ {3 , j} ^ {2}}} \cdot \frac {1}{\sqrt {c _ {3 , j} ^ {2} + 1}} \\ \end{array}
$$
The  $S_{2}$  part is the integration form of  $\phi_{2,j} \sim \mathcal{N}\left(\frac{c_{3,j}\phi_{3,j} + \bar{c}_{2,j}\phi_{1,j} + \bar{d}_{2,j} - d_{3,j}c_{3,j}}{c_{3,j}^{2} + 1},\frac{\sigma_{3,j}^{2}}{c_{3,j}^{2} + 1}\right)$ , which is equal to 1, so we have
$$
\begin{array}{l} q \left(\phi_ {m, j}, \phi_ {m - 1, j}, \dots , \phi_ {3, j} \mid \phi_ {1, j}\right) \\ = \frac {1}{\sqrt {2 \pi} \sigma_ {2 , j}} \cdot \prod_ {i = 4} ^ {m} \frac {1}{\sqrt {2 \pi} \sigma_ {i , j}} \cdot e ^ {\frac {- 1}{2 \sigma_ {i , j} ^ {2}} (\phi_ {i, j} - \mu_ {i, j}) ^ {2}} \cdot e ^ {\frac {- 1}{2 \sigma_ {3 , j} ^ {2} \cdot (c _ {3 , j} ^ {2} + 1)} \left[ \phi_ {3, j} - (c _ {3, j} \bar {c} _ {2, j} \phi_ {1, j} + \bar {d} _ {2, j} c _ {3, j} + d _ {3, j}) \right] ^ {2}}. \\ e ^ {\frac {d _ {3 , j} ^ {2}}{2 \sigma_ {3 , j} ^ {2}}} \cdot \frac {1}{\sqrt {c _ {3 , j} ^ {2} + 1}}. \\ \end{array}
$$
We denote  $\bar{c}_{3,j} = c_{3,j}\bar{c}_{2,j}$  and  $\bar{d}_{3,j} = \bar{d}_{2,j}c_{3,j} + d_{3,j}$ , and denote  $r_{2,j} = e^{\frac{d_{3,j}^2}{2\sigma_{3,j}^2}} \cdot \frac{1}{\sqrt{c_{3,j}^2 + 1}}$ , so we have
$$
\begin{array}{l} q \left(\phi_ {m, j}, \phi_ {m - 1, j}, \dots , \phi_ {3, j} \mid \phi_ {1, j}\right) \\ = \frac {1}{\sqrt {2 \pi} \sigma_ {2 , j}} \cdot \prod_ {i = 4} ^ {m} \frac {1}{\sqrt {2 \pi} \sigma_ {i , j}} \cdot e ^ {\frac {- 1}{2 \sigma_ {i , j} ^ {2}} \left(\phi_ {i, j} - \mu_ {i, j}\right) ^ {2}} \cdot e ^ {\frac {- 1}{2 \sigma_ {3 , j} ^ {2} \cdot \left(c _ {3 , j} ^ {2} + 1\right)} \left[ \phi_ {3, j} - \left(\bar {c} _ {3, j} \phi_ {1, j} + \bar {d} _ {3, j}\right) \right] ^ {2}} \cdot r _ {2, j} \\ = \frac {1}{\sqrt {2 \pi} \sigma_ {2 , j}} \cdot \prod_ {i = 5} ^ {m} \frac {1}{\sqrt {2 \pi} \sigma_ {i , j}} \cdot e ^ {\frac {- 1}{2 \sigma_ {i , j} ^ {2}} (\phi_ {i, j} - \mu_ {i, j}) ^ {2}} \cdot \frac {1}{\sqrt {2 \pi} \sigma_ {4}} \cdot e ^ {\frac {- 1}{2 \sigma_ {4} ^ {2}} [ \phi_ {4} - (c _ {4} \phi_ {3, j} + d _ {4}) ] ^ {2}}. \\ e ^ {\frac {- 1}{2 \sigma_ {3 , j} ^ {2} \cdot (c _ {3 , j} ^ {2} + 1)} \left[ \phi_ {3, j} - (\bar {c} _ {3, j} \phi_ {1, j} + \bar {d} _ {3, j}) \right] ^ {2}} \cdot r _ {2, j} \\ = \frac {1}{\sqrt {2 \pi} \sigma_ {2 , j}} \cdot \prod_ {i = 5} ^ {m} \frac {1}{\sqrt {2 \pi} \sigma_ {i , j}} \cdot e ^ {\frac {- 1}{2 \sigma_ {i , j} ^ {2}} (\phi_ {i, j} - \mu_ {i, j}) ^ {2}} \cdot \frac {1}{\sqrt {2 \pi} \sigma_ {4}} \cdot e ^ {\frac {- 1}{2 \sigma_ {4} ^ {2}} [ \phi_ {4} - (c _ {4} \phi_ {3, j} + d _ {4}) ] ^ {2}}. \\ e ^ {\frac {- 1}{2 \sigma_ {4} ^ {2}} \left[ \phi_ {3, j} - (\bar {c} _ {3, j} \phi_ {1, j} + \bar {d} _ {3, j}) \right] ^ {2}} \cdot r _ {2, j} \\ \end{array}
$$
Similarly, then we integrate  $\phi_{3,j}$
$$
\begin{array}{l} q \left(\phi_ {m, j}, \phi_ {m - 1, j}, \dots , \phi_ {4} \mid \phi_ {1, j}\right) \\ = \int q (\phi_ {m, j}, \phi_ {m - 1, j}, \dots , \phi_ {3, j} | \phi_ {1, j}) d \phi_ {3, j} \\ = \frac {r _ {2 , j}}{\sqrt {2 \pi} \sigma_ {2 , j}} \cdot \prod_ {i = 5} ^ {m} \frac {1}{\sqrt {2 \pi} \sigma_ {i , j}} \cdot e ^ {\frac {- 1}{2 \sigma_ {i , j} ^ {2}} (\phi_ {i, j} - \mu_ {i, j}) ^ {2}} \cdot \int \frac {1}{\sqrt {2 \pi} \sigma_ {4}} \cdot e ^ {\frac {- 1}{2 \sigma_ {4} ^ {2}} [ \phi_ {4} - (c _ {4} \phi_ {3, j} + d _ {4}) ] ^ {2}}. \\ e ^ {\frac {- 1}{2 \sigma_ {4} ^ {2}} \left[ \phi_ {3, j} - (\bar {c} _ {3, j} \phi_ {1, j} + \bar {d} _ {3, j}) \right] ^ {2}} d \phi_ {2, j}. \\ \end{array}
$$
The formulation is similar to the previous one, so we can utilize the process above to integrate successively, and we finally obtain
$$
q (\phi_ {m, j} | \phi_ {1, j}) = \frac {\prod_ {i = 2} ^ {m - 1} r _ {i , j}}{\sqrt {2 \pi} \sigma_ {2 , j}} \cdot e ^ {\frac {- 1}{2 \sigma_ {m , j} ^ {2} \cdot (c _ {m , j} ^ {2} + 1)} \left[ \phi_ {m, j} - (\bar {c} _ {m, j} \phi_ {1, j} + \bar {d} _ {m, j}) \right] ^ {2}},
$$
which indicates
$$
p (\phi_ {m, j} | \phi_ {1, j}, D) \approx \frac {\prod_ {i = 2} ^ {m - 1} r _ {i , j}}{\sqrt {2 \pi} \sigma_ {2 , j}} \cdot e ^ {\frac {- 1}{2 \sigma_ {m , j} ^ {2} \cdot (c _ {m , j} ^ {2} + 1)} \left[ \phi_ {m, j} - (\bar {c} _ {m} \phi_ {1, j} + \bar {d} _ {m, j}) \right] ^ {2}},
$$
where we have the iterative calculation by
$$
r _ {i, j} = e ^ {\frac {d _ {i + 1 , j} ^ {2}}{2 \sigma_ {i + 1 , j} ^ {2}}} \cdot \frac {1}{\sqrt {c _ {i + 1 , j} ^ {2} + 1}},
$$
$$
\bar {c} _ {i, j} = c _ {i, j} \bar {c} _ {i - 1, j}, (i \geq 3),
$$
$$
\bar {d} _ {i, j} = \bar {d} _ {i - 1, j} c _ {i, j} + d _ {i, j}, (i \geq 3).
$$
A.3 Sampling from  $p(\phi_m|\phi_M,D)$
To sample from the distribution  $p(\phi_m|\phi_M,D)$ , we first obtain a sample  $\phi_1$  from the prior distribution (i.e.,  $\phi_1\sim \mathcal{N}(\mathbf{0},\mathbf{I})$ ), then get  $\phi_{m}$  from a consecutive sampling process:
$$
\phi_ {M - 1} \sim p \left(\phi_ {M - 1} \mid \phi_ {M}, \phi_ {1}, D\right),
$$
$$
\phi_ {M - 2} \sim p \left(\phi_ {M - 2} \mid \phi_ {M - 1}, \phi_ {1}, D\right),
$$
中 中
$$
\phi_ {m} \sim p (\phi_ {m} | \phi_ {m + 1}, \phi_ {1}, D),
$$
because of the Markov property in our cascaded model. So our target is obtaining the distributions  $p(\phi_{i-1}|\phi_i, \phi_1, D)$ . For a certain  $p(\phi_{i-1}|\phi_i, \phi_1, D)$ , according to the Bayes rule, we have
$$
p \left(\phi_ {i - 1} \mid \phi_ {i}, \phi_ {1}, D\right) = \frac {p \left(\phi_ {i} \mid \phi_ {i - 1} , \phi_ {1} , D\right) \cdot p \left(\phi_ {i - 1} \mid \phi_ {1} , D\right)}{p \left(\phi_ {i} \mid \phi_ {1} , D\right)}.
$$
Similarly with the last section, we use non-bold symbols to represent one dimension of the multidimensional parameters, where they are able to transfer independently, and finally construct the eventual parameters by concatenating, that is,
$$
p \left(\phi_ {i - 1} \mid \phi_ {i}, \phi_ {1}, D\right) = \prod_ {j = 1} ^ {d} p \left(\phi_ {i, j} \mid \phi_ {i - 1, j}, \phi_ {1, j}, D\right).
$$
So our target can be converted to calculate the probability  $p(\phi_{i,j}|\phi_{i - 1,j},\phi_{1,j},D)$  for all dimensions  $\forall 1\leq j\leq d$ . According to the Markov property and the transportation probability, we have
$$
\begin{array}{l} p \left(\phi_ {i, j} \mid \phi_ {i - 1, j}, \phi_ {1, j}, D\right) \approx q \left(\phi_ {i, j} \mid \phi_ {i - 1, j}, \phi_ {1, j}\right) \\ = \frac {1}{\sqrt {2 \pi} \sigma_ {i , j}} \cdot e ^ {\frac {- 1}{2 \sigma_ {i , j} ^ {2}} \cdot [ \phi_ {i, j} - (c _ {i} \phi_ {i - 1, j} + d _ {i, j}) ] ^ {2}}. \\ \end{array}
$$
According to the previous section, we have
$$
p (\phi_ {i, j} | \phi_ {1, j}, D) \approx q (\phi_ {i, j} | \phi_ {1, j}) = \frac {\prod_ {i = 2} ^ {i - 1} r _ {i , j}}{\sqrt {2 \pi} \sigma_ {2 , j}} \cdot e ^ {\frac {- 1}{2 \sigma_ {i , j} ^ {2} \cdot (c _ {i , j} ^ {2} + 1)} \left[ \phi_ {i, j} - (\bar {c} _ {i} \phi_ {1, j} + \bar {d} _ {i}) \right] ^ {2}},
$$
$$
p \big (\phi_ {i - 1, j} | \phi_ {1, j}, D \big) \approx q \big (\phi_ {i - 1, j} | \phi_ {1, j} \big) = \frac {\prod_ {i = 2} ^ {i - 2} r _ {i , j}}{\sqrt {2 \pi} \sigma_ {2 , j}} \cdot e ^ {\frac {- 1}{2 \sigma_ {i - 1 , j} ^ {2} \cdot (c _ {i - 1 , j} ^ {2} + 1)} \left[ \phi_ {i - 1, j} - (\bar {c} _ {i - 1, j} \phi_ {1, j} + \bar {d} _ {i - 1, j}) \right] ^ {2}}.
$$
$$
\begin{array}{l} p \left(\phi_ {i - 1, j} \mid \phi_ {i, j}, \phi_ {1, j}, D\right) \approx \frac {q \left(\phi_ {i , j} \mid \phi_ {i - 1 , j} , \phi_ {1 , j}\right) \cdot q \left(\phi_ {i - 1 , j} \mid \phi_ {1 , j}\right)}{q \left(\phi_ {i , j} \mid \phi_ {1 , j}\right)} \\ = \frac {1}{\sqrt {2 \pi} \sigma_ {i , j} \cdot r _ {i - 1}} \cdot e ^ {\frac {- 1}{2 \sigma_ {i , j} ^ {2}} \cdot [ \phi_ {i, j} - (c _ {i, j} \phi_ {i - 1, j} + d _ {i, j}) ] ^ {2}} \cdot \frac {e ^ {\frac {[ \phi_ {i - 1 , j} - (\bar {c} _ {i - 1 , j} \phi_ {1 , j} + \bar {d} _ {i - 1 , j}) ] ^ {2}}{- 2 \sigma_ {i - 1 , j} ^ {2} \cdot (c _ {i - 1 , j} ^ {2} + 1)}}}{e ^ {\frac {[ \phi_ {i , j} - (c _ {i , j} \phi_ {i - 1 , j} + d _ {i , j}) ] ^ {2}}{- 2 \sigma_ {i , j} ^ {2} \cdot (c _ {i , j} ^ {2} + 1)}}} \\ = \sqrt {\frac {c _ {i , j} + 1}{2 \pi \sigma_ {i , j} ^ {2}}} \cdot e ^ {\frac {2 \sigma_ {i , j} ^ {2}}{d _ {i , j} ^ {2}}} \cdot e ^ {\frac {[ \phi_ {i , j} - (c _ {i , j} \phi_ {i - 1 , j} + d _ {i , j}) ] ^ {2}}{2 \sigma_ {i + 1 , j} ^ {2}}} \cdot e ^ {\frac {[ \phi_ {i , j} - (c _ {i , j} \phi_ {i - 1 , j} + d _ {i , j}) ] ^ {2}}{- 2 \sigma_ {i , j} ^ {2}}} \cdot e ^ {\frac {[ \phi_ {i - 1 , j} - (\bar {c} _ {i - 1 , j} \phi_ {1 , j} + \bar {d} _ {i - 1 , j}) ] ^ {2}}{- 2 \sigma_ {i , j} ^ {2}}} \\ = \sqrt {\frac {c _ {i , j} + 1}{2 \pi \sigma_ {i , j} ^ {2}}} \cdot e ^ {\frac {2 \sigma_ {i , j} ^ {2}}{d _ {i , j} ^ {2}} + \frac {\left[ \phi_ {i , j} - (c _ {i , j} \phi_ {i - 1 , j} + d _ {i , j}) \right] ^ {2}}{2 \sigma_ {i + 1 , j} ^ {2}}}. \\ \frac {\frac {1}{- 2 \sigma_ {i , j} ^ {2}}}{2} \cdot \underbrace {\left\{\left[ \phi_ {i , j} - \left(c _ {i , j} \phi_ {i - 1 , j} + d _ {i , j}\right) \right] ^ {2} + \left[ \phi_ {i - 1 , j} - \left(\bar {c} _ {i - 1 , j} \phi_ {1 , j} + \bar {d} _ {i - 1 , j}\right) \right] ^ {2} \right\}} _ {C} \\ \end{array}
$$
Then we calculate the part  $C$  as
$$
\begin{array}{l} C = \left[ \phi_ {i, j} - \left(c _ {i, j} \phi_ {i - 1, j} + d _ {i, j}\right) \right] ^ {2} + \left[ \phi_ {i - 1, j} - \left(\bar {c} _ {i - 1, j} \phi_ {1, j} + \bar {d} _ {i - 1, j}\right) \right] ^ {2} \\ = \phi_ {i, j} ^ {2} + \left(c _ {i, j} \phi_ {i - 1, j} + d _ {i, j}\right) ^ {2} - 2 \left(c _ {i, j} \phi_ {i - 1, j} + d _ {i, j}\right) \phi_ {i, j} \\ + \phi_ {i - 1, j} ^ {2} + (\bar {w} _ {i - 1} \phi_ {1, j} + \bar {d} _ {i - 1, j}) ^ {2} - 2 \phi_ {i - 1, j} (\bar {c} _ {i - 1, j} \phi + \bar {d} _ {i - 1, j}) \\ = \phi_ {i, j} ^ {2} + c _ {i, j} ^ {2} \phi_ {i - 1, j} ^ {2} + d _ {i, j} ^ {2} + 2 d _ {i, j} c _ {i, j} \phi_ {i - 1, j} - 2 c _ {i, j} \phi_ {i - 1, j} \phi_ {i, j} - 2 d _ {i, j} \phi_ {i, j} + \phi_ {i - 1, j} ^ {2} \\ + \bar {c} _ {i - 1, j} ^ {2} \phi_ {1, j} ^ {2} + \bar {d} _ {i - 1, j} ^ {2} + 2 \bar {c} _ {i - 1, j} \bar {d} _ {i - 1, j} \phi_ {1, j} - 2 \bar {c} _ {i - 1, j} \phi_ {1, j} \phi_ {i - 1, j} - 2 \bar {d} _ {i - 1, j} \phi_ {i - 1, j} \\ = \left(\bar {w} _ {i - 1} ^ {2} + 1\right) \cdot \left[ \phi_ {i - 1, j} - \frac {c _ {i , j} \phi_ {i , j} + \bar {c} _ {i - 1 , j} \phi_ {1 , j} + \bar {d} _ {i - 1 , j} - d _ {i , j} c _ {i , j}}{c _ {i , j} ^ {2} + 1} \right] ^ {2} + B, \\ \end{array}
$$
where  $B$  does not include  $\phi_{i}$ , which indicates
$$
p (\phi_ {i - 1, j} | \phi_ {i, j}, \phi_ {1, j}, D) \sim \mathcal {N} (\frac {c _ {i , j} \phi_ {i , j} + \bar {c} _ {i - 1 , j} \phi_ {1 , j} + \bar {d} _ {i - 1 , j} - d _ {i , j} w _ {i - 2}}{c _ {i , j} ^ {2} + 1}, \frac {\sigma_ {i , j} ^ {2}}{c _ {i , j} ^ {2} + 1}).
$$
A.4 Calculation of  $p(x|e, \phi_m)$
In this section, we will show how to calculate the graph probability  $p(\boldsymbol{x}|\boldsymbol{e}, \boldsymbol{\phi}_m)$ . Remember the graph parameters  $\phi_m = [\theta_m; \mathbf{S}_m; \mathbf{T}_m]$ , so we have
$$
\begin{array}{l} p (\boldsymbol {x} | e, \phi_ {m}) = \int_ {\mathbf {E}} p (\boldsymbol {x} | e, \boldsymbol {\theta} _ {m}, \mathbf {E}) \cdot p (\mathbf {E} | e, \boldsymbol {S} _ {m}, \boldsymbol {T} _ {m}) d \mathbf {E} \\ = \mathbb {E} _ {\mathbf {E} \sim p (\mathbf {E} | \mathbf {S} _ {m}, \mathbf {T} _ {m})} [ p (\boldsymbol {x} | \boldsymbol {e}, \boldsymbol {\theta} _ {m}, \mathbf {E}) ]. \\ \end{array}
$$
According to Monte Carlo sampling, we have
$$
p (\boldsymbol {x} | \boldsymbol {e}, \boldsymbol {\phi} _ {m}) = \frac {1}{K} \cdot \sum_ {l = 1} ^ {K} p (\boldsymbol {x} | \boldsymbol {e}, \boldsymbol {\theta} _ {m}, \mathbf {E} _ {l}),
$$
where  $\mathbf{E}_l[i,j] \sim \mathrm{Bernoulli}(\sigma(\mathbf{S}_m^T[i] \cdot \mathbf{T}_m[j]))$ . In order to conduct intervention process, we change the  $j$ th column of  $\mathbf{E}_l$  to zeros, and represent it with  $\tilde{\mathbf{E}}_l$ . Moreover, we replace the  $j$ th element of  $\boldsymbol{x}$
with  $v$ , and get the result  $\tilde{\pmb{x}}$ . We change the  $j$ th element of  $\epsilon_{m}$  with zero, and get the result  $\tilde{\epsilon}_{m}$ . Then according the definition of causal graphs, we have
$$
p (\boldsymbol {x} | \boldsymbol {e}, \phi_ {m}) = \frac {1}{K} \sum_ {l = 1} ^ {K} \mathcal {N} (\boldsymbol {x}; \boldsymbol {f} (\tilde {\boldsymbol {x}}; \tilde {\mathbf {E}} _ {l}, \boldsymbol {\gamma} _ {m}), \tilde {\boldsymbol {\epsilon}} _ {m}),
$$
where  $\pmb{f}$  is the causal function that depends on the parameter  $\gamma_{m}$ .
B Bayesian Optimization for Determining  $(j^{*},v^{*},m^{*})$
We intend to find the best tuple for acquisition, that is,
$$
(j ^ {*}, v ^ {*}, m ^ {*}) = \underset {(j, v, m)} {\arg \max } f (j, v, m).
$$
We define the best interventional value  $v$  under interventional node  $j$  and fidelity  $m$  as
$$
\begin{array}{l} v ^ {*} (j, m) = \underset {v} {\arg \max } f (j, v, m) \\ = \operatorname * {a r g   m a x} _ {v} f _ {j, m} (v). \\ \end{array}
$$