SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left(\boldsymbol {\Sigma} _ {*} ^ {(1)} \otimes \mathbf {S} ^ {(1)}\right) ^ {- 1} - \left(\boldsymbol {\Sigma} _ {*} ^ {(2)}\right) ^ {- 1} \left(\boldsymbol {\Gamma} ^ {\top} \left(\boldsymbol {\Sigma} _ {*} ^ {(2)}\right) ^ {- 1} \boldsymbol {\Gamma} + \boldsymbol {\Sigma} _ {*} ^ {(1)}\right) ^ {- 1} \left(\boldsymbol {\Sigma} _ {*} ^ {(1)} \otimes \mathbf {S} ^ {(1)}\right) ^ {- 1} = \boldsymbol {\Gamma} ^ {\top} \left(\boldsymbol {\Sigma} _ {*} ^ {(2)} + \boldsymbol {\Gamma} ^ {\top} \left(\boldsymbol {\Sigma} _ {*} ^ {(1)} \otimes \mathbf {S} ^ {(1)}\right) ^ {- 1} \boldsymbol {\Gamma}\right) ^ {- 1} \boldsymbol {\Gamma}
$$
$$
\left(\boldsymbol {\Sigma} _ {*} ^ {(2)}\right) ^ {- 1} \boldsymbol {\Gamma} \left(\boldsymbol {\Gamma} ^ {\top} \left(\boldsymbol {\Sigma} _ {*} ^ {(2)}\right) ^ {- 1} \boldsymbol {\Gamma} + \boldsymbol {\Sigma} _ {*} ^ {(1)}\right) ^ {- 1} \left(\boldsymbol {\Sigma} _ {*} ^ {(1)} \otimes \mathbf {S} ^ {(1)}\right) ^ {- 1} = - \left(\boldsymbol {\Sigma} _ {*} ^ {(2)} + \boldsymbol {\Gamma} ^ {\top} \left(\boldsymbol {\Sigma} _ {*} ^ {(1)} \otimes \mathbf {S} ^ {(1)}\right) ^ {- 1} \boldsymbol {\Gamma}\right) ^ {- 1} \boldsymbol {\Gamma}.
$$
Therefore, the likelihood of the posterior distribution is
$$
\begin{array}{l} p (\bar {\mathbf {Y}} (\mathbf {x} _ {*}, 2) | \mathbf {Y} ^ {(2)}, \mathbf {Y} ^ {(1)}) \\ = 2 \pi^ {- \frac {N _ {p} D}{2}} \times \left| \boldsymbol {\Sigma} _ {*} ^ {(2)} + \boldsymbol {\Gamma} ^ {\top} \left(\boldsymbol {\Sigma} _ {*} ^ {(1)} \otimes \mathbf {S} ^ {(1)}\right) ^ {- 1} \boldsymbol {\Gamma} \right| ^ {- \frac {1}{2}} \tag {A.41} \\ \times \exp \left[ - \frac {1}{2} \left(\mathbf {y} _ {*} ^ {(2)} - \bar {\mathbf {y}} _ {*} ^ {(2)}\right) ^ {\top} \left(\boldsymbol {\Sigma} _ {*} ^ {(2)} + \boldsymbol {\Gamma} ^ {\top} \left(\boldsymbol {\Sigma} _ {*} ^ {(1)} \otimes \mathbf {S} ^ {(1)}\right) ^ {- 1} \boldsymbol {\Gamma}\right) ^ {- 1} \left(\mathbf {y} _ {*} ^ {(2)} - \bar {\mathbf {y}} _ {*} ^ {(2)}\right) \right]. \\ \end{array}
$$
From the upper formula, for the non-subset data structure, the posetrior mean and covariance matrix are,
$$
\bar {\mathbf {y}} _ {*} ^ {(2)} = \left[ \left(\mathbf {k} _ {a *} ^ {(1 1)}\right) \left(\mathbf {K} _ {a} ^ {(1 1)}\right) ^ {- 1} \otimes \mathbf {I} \right] \left( \begin{array}{c} \vec {\mathbf {y}} ^ {(1)} \\ \vec {\mathbf {y}} ^ {(1)} \end{array} \right) + \left[ \left(\mathbf {k} _ {a *} ^ {(2 2)}\right) \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} \otimes \mathbf {I} \right] \mathbf {y} ^ {(2)}, \tag {A.42}
$$
$$
\boldsymbol {\Sigma} (\mathbf {x} _ {*}, 2) = \boldsymbol {\Sigma} _ {*} ^ {(2)} + \boldsymbol {\Gamma} ^ {\top} \left(\boldsymbol {\Sigma} _ {*} ^ {(1)} \otimes \mathbf {S} ^ {(1)}\right) ^ {- 1} \boldsymbol {\Gamma}.
$$
In Eq. (A42), we also prove that matrix  $\mathbf{S}$  does not take part in posterior mean function which means autokriging ability still holds in the non-subset data structure. By recursively applying this conclusion, we can easily extend it to a multi-fidelity problem and we show the detailed application in our main paper.
I Decomposition of Joint Likelihood With Non-Subset Structure
As we have shown that the autokrigeability holds in subset and non-subset data structure, we thus assume an identical spatial correlation,  $\mathbf{S} = \mathbf{I}$ , for easy computation acceleration (with the cost of losing the accuracy in the predictive variance). Due to the identical spatial correlations, we no longer need vectorization. First, we decompose the joint likelihood  $\mathcal{L}$  into several independent parts,
$$
\begin{array}{l} \mathcal {L} = \log p (\mathbf {Y} ^ {(0: 2)}) = \log p (\mathbf {Y} ^ {(0)}, \mathbf {Y} ^ {(1)}, \mathbf {Y} ^ {(2)}) \\ = \log p \left(\mathbf {Y} ^ {(0)}\right) + \log p \left(\mathbf {Y} ^ {(1)} \mid \mathbf {Y} ^ {(0)}\right) + \log p \left(\mathbf {Y} ^ {(2)} \mid \mathbf {Y} ^ {(1)}, \mathbf {Y} ^ {(0)}\right) \\ = \log p (\mathbf {Y} ^ {(0)}) + \log p (\mathbf {Y} ^ {(1)} | \mathbf {Y} ^ {(0)}) + \log \int \overbrace {p (\mathbf {Y} ^ {(2)} | \mathbf {Y} ^ {(1)} , \hat {\mathbf {Y}} ^ {(1)})} ^ {\text {p a r t 1}} \overbrace {p (\hat {\mathbf {Y}} ^ {(1)} | \mathbf {Y} ^ {(1)})} ^ {\text {p a r t 2}} d \hat {\mathbf {Y}} ^ {(1)}. \\ \end{array}
$$
Once the  $\hat{\mathbf{Y}}^{(1)}$  is fixed, the probability of part 1 in Eq. (A43) can be written as,
$$
\begin{array}{l} p \left(\mathbf {Y} ^ {(2)} \mid \mathbf {Y} ^ {(1)}, \hat {\mathbf {Y}} ^ {(1)}\right) = 2 \pi^ {- \frac {N ^ {(2)} D}{2}} \times \left| \mathbf {K} _ {a} ^ {(2 2)} \right| ^ {- \frac {1}{2}} \times \exp \left[ - \frac {1}{2} \left(\mathbf {Y} _ {a} ^ {(2)}\right) ^ {\top} \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} \mathbf {Y} _ {a} ^ {(2)} \right] \\ = 2 \pi^ {- \frac {N ^ {(2)} D}{2}} \times \left| \mathbf {K} _ {a} ^ {(2 2)} \right| ^ {- \frac {1}{2}} \times \exp \left[ - \frac {1}{2} \left[ \left( \begin{array}{c} \check {\mathbf {Y}} ^ {(2)} \\ \check {\mathbf {Y}} ^ {(2)} \end{array} \right) - \left( \begin{array}{c} \check {\mathbf {Y}} ^ {(1)} \\ \check {\mathbf {Y}} ^ {(1)} \end{array} \right) \right] ^ {\top} \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} \left[ \left( \begin{array}{c} \check {\mathbf {Y}} ^ {(2)} \\ \check {\mathbf {Y}} ^ {(2)} \end{array} \right) - \left( \begin{array}{c} \check {\mathbf {Y}} ^ {(1)} \\ \check {\mathbf {Y}} ^ {(1)} \end{array} \right) \right] \right]. \\ \end{array}
$$
Then is the part 2 in Eq. (A43), it is based on the posterior distribution of lower fidelity, zu
$$
p \left(\hat {\mathbf {Y}} ^ {(1)} \mid \mathbf {Y} ^ {(1)}\right) = 2 \pi^ {- \frac {N _ {n} ^ {(2)} D}{2}} \times \left| \hat {\boldsymbol {\Sigma}} ^ {(1)} \right| ^ {- \frac {1}{2}} \times \exp \left[ - \frac {1}{2} \left(\hat {\mathbf {Y}} ^ {(1)} - \bar {\mathbf {Y}} ^ {(1)}\right) ^ {\top} \left(\hat {\boldsymbol {\Sigma}} ^ {(1)}\right) ^ {- 1} \left(\hat {\mathbf {Y}} ^ {(1)} - \bar {\mathbf {Y}} ^ {(1)}\right) \right]. \tag {A.45}
$$
where we use the  $N_{n}^{(2)}$  to denote the missing point in second fidelity corresponding with fidelity 2, which means the  $\hat{\mathbf{Y}}^{(2)}$  parts, yellow part, in Fig. 7.
Therefore, we can combine the Eq. (A44) with the Eq. (A45), the integral part in Eq. (A43) can be written as,
$$
= \log \int \left\{ \right.2 \pi^ {- \frac {N ^ {(1)} D}{2}} \times \left| \mathbf {K} _ {a} ^ {(1 1)} \right| ^ {- \frac {1}{2}} \times \exp \left[ - \frac {1}{2} \left(\left(\begin{array}{c}\breve {\mathbf {Y}} ^ {(2)}\\\hat {\mathbf {Y}} ^ {(2)}\end{array}\right) - \left(\begin{array}{c}\breve {\mathbf {Y}} ^ {(1)}\\\bar {\mathbf {Y}} ^ {(1)}\end{array}\right)\right) ^ {\top} \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} \left(\left(\begin{array}{c}\breve {\mathbf {Y}} ^ {(2)}\\\hat {\mathbf {Y}} ^ {(2)}\end{array}\right) - \left(\begin{array}{c}\breve {\mathbf {Y}} ^ {(1)}\\\bar {\mathbf {Y}} ^ {(1)}\end{array}\right)\right)\right]
$$
$$
\begin{array}{l} \log \int p (\mathbf {Y} ^ {(2)} | \mathbf {Y} ^ {(1)}, \hat {\mathbf {Y}} ^ {(1)}) p (\hat {\mathbf {Y}} ^ {(1)} | \mathbf {Y} ^ {(1)}) d \hat {\mathbf {Y}} ^ {(0)} \\ \times 2 \pi^ {- \frac {N _ {n} ^ {(2)} D}{2}} \times \left| \hat {\mathbf {S}} ^ {(1)} \right| ^ {- \frac {1}{2}} \times \exp - \frac {1}{2} \left[ \left(\hat {\mathbf {Y}} ^ {(1)} - \bar {\mathbf {Y}} ^ {(1)}\right) ^ {\top} \left(\hat {\mathbf {S}} ^ {(1)}\right) ^ {- 1} \left(\hat {\mathbf {Y}} ^ {(0)} - \bar {\mathbf {Y}} ^ {(0)}\right) \right] \Biggr \} d \hat {\mathbf {Y}} ^ {(1)} \\ = - \frac {\left(N ^ {(2)} + N _ {n} ^ {(2)}\right) D}{2} \log (2 \pi) - \frac {1}{2} \log \left| \mathbf {K} _ {a} ^ {(1 1)} \right| - \frac {1}{2} \log \left| \hat {\boldsymbol {\Sigma}} ^ {(1)} \right| \\ + \log \int \left\{\exp \left[ - \frac {1}{2} \left(\left( \begin{array}{c} \check {\mathbf {Y}} ^ {(2)} \\ \hat {\mathbf {Y}} ^ {(2)} \end{array} \right) - \left( \begin{array}{c} \check {\mathbf {Y}} ^ {(1)} \\ \bar {\mathbf {Y}} ^ {(1)} \end{array} \right)\right) ^ {\top} \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} \left(\left( \begin{array}{c} \check {\mathbf {Y}} ^ {(2)} \\ \hat {\mathbf {Y}} ^ {(2)} \end{array} \right) - \left( \begin{array}{c} \check {\mathbf {Y}} ^ {(1)} \\ \bar {\mathbf {Y}} ^ {(1)} \end{array} \right)\right) \right] \right. \\ \left. - \frac {1}{2} \left[ \left(\hat {\mathbf {Y}} ^ {(1)} - \bar {\mathbf {Y}} ^ {(1)}\right) ^ {\top} \left(\hat {\boldsymbol {\Sigma}} ^ {(1)}\right) ^ {- 1} \left(\hat {\mathbf {Y}} ^ {(1)} - \bar {\mathbf {Y}} ^ {(1)}\right) \right] \right\} d \hat {\mathbf {Y}} ^ {(1)}. \tag {A.46} \\ \end{array}
$$
After that, we try to decompose the vector  $\mathbf{Y}$  as,
$$
\left( \begin{array}{c} \mathbf {\bar {Y}} ^ {(2)} \\ \mathbf {\bar {Y}} ^ {(2)} \end{array} \right) - \left( \begin{array}{c} \mathbf {\bar {Y}} ^ {(1)} \\ \mathbf {\bar {Y}} ^ {(1)} \end{array} \right) = \mathbf {E} ^ {(2)} \mathbf {\bar {Y}} ^ {(2)} - \mathbf {E} ^ {(2)} \mathbf {\bar {Y}} ^ {(1)} + \hat {\mathbf {E}} ^ {(2)} \mathbf {\bar {Y}} ^ {(2)} - \hat {\mathbf {E}} _ {n} ^ {(2)} \mathbf {\hat {Y}} ^ {(1)}. \tag {A.47}
$$
The  $\mathbf{Y}$  is divided into subset parts  $\hat{\mathbf{Y}}$  and non-subset part  $\hat{\mathbf{Y}}$ . Then, we consider the data fitting part by substituting Eq. (A47) into Eq. (A46), which gives the method to make the Eq. (A46) calculable,
where  $\phi = \left(\check{\mathbf{Y}}^{(2)},\hat{\mathbf{Y}}^{(2)}\right)^{\top} - \left(\check{\mathbf{Y}}^{(1)},\mathbf{0}\right)^{\top}$ . After that we try to simplify (A48) by different parts with Sherman-Morrison formula.
First, we consider the determinant part,
$$
\begin{array}{l} - \frac {1}{2} \log \left| \mathbf {K} _ {a} ^ {(1 1)} \right| - \frac {1}{2} \log \left| \hat {\mathbf {S}} ^ {(1)} \right| + \frac {1}{2} \log \left| (\hat {\mathbf {E}} ^ {(2)}) ^ {\top} \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} \hat {\mathbf {E}} ^ {(2)} + \left(\hat {\mathbf {S}} ^ {(1)}\right) ^ {- 1} \right| \\ = -\frac{1}{2}\log \left|\mathbf{K}_{a}^{(11)}\right| - \frac{1}{2}\log \left|\hat{\mathbf{\Sigma}}^{(1)}\right| + \frac{1}{2}\log \left|\mathbf{K}_{a}^{(11)}\right| + \frac{1}{2}\log \left|\hat{\mathbf{\Sigma}}^{(1)}\right| - \frac{1}{2}\log \left|( \mathbf{K}_{a}^{(22)})^{-1} + \hat{\mathbf{E}}^{(2)}\left(\hat{\mathbf{\Sigma}}^{(1)}\right)^{-1}\left(\hat{\mathbf{E}}^{(2)}\right)^{\top}\right| \\ = - \frac {1}{2} \log \left| \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} + \hat {\mathbf {E}} ^ {(2)} \left(\hat {\mathbf {S}} ^ {(1)}\right) ^ {- 1} \left(\hat {\mathbf {E}} ^ {(2)}\right) ^ {\top} \right|. \tag {A.49} \\ \end{array}
$$
Then, we gather terms with  $\phi$  and simplify them as,
$$
\begin{array}{l} - \frac {1}{2} \left(\mathbf {E} ^ {(2)} \tilde {\mathbf {Y}} ^ {(2)} - \mathbf {E} ^ {(2)} \tilde {\mathbf {Y}} ^ {(1)}\right) ^ {\top} \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} \left(\mathbf {E} ^ {(2)} \tilde {\mathbf {Y}} ^ {(2)} - \mathbf {E} ^ {(2)} \tilde {\mathbf {Y}} ^ {(1)}\right) - \frac {1}{2} \left(\mathbf {E} _ {n} ^ {(2)} \hat {\mathbf {Y}} ^ {(2)}\right) ^ {\top} \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} \left(\mathbf {E} _ {n} ^ {(2)} \hat {\mathbf {Y}} ^ {(2)}\right) \\ - \left(\mathbf {E} ^ {(2)} \check {\mathbf {Y}} ^ {(2)} - \mathbf {E} ^ {(2)} \check {\mathbf {Y}} ^ {(1)}\right) ^ {\top} \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} \left(\mathbf {E} _ {n} ^ {(2)} \hat {\mathbf {Y}} ^ {(2)}\right) \\ + \frac {1}{2} \left(\phi^ {\top} \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} \hat {\mathbf {E}} ^ {(2)}\right) \left((\hat {\mathbf {E}} ^ {(2)}) ^ {\top} (\mathbf {K} _ {a} ^ {(2 2)}) ^ {- 1} \hat {\mathbf {E}} ^ {(2)} + \left(\hat {\mathbf {\Sigma}} ^ {(1)}\right) ^ {- 1}\right) ^ {- 1} \left(\left(\hat {\mathbf {E}} ^ {(2)}\right) \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} \phi\right) \\ = - \frac {1}{2} \phi^ {\top} \left[ \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} - \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} \hat {\mathbf {E}} ^ {(2)} \left(\left(\hat {\mathbf {E}} ^ {(2)}\right) ^ {\top} \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} \hat {\mathbf {E}} ^ {(2)} + \left(\hat {\mathbf {\Sigma}} ^ {(1)}\right) ^ {- 1}\right) ^ {- 1} \left(\hat {\mathbf {E}} ^ {(2)}\right) \right] \phi \\ = - \frac {1}{2} \phi^ {\top} \left[ \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} + \hat {\mathbf {E}} ^ {(2)} \left(\hat {\mathbf {S}} ^ {(1)}\right) ^ {- 1} \left(\hat {\mathbf {E}} ^ {(2)}\right) ^ {\top} \right] \phi . \tag {A.50} \\ \end{array}
$$
After that, we consider the interaction part between  $\phi$  and  $\bar{\mathbf{Y}}^{(1)}$ ,
$$
\begin{array}{l} \left(\phi^ {\top} \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} \hat {\mathbf {E}} ^ {(2)}\right) \left((\hat {\mathbf {E}} ^ {(2)}) ^ {\top} \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} \hat {\mathbf {E}} ^ {(2)} + \left(\hat {\mathbf {S}} ^ {(1)}\right) ^ {- 1}\right) ^ {- 1} \left(\left(\hat {\mathbf {S}}\right) ^ {- 1} \bar {\mathbf {Y}} ^ {(1)}\right) \\ = \phi^ {\top} \left[ \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} - \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} \hat {\mathbf {E}} ^ {(2)} \left(\left(\hat {\mathbf {E}} ^ {(2)}\right) ^ {\top} \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} \hat {\mathbf {E}} ^ {(2)} + \left(\hat {\boldsymbol {\Sigma}} ^ {(1)}\right) ^ {- 1}\right) ^ {- 1} \right] \bar {\mathbf {Y}} ^ {(1)} \tag {A.51} \\ = \phi^ {\top} \left[ \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} + \hat {\mathbf {E}} ^ {(2)} \left(\hat {\mathbf {S}} ^ {(1)}\right) ^ {- 1} \left(\hat {\mathbf {E}} ^ {(2)}\right) ^ {\top} \right] \hat {\mathbf {E}} ^ {(2)} \bar {\mathbf {Y}} ^ {(1)}. \\ \end{array}
$$
Finally, we simplify the terms related with  $\bar{\mathbf{Y}}^{(1)}$
$$
\begin{array}{l} - \frac {1}{2} \left(\bar {\mathbf {Y}} ^ {(1)}\right) ^ {\top} \left(\hat {\boldsymbol {\Sigma}} ^ {(1)}\right) ^ {- 1} \left(\bar {\mathbf {Y}} ^ {(1)}\right) \\ + \frac {1}{2} \left(\left(\bar {\mathbf {Y}} ^ {(1)}\right) ^ {\top} \left(\hat {\boldsymbol {\Sigma}}\right) ^ {- 1}\right) \left(\left(\hat {\mathbf {E}} ^ {(2)}\right) ^ {\top} \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} \hat {\mathbf {E}} ^ {(2)} + \left(\hat {\boldsymbol {\Sigma}} ^ {(1)}\right) ^ {- 1}\right) ^ {- 1} \left(\left(\hat {\boldsymbol {\Sigma}}\right) ^ {- 1} \bar {\mathbf {Y}} ^ {(1)}\right) \\ = - \frac {1}{2} \left(\bar {\mathbf {Y}} ^ {(1)}\right) ^ {\top} \left[ \left(\hat {\boldsymbol {\Sigma}} ^ {(1)}\right) ^ {- 1} - \left(\hat {\boldsymbol {\Sigma}}\right) ^ {- 1} \left(\left(\hat {\mathbf {E}} ^ {(2)}\right) ^ {\top} \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} \hat {\mathbf {E}} ^ {(2)} + \left(\hat {\boldsymbol {\Sigma}} ^ {(1)}\right) ^ {- 1}\right) ^ {- 1} \left(\hat {\boldsymbol {\Sigma}}\right) ^ {- 1} \right] \left(\bar {\mathbf {Y}} ^ {(1)}\right) \\ = - \frac {1}{2} \left(\bar {\mathbf {Y}} ^ {(1)}\right) ^ {\top} \left(\hat {\mathbf {E}} ^ {(2)}\right) ^ {\top} \left[ \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} + \hat {\mathbf {E}} ^ {(2)} \left(\hat {\mathbf {S}} ^ {(1)}\right) ^ {- 1} \left(\hat {\mathbf {E}} ^ {(2)}\right) ^ {\top} \right] \hat {\mathbf {E}} ^ {(2)} \left(\bar {\mathbf {Y}} ^ {(1)}\right). \tag {A.52} \\ \end{array}
$$
Putting all the parts together, the joint likelihood for the  $R = 1$  is,
$$
\begin{array}{l} \mathcal {L} = \log p (\mathbf {Y} ^ {(0: 2)}) = \log p (\mathbf {Y} ^ {(0)}, \mathbf {Y} ^ {(1)}, \mathbf {Y} ^ {(2)}) \\ = \log p (\mathbf {Y} ^ {(0)}) + \log p (\mathbf {Y} ^ {(1)} | \mathbf {Y} ^ {(0)}) + \log \int p (\mathbf {Y} ^ {(2)} | \mathbf {Y} ^ {(1)}, \hat {\mathbf {Y}} ^ {(1)}) p (\hat {\mathbf {Y}} ^ {(1)} | \mathbf {Y} ^ {(1)}) d \hat {\mathbf {Y}} ^ {(1)} \\ = \log p (\mathbf {Y} ^ {(0)}) + \log p (\mathbf {Y} ^ {(1)} | \mathbf {Y} ^ {(0)}) + \log \int p (\mathbf {Y} ^ {(2)} | \mathbf {Y} ^ {(1)}, \hat {\mathbf {Y}} ^ {(1)}) p (\hat {\mathbf {Y}} ^ {(1)} | \mathbf {Y} ^ {(1)}) d \hat {\mathbf {Y}} ^ {(0)} \\ = \log p (\mathbf {Y} ^ {(0)}) + \log p (\mathbf {Y} ^ {(1)} | \mathbf {Y} ^ {(0)}) - \frac {N ^ {(2)} D}{2} \log (2 \pi) - \frac {1}{2} \log \left| \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} + \hat {\mathbf {E}} ^ {(2)} \left(\hat {\mathbf {S}} ^ {(1)}\right) ^ {- 1} \left(\hat {\mathbf {E}} ^ {(2)}\right) ^ {\top} \right| \\ - \frac {1}{2} \left(\mathbf {Y} _ {a} ^ {(2)}\right) ^ {\top} \left[ \left(\mathbf {K} _ {a} ^ {(2 2)}\right) ^ {- 1} + \hat {\mathbf {E}} ^ {(2)} \left(\hat {\mathbf {S}} ^ {(1)}\right) ^ {- 1} \left(\hat {\mathbf {E}} ^ {(2)}\right) ^ {\top} \right] \mathbf {Y} _ {a} ^ {(2)}. \tag {A.53} \\ \end{array}
$$
Therefore, we can see that the joint likelihood of three different fidelities with non-subset dataset can be decomposed as three independent model to train. For problems with arbitrary number of fidelity, we can easily apply this conclusion recursively to decompose them into the summation structure, which is scalable to the number of training data and fidelity level and is easy to code up.
J Experiment in Detail
J.1 Canonical PDEs
Three canonical PDEs are under consideration: Poisson's equation, Heat equation, and Burger's equation. These equations play pivotal roles in scientific and technological applications [41-43]. They present typical simulation scenarios that include high-dimensional spatial-temporal field outputs, nonlinearities, and discontinuities. These scenarios are frequently used as benchmark problems for surrogate models [28, 37-39].  $x$  and  $y$  denote the spatial coordinates, with  $t$  representing the time coordinate, a departure from the notation used in the primary paper. The notation in the appendix is employed solely to improve comprehension and does not influence or correlate with the main text.
Let's start with Burger's equation, a renowned nonlinear hyperbolic PDE commonly used to depict various physical phenomena such as fluid dynamics [42], nonlinear acoustics [48], and traffic flows [49]. It is often employed as a standard benchmark for multiple numerical solvers and surrogate models [50, 51] due to its ability to generate discontinuities (shock waves) based on a standard conservation equation. Here is the equation in its viscous form,
$$
\frac {\partial u}{\partial t} + u \frac {\partial u}{\partial x} = v \frac {\partial^ {2} u}{\partial x ^ {2}}.
$$
Within the context,  $u$  represents volume,  $x$  pertains to spatial location,  $t$  signifies time, and  $v$  stands for viscosity. The parameters are defined as follows:  $x$  ranges between 0 and 1 (in meters),  $t$  spans from 0 to 3 (in seconds), and the initial condition is set as  $u(x,0) = \sin (x\pi /2)$  with homogeneous Dirichlet boundary conditions. We conducted uniform sampling of viscosities  $v$  within the interval of [0.001,0.1] (in milliPascal seconds) as the input parameter for generating the solution field.
Solving the problem in the space and time domains involves the utilization of finite elements employing hat functions and backward Euler, respectively. To produce solutions at various fidelities, the solver addresses the PDEs by employing discretized spatial-temporal domains consisting of regular rectangular meshes with grid points ranging from  $8^{2}$ ,  $16^{2}$ ,  $32^{2}$ ,  $64^{2}$ ,  $128^{2}$ , facilitating simulations from low to high fidelity.
Next, the focus shifts to Poisson's equation, a common elliptic PDE employed in mechanical engineering and physics to simulate potential fields, including gravitational and electrostatic fields [41]. Expressed in the form of
$$
\frac {\partial^ {2} u}{\partial x ^ {2}} + \frac {\partial^ {2} u}{\partial y ^ {2}} = 0.
$$
It represents an extension of Laplace's equation [53]. Encountered frequently in physics, Poisson's equation, despite its simplicity, serves as a fundamental test case for surrogate models, as highlighted in various studies [B7, 54]. Within our experiment, we apply Dirichlet boundary conditions to a 2D spatial domain, where  $\mathbf{x}$  ranges over  $[0,1] \times [0,1]$ . The input parameters involve fixed values assigned to the four boundaries and the central point of the rectangular domain, each ranging from 0.1 to 0.9. To generate the matching potential fields as outputs, we sampled the input parameters uniformly. Employing a first-order center differencing scheme and regular rectangular meshes, we solved the PDE using the finite difference approach. Simulations of five different fidelities were generated using meshes containing grid nodes  $8^2$ ,  $16^2$ ,  $32^2$ ,  $64^2$ ,  $128^2$ .
Heat equation, established in 1822 to depict time-dependent changes in heat fluxes, remains a fundamental PDE. Its scope extends beyond heat fluxes, finding applications in diverse scientific domains such as probability
Figure 8: Geometry, boundary conditions, and simulation parameters for cantilever beam
theory [55, 43] and financial mathematics [56]. As a result, it is widely adopted as a representative model. Here is the heat equation:
$$
\frac {\partial}{\partial x} \left(k \frac {\partial T}{\partial x}\right) + \frac {\partial}{\partial y} \left(k \frac {\partial T}{\partial y}\right) + \frac {\partial}{\partial z} \left(k \frac {\partial T}{\partial z}\right) + q _ {V} = \rho c _ {p} \frac {\partial T}{\partial t},
$$
where  $k$  is the materials conductivity  $q_{V}$  is the rate at which energy is generated per unit volume of the medium  $\rho$  is the density and  $c_{p}$  is the specific heat capacity. The flux rate at the left boundary, ranging from 0 to 1 at  $x = 0$ , the flux rate at the right boundary, ranging from -1 to 0 at  $x = 1$ , and the thermal conductivity, ranging from 0.01 to 0.1, serve as the input parameters. Within a 2D spatial-temporal domain  $x \in [0,1]$  and  $t \in [0,5]$ , we set up the Neumann boundary condition at  $x = 0$  and  $x = 1$ . Additionally, we define  $u(x,0)$  as  $H(x - 0.25) - H(x - 0.75)$ , where  $H(\cdot)$  represents the Heaviside step function. Using finite difference in space and backward Euler in time domains, the equation is solved. The spatial-temporal domain is discretized into a  $16 \times 16$  regular rectangular mesh for the first (lowest) fidelity solver, while a refined solver employs a  $32 \times 32$  mesh for the second fidelity. The computed result fields are on a  $100 \times 100$  spatial-temporal grid. Solving the equation involves applying a finite difference approach in the spatial domain and employing reverse Euler in the temporal domain. We discretize the spatial-temporal domain into a regular rectangular mesh, using  $8^{2}, 16^{2}, 32^{2}, 64^{2}, 128^{2}$  nodes to produce simulation results at five varying fidelity levels.
J.2 Multi-Fidelity Fusion for Topology Optimization
In a topology structure optimization problem, we employ ContinuAR to determine the optimal topology structure, maximizing mechanical metrics such as stiffness, for a material layout comprising alloys and concrete. This is accomplished by considering various design parameters including external force and angle. Particularly with the advancements in 3D printing methods, where materials are deposited in small increments, topology structure optimization has become a crucial technique in mechanical design, applied in areas like airfoils and slab bridges. Notably, the computational intensity of topology optimization stems from the need for gradient-based optimization and mechanical simulations. The demanding nature of a high-fidelity solution, requiring extensive discretization mesh and imposing substantial computational burdens in both space and time, exacerbates the situation.
Gaining popularity is the utilization of data-driven methods that offer suitable structures to facilitate the process [57]. In this study, we examine the topology optimization of a cantilever beam. Utilizing the efficient implementation [58], we conduct density-based topology optimization, aiming to minimize the compliance  $C$  while adhering to volume constraints  $V \leq \bar{V}$ .
Utilizing the SIMP scheme [59], we convert continuous density measurements into discrete, optimal topologies. System inputs include the position of point load  $P1$ , the angle of point load  $P2$ , and the filter radius  $P3$  [60]. The issue is addressed using a standard mesh containing nodes in the set  $32^2, 40^2, 48^2, 56^2, 64^2$ .
J.3 Multi-Fidelity Fusion for Plasmonic nanoparticle arrays