SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wei W. Xing *
School of Mathematics and Statistics, University of Sheffield
Hicks Building, Hounsfield Rd, Sheffield, UK, S3 7RH
w.xing@sheffield.ac.uk
Yuxin Wang
Department of Statistics and Data Science National University of Singapore
21 Lower Kent Ridge Road, Singapore, 119077.
yuxinwang@u.nus.edu
Zheng Xing
Graphics& Computing Department
Rockchip Electronics Co., Ltd
Fuzhou, China, 350003.
zheng.xing@rock-chips.com
Multi-fidelity fusion has become an important surrogate technique, which provides insights into expensive computer simulations and effectively improves decision-making, e.g., optimization, with less computational cost. Multi-fidelity fusion is much more computationally efficient compared to traditional single-fidelity surrogates. Despite the fast advancement of multi-fidelity fusion techniques, they lack a systematic framework to make use of the fidelity indicator, deal with high-dimensional and arbitrary data structure, and scale well to infinite-fidelity problems. In this work, we first generalize the popular autoregression (AR) to derive a novel linear fidelity differential equation (FiDE), paving the way to tractable infinite-fidelity fusion. We generalize FiDE to a high-dimensional system, which also provides a unifying framework to seemingly bridge the gap between many multi- and single-fidelity GP-based models. We then propose ContinuAR, a rank-1 approximation solution to FiDEs, which is tractable to train, compatible with arbitrary multi-fidelity data structure, linearly scalable to the output dimension, and most importantly, delivers consistent SOTA performance with a significant margin over the baseline methods. Compared to the SOTA infinite-fidelity fusion, IFC, ContinuAR achieves up to 4x improvement in accuracy and 62,500x speedup in training time.
Contemporary scientific and engineering endeavors depend significantly on analyzing highly complex systems, where the repeated execution of large-scale differential equation numerical simulations, often with intricate interconnections, is essential. For instance, in the design of a system-on-chip (SoC), more than  $80\%$  of the design time is spent on analysis based on different types of simulations, e.g., timing analysis and yield optimization. Such an intense computational demand prompts the use of a data-driven surrogate model, which essentially acts as a functional approximation for the input-output mapping of a simulation. To enhance convergence efficiency, especially in scenarios involving recurrent simulations, such as those encountered in Bayesian optimization (BO) [1] and uncertainty quantification (UQ) [2].
Traditional surrogates are trained on many high-fidelity simulation results, which are still computationally expensive to generate. To make high-fidelity predictions [3] while further reducing the computational burden, it is a possible way to combine low-fidelity results. More specifically, we can run low-fidelity simulations based on simplified equations (e.g., reducing the levels of physical detail) or coarse solver setups (e.g., using a coarse mesh, a large time step, a lower order of approximating basis, and a higher error tolerance) to generate cheap but inaccurate results, which offer a chance to train a rough model with low cost. The multi-fidelity fusion techniques then improve such a rough model by using only a few high-fidelity simulation samples. In total, the computational cost is dramatically reduced. Multi-fidelity fusion, owing to its efficiency, has garnered growing interest in BO [4, 5], UQ [6], and surrogate modeling [7].
While many state-of-the-art (SOTA) fusion approaches have been rapidly developing, most focus on improving the model accuracy or scalability for large-scale problems. They normally assume a small number of fidelities (say, less than five), and a particular fidelity data structure (i.e., the output space is well aligned and the high-fidelity samples' corresponding inputs must form a subset of the low-fidelity inputs). However, in practice, a natural multi-fidelity problem can be much more complicated. For instance, the data does not admit any particular structure. Furthermore, the number of fidelities is in general countably infinite, very few works actually utilize the fact that the fidelity levels are implicitly quantified by a continuous variable, e.g., the number of nodes in a mesh and there are infinite number of fidelities. These limitations seriously hinder the applications of multi-fidelity-based methods, e.g., multi-fidelity BO and multi-fidelity Monte Carlo.
Recently, Li et al. [8] propose the first infinite-fidelity fusion, IFC, which utilizes NeuralODE [9] to resolve these challenges. Despite its success, IFC is difficult to train and scale poorly to high-dimensional problems. To make a further step towards practicality while preserving tractability and accuracy, we propose the first tractable infinite-fidelity fusion framework, linear fidelity differential equations (FiDEs), and its rank-1 solution, ContinuAR, to deliver a powerful yet tractable fusion model. The novelty of this work is as follows,
1. We propose the first linear fidelity differential equation (FiDE) and its general solution, paving the way to tractable infinite-fidelity fusion.
2. We extend FiDE to FiDEs, which handle the common high-dimensional simulation problems. Furthermore, FiDEs bridge the gap between multi-fidelity and single-fidelity surrogates and serve as a unifying framework for many existing GP-based surrogates, revealing some future directions for classic surrogate models (a.k.a emulators) with multi-fidelity fusion.
3. We propose ContinuAR, a tractable and efficient rank-1 solution to FiDEs. It is scalable to infinite fidelity, capable of handling high-dimension problems, compatible with arbitrary multi-fidelity data structure, and delivering SOTA accuracy with low computational cost.
2.1 Statement of the problem
Provided with multi-fidelity dataset  $\mathcal{D}^i$ , where  $i = 0, \dots, T$ , each set comprising entries  $\{t_i, \mathbf{x}_n^{(i)}, \mathbf{y}_n^{(i)}\}_{n=1}^{N^i}$ , with  $t_i$  representing the fidelity indicator (e.g., the number of nodes in a mesh generation), where a larger  $t$  indicates a more accurate solution;  $\mathbf{x}^{(i)} \in \mathbb{R}^Q$  denotes the system inputs (e.g., a vector containing parameters found in the system of equations or initial-boundary conditions for a simulation), where  $Q$  is the dimensionality for  $\mathbf{x}$ ;  $\mathbf{y}^{(i)} \in \mathbb{R}^D$  indicates the vectorized outputs associated with  $\mathbf{x}$ , where  $D$  is the dimensionality for  $\mathbf{y}$ ;  $T$  is the total number of fidelities. In general, higher fidelity simulations, being closer to the ground truth and more costly to acquire, result in a limited number of available samples, i.e.,  $N^0 > N^1 > \dots > N^T$ . In most works, e.g., [M-D], the system inputs of higher-fidelity are chosen to be the subset of the lower-fidelity, i.e.,  $\mathbf{X}^T \subset \dots \subset \mathbf{X}^2 \subset \mathbf{X}^1$ . We call this the subset structure for a multi-fidelity dataset, as opposed to arbitrary data structures, which we will resolve in Section 4. Our objective is to estimate the function  $\mathbf{y}^{(T)}(\mathbf{x})$  based on observations at various fidelities  $\{\mathcal{D}^i\}_{i=1}^T$ .
2.2 Autoregression (AR)
The classic AR model [8] only considers the scalar problem (i.e.,  $y^{(i)}\in \mathbb{R}$ ) and imposes a Markov property for its multi-fidelity formulation. Considering that  $t_0$  and  $t_T$  are indicators (e.g., time steps) for the low- and high-fidelity data, respectively, AR defines
$$
y ^ {(T)} (\mathbf {x}) = \rho y ^ {(0)} (\mathbf {x}) + u (\mathbf {x}), \tag {1}
$$
where  $\rho$  is a factor transferring knowledge from the low-fidelity in a linear fashion, while  $u(\mathbf{x})$  aims to encapsulate the remaining information. Assuming a zero mean Gaussian process (GP) prior [13] for  $y^{(0)}(\mathbf{x})$  and  $u(\mathbf{x})$ , denoted as  $y^{(0)}(\mathbf{x}) \sim \mathcal{N}(0,k^0 (\mathbf{x},\mathbf{x}'))$  and  $u(\mathbf{x}) \sim \mathcal{N}(0,k^u (\mathbf{x},\mathbf{x}'))$ , the high-fidelity function similarly adheres to a GP. This results in an elegant joint GP applicable to the combined observations  $\mathbf{y} = [\mathbf{y}^{(0)};\mathbf{y}^{(T)}]$ ,
$$
\left( \begin{array}{c} \mathbf {y} ^ {(0)} \\ \mathbf {y} ^ {(T)} \end{array} \right) \sim \mathcal {N} \left(\mathbf {0}, \begin{array}{c c} \mathbf {K} _ {0} ^ {(0)} & \rho \mathbf {K} _ {0} ^ {(0 T)} \\ \rho \mathbf {K} _ {0} ^ {(T 0)} & \rho^ {2} \mathbf {K} _ {0} ^ {(T)} + \mathbf {K} _ {u} ^ {(T)} \end{array} \right) \tag {2}
$$
where  $[\mathbf{K}_0^{(0)}]_{ij} = k^0 (\mathbf{x}_i^{(0)},\mathbf{x}_j^{(0)})$  .  $[\mathbf{K}_0^{(0T)}]_{ij} = k^0 (\mathbf{x}_i^{(0)},\mathbf{x}_j^{(T)})$  .  $[\mathbf{K}_0^{(T0)}]_{ij} = k^0 (\mathbf{x}_i^{(T)},\mathbf{x}_j^{(0)})$  ..  $[\mathbf{K}_0^{(T)}]_{ij} = k^0 (\mathbf{x}_i^{(T)},\mathbf{x}_j^{(T)})$  .  $[\mathbf{K}_u^{(T)}]_{ij} = k^u (\mathbf{x}_i^{(T)},\mathbf{x}_j^{(T)})$  . We can see that the joint likelihood is still Gaussian, which admits a tractable solution for model training as in a standard GP. Furthermore, the predictive posterior is also a standard GP posterior, which can effectively utilize low- and high-fidelity data. Moreover, by decomposing the likelihood and predictive posterior into two separate components, Le Gratiet [10] managed to reduce the complexity from  $O((N^0 +N^T)^3)$  to  $O((N^0)^3 +(N^T)^3)$  using a subset data structure, where  $\mathbf{X}^T\subset \mathbf{X}^0$
2.3 Fidelity Differential Equation
In scientific numerical simulations, the fidelity can be determined by quite different factors, e.g., using simplified/complete equations to describe the target system or implementing a dense/sparse mesh to discretize the domain. For problems where the fidelity factor cannot be easily quantified, the classic multi-fidelity methods, e.g., AR, and other methods should work just fine. However, it is more frequent to find that the fidelity indicators have valuable information in them. For instance, the fidelity is often indicated by the number of elements in a finite element simulation or time steps in a forward/backward Euler timing scheme. In these cases, we should treat the fidelity indicators as continuous variables and utilize the information they carry to boost the predictive accuracy. To this end, Li et al. [8] propose a general formulation of infinite-fidelity models,
$$
\mathrm {d} y (\mathbf {x}, t) / \mathrm {d} t = \phi (\mathbf {x}, t, y (\mathbf {x}, t)). \tag {3}
$$
Although this formulation is general enough to cover all multi-fidelity models, it lacks an insightful interpretation—it is not clear how to design the function  $\phi(\mathbf{x}, t, y(\mathbf{x}, t))$ . A clever workaround is to use a neural network to approximate  $\phi(\mathbf{x}, t, y(\mathbf{x}, t))$  [8], which introduces many challenges such as 1) requiring a large amount of training data, 2) expensive computational cost for backpropagation even with adjoint method, and 3) instability as we can see how easy a simple nonlinear system can lead to chaotic behaviors (e.g., the Lorenz system [14]).
3.1 Linear Fidelity Differential Equation
To remedy the over-general formulation of Eq. (B), we first revisit the classic AR and reveal its connection to a linear ODE's forward Euler solution. Rewrite Eq. (B) as
$$
\frac {y (\mathbf {x} , t _ {T}) - y (\mathbf {x} , t _ {0})}{t _ {T} - t _ {0}} = \beta_ {0} y (\mathbf {x}, t _ {0}) + u (\mathbf {x}, t _ {0}), \tag {4}
$$
and take the limit  $(t_T - t_0) \to 0$ , we derive Proposition II (see Appendix B for detailed derivations).
Proposition 1. The linear fidelity differential equation (FiDE):
$$
\dot {y} (\mathbf {x}, t) = \beta (t) y (\mathbf {x}, t) + u (\mathbf {x}, t). \tag {5}
$$
If we take a forward Euler solution to solve Eq. (5) discretized at each fidelity, we recover the classic AR in Eq. (II). Let us stick to the continuous formulation and utilize calculus techniques [15], we can derive the general solution to Eq. (5),
$$
y (\mathbf {x}, t) = e ^ {- B (t)} \left(\int^ {t} e ^ {B (\lambda)} u (\mathbf {x}, \lambda) \mathrm {d} \lambda + C (\mathbf {x})\right), \tag {6}
$$
where  $C(\mathbf{x})$  is a function of  $\mathbf{x}$ , and  $B(t) = \int^{t} \beta(\tau) \mathrm{d}\tau$  is the antiderivative of  $\beta(t)$ . In order to design  $\beta(t)$  such that the solution converges to the ground truth as  $t \to \infty$ , a detailed stability analysis is necessitated. General stability considerations for systems of this kind are often complex and may warrant the application of Lyapunov's direct method. For the sake of maintaining a tractable model, we make the simplifying assumption that  $\beta(t) = \beta$  is a constant value. Provided that  $\beta > 0$ , our system is guaranteed to converge to what we define as the "ultra-high-fidelity ground truth" as
$t\to \infty$  . Setting  $t_0$  as the lowest-fidelity indicator, we derive a general solution (see Appendix C)
$$
y (\mathbf {x}, t) = y (\mathbf {x}, t _ {0}) e ^ {- \beta (t - t _ {0})} + \int_ {t _ {0}} ^ {t} e ^ {- \beta (t - \tau)} u (\mathbf {x}, \tau) d \tau . \tag {7}
$$
To design a subtle model with enough flexibility while trying to maintain tractability, we place zero mean GP priors for  $\mathbf{y}(\mathbf{x},t_0)$  and  $u(\mathbf{x},t)$  (with all data being normalized), i.e.,
$$
\mathbf {y} (\mathbf {x}, t _ {0}) \sim (0, k ^ {0} (\mathbf {x}, \mathbf {x} ^ {\prime})), \quad u (\mathbf {x}, t) \sim \mathcal {N} \left(0, k ^ {u} (\mathbf {x}, t, \mathbf {x} ^ {\prime}, t ^ {\prime})\right). \tag {8}
$$
Similarly to the latent force model [16], the resulting general solution  $y(\mathbf{x}, t)$  is a also GP with zero mean and covariance with the analytical form:
$$
\operatorname {C o v} \left[ y \left(\mathbf {x}, t\right), y \left(\mathbf {x} ^ {\prime}, t ^ {\prime}\right) \right] = k ^ {0} \left(\mathbf {x}, \mathbf {x} ^ {\prime}\right) e ^ {- \beta \left(t - t _ {0}\right)} + \int_ {t _ {0}} ^ {t} e ^ {- \beta (t - \tau)} \int_ {t _ {0}} ^ {t ^ {\prime}} e ^ {- \beta \left(t ^ {\prime} - \tau^ {\prime}\right)} k ^ {u} \left(\mathbf {x}, \tau , \mathbf {x} ^ {\prime}, \tau^ {\prime}\right) d \tau^ {\prime} d \tau . \tag {9}
$$
Alvarez et al. [16] demonstrate that a simple ODE system, devoid of  $\mathbf{x}$ , yields a nonstationary output covariance with a closed-form solution when employing a squared-exponential (SE) kernel for  $u(t)$ . This result, however, is not directly applicable to the FiDE due to the inherent  $\mathbf{x}$  dependency.
To improve model flexibility, we enable the usage of arbitrary kernels, e.g., deep kernel [17], by implementing a Monte-Carlo integration and reparameterization trick [18] to conduct efficient backpropagation for model training—the integral approximated by  $\frac{1}{M}\sum_{i,j}^{M}e^{-\beta (t_i + t_j - 2t_0)}k^u (\mathbf{x},\tau_i,\mathbf{x}',\tau_j)$ , where  $t_i$  and  $t_j$  are  $M$  are random samples from  $[t_0,t]$  and  $[t_0,t']$ , respectively.
3.2 FiDEs: Multi-Variate Extension For High-dimensional Problems
In practice, the outputs of a simulation are generally high-dimensional [0], i.e.,  $\mathbf{y}^{(t)}\in \mathbb{R}^D$ . We hereby generalize Eq. (5) for a more general formulation that describes a multi-variate system:
Proposition 2. The linear fidelity differential equations (FiDEs) for a multi-variate system:
$$
\dot {\mathbf {y}} (\mathbf {x}, t) + \mathbf {B} (t) \mathbf {y} (\mathbf {x}, t) = \mathbf {S u} (\mathbf {x}, t), \tag {10}
$$
where  $\mathbf{B}(t) \in \mathbb{R}^{D \times D}$  and  $\mathbf{S} \in \mathbb{R}^{D \times R}$  are affine transformations;  $\mathbf{u}(\mathbf{x}, t) \in \mathbb{R}^R$  is a source function.
We can derive a general solution for Eq. (III) with some calculus as model design guideline (see Appendix D). For instance, for a constant  $\mathbf{B}$ , all eigenvalues of  $\mathbf{B}$  must have positive real parts to keep the system stable. To derive an efficient model, we define a constant diagonal matrix  $\mathbf{B}$  and derive a tractable general solution
$$
y _ {d} (\mathbf {x}, t) = C _ {d} (\mathbf {x}) e ^ {- B _ {d} t} + \sum_ {, r = 1} ^ {R} S _ {d, r} \mathcal {G} _ {d} \left(u _ {r} (\mathbf {x}, t)\right), \tag {11}
$$
$$
\text {w h e r e} \quad \mathcal {G} _ {d} \left(u _ {r} (\mathbf {x}, t)\right) = \int_ {t _ {0}} ^ {t} e ^ {- B _ {d} (t - \tau)} u _ {r} (\mathbf {x}, \tau) \mathrm {d} \tau , \tag {12}
$$
with  $B_{d}$  being the  $d$ -diagonal element of  $\mathbf{B}$ . Due to the linearity of the integral in  $\mathcal{G}_d(u_r(\mathbf{x},t))$ , we can derive the output correlation with an analytical form:
$$
\operatorname {C o v} \left[ y _ {d} (\mathbf {x}, t), y _ {d ^ {\prime}} \left(\mathbf {x} ^ {\prime}, t ^ {\prime}\right) \right] = e ^ {- \beta (t - t _ {0})} k ^ {0} \left(\mathbf {x}, \mathbf {x} ^ {\prime}\right) H _ {d, d ^ {\prime}} + \sum_ {r = 1} ^ {R} S _ {d, r} S _ {d ^ {\prime}, r} \operatorname {C o v} \left[ \mathcal {G} _ {d} \left(u _ {r} (\mathbf {x}, t)\right), \mathcal {G} _ {d ^ {\prime}} \left(u _ {r} \left(\mathbf {x} ^ {\prime}, t ^ {\prime}\right)\right) \right], \tag {13}
$$
$$
\operatorname {C o v} \left[ \mathcal {G} _ {d} \left(u _ {r} (\mathbf {x}, t)\right), \mathcal {G} _ {d ^ {\prime}} \left(u _ {r} \left(\mathbf {x} ^ {\prime}, t ^ {\prime}\right)\right) \right] = \int_ {t _ {0}} ^ {t} e ^ {- B _ {d} (t - \tau)} \int_ {t _ {0}} ^ {t ^ {\prime}} e ^ {- B _ {d ^ {\prime}} \left(t ^ {\prime} - \tau^ {\prime}\right)} k ^ {u _ {r}} \left(\mathbf {x}, \tau , \mathbf {x} ^ {\prime}, \tau^ {\prime}\right) d \tau^ {\prime} d \tau . \tag {14}
$$
We recognize that this model is a generalization of semiparametric latent factor model (SLFM) [20], which is a special case by setting  $t = t_0$  or  $B_d = 0$  to consider a single-fidelity problem.
3.3 ContinuAR: A Rank-1 Solution to FiDEs
A special yet practical case of SLFM is the intrinsic model of coregionalization (IMC, also a rank-1 approximation to LMC [2]), where all  $u_{r}(\mathbf{x},t)$  share the same kernel function, i.e.,
$$
\mathbf {u} (\mathbf {x}, t) \sim \mathcal {N} (\mathbf {0}, \mathbf {k} ^ {u} (\mathbf {x}, t, \mathbf {x} ^ {\prime}, t ^ {\prime}) \otimes \mathbf {I}). \tag {15}
$$
Here  $\otimes$  means the Kronecker product. Similar IMC assumptions are made in ResGP, a popular modification of AR for high-dimensional output fusion [12], which shows promising results in
many physics applications. Based on their conclusions and our intention to keep our model simple and efficient, we define  $\mathbf{B} = \beta \mathbf{I}$  following our solution to FiDE and place an IMC model for the lowest-fidelity model, i.e.,  $\mathbf{y}(\mathbf{x},t_0)\sim \mathcal{N}(\mathbf{0},k^0 (\mathbf{x},\mathbf{x}')\otimes \mathbf{H})$ , where  $\mathbf{H}$  is the output correlations. Substituting the new  $\mathbf{u}(\mathbf{x},t)$  and  $\mathbf{B}$  back into Eq. (13), we get the output correlation  $k^{y}(\mathbf{x},t,\mathbf{x}',t') = \mathrm{Cov}[\mathbf{y}(\mathbf{x},t),\mathbf{y}(\mathbf{x}',t')] =$
$$
e ^ {- \beta (t - t _ {0})} k ^ {0} (\mathbf {x}, \mathbf {x} ^ {\prime}) \otimes \mathbf {H} + \int_ {t _ {0}} ^ {t} e ^ {- \beta (t - \tau)} \int_ {t _ {0}} ^ {t ^ {\prime}} e ^ {- \beta \left(t ^ {\prime} - \tau^ {\prime}\right)} k ^ {u} (\mathbf {x}, \tau , \mathbf {x} ^ {\prime}, \tau^ {\prime}) d \tau^ {\prime} d \tau \otimes \mathbf {S S} ^ {\top}, \tag {16}
$$
which uniquely defines a rank-1 solution to FiDEs. We call it ContinuAR.
Lemma 1. Autokrigeability in ContinuAR: the particular values of the spatial correlation matrix  $\mathbf{H}$  and  $\mathbf{S}\mathbf{S}^{\top}$  do not matter in the predictive mean as they will be canceled out.
The proof is given in Appendix E for clarity by basically deriving the predictive mean of ContinuAR. Since the predictive mean is the main concern in high-dimensional problems [8, 12], we simply set  $\mathbf{H} = \mathbf{S}\mathbf{S}^{\top} = \mathbf{I}$  to significantly improve model efficiency without introducing any additional error in the mean predictions. The computational complexity w.r.t the output dimension is reduced from  $\mathcal{O}(D^3)$  to  $\mathcal{O}(D)$  and the memory consumption is reduced from  $\mathcal{O}(D^2)$  to  $\mathcal{O}(1)$ .
4 Efficient Training and Inference Through Subset Decomposition
We have implicitly marginalized out the underlying function  $\mathbf{u}(\mathbf{x},t)$  based on its GP prior and derived the output covariance of Eq. (16). Given a set of observations  $\mathbf{Y} = [\mathbf{Y}^{(0)};\mathbf{Y}^{(1)};\dots ;\mathbf{Y}^{(T)}]$ , we have the joint distribution
$$
\left( \begin{array}{c} \vec {\mathbf {y}} ^ {(0)} \\ \vdots \\ \vec {\mathbf {y}} ^ {(T)} \end{array} \right) \sim \mathcal {N} \left(\mathbf {0}, \begin{array}{c c c c} \mathbf {K} ^ {(0 0)} & \dots & \mathbf {K} ^ {(0 T)} \\ \vdots & \ddots & \vdots \\ \mathbf {K} ^ {(T 0)} & \dots & \mathbf {K} ^ {(T T)} \end{array} \right), \tag {17}
$$
where  $\vec{\mathbf{y}}^{(0)} = \mathrm{vec}\left(\mathbf{Y}^{(0)}\right)$  is the vectorization;  $[\mathbf{K}^{(kl)}]_{ij} = k^y (\mathbf{x}_i,t_k,\mathbf{x}_j,t_l)$  is the shorthand notation of output correlation Eq. (16). For a small number of total training data of all fidelity, we can simply opt for a maximum likelihood estimation (MLE) for the joint likelihood
$$
\mathcal {L} = - \frac {1}{2} \vec {\mathbf {y}} ^ {\top} \boldsymbol {\Sigma} ^ {- 1} \vec {\mathbf {y}} - \frac {1}{2} \log | \boldsymbol {\Sigma} | - \frac {N D}{2} \log (2 \pi), \tag {18}
$$
where  $\boldsymbol{\Sigma}$  is the whole covariance matrix in Eq. (17) and  $\vec{\mathbf{y}} = [\mathbf{y}^{(0)},\dots,\mathbf{y}^{(T)}]^{\top}$ . However, this approach will soon become invalid, particularly in a multi-fidelity scenario where we expect many low-fidelity simulations.
Since the integration of Eq. (16) can be done by parts,  $\mathbf{K}^{(kl)}$  admits an additive structure exactly as in AR. We can follow Gratiet and Cannamela [22] to decompose the joint likelihood Eq. (18) into independent components provided that corresponding inputs strictly follow a subset structure, i.e.,  $\mathbf{X}^T\subset \dots \subset \mathbf{X}^2\subset \mathbf{X}^1$ . For problems with only a small number of fidelity (a small  $T$ ), the subset structure may not be too difficult to satisfy. However, for the infinite (countable) fidelity setting, such a requirement is not practical. Here, we derive a decomposition by introducing virtual observations  $\hat{\mathbf{Y}}$  for each fidelity such that  $\mathbf{Y}^{(T)}$  satisfies the subset requirement for the completed set  $\{\mathbf{Y}^{(T - 1)},\hat{\mathbf{Y}}^{(T - 1)}\}$ .  $\check{\mathbf{Y}}^{(T)}$  is the part of  $\mathbf{Y}^{(T)}$  that forms the subset of  $\mathbf{Y}^{(T - 1)}$  (with a selection formulation  $\tilde{\mathbf{X}}^{(T)} = \mathbf{E}^{(T)}\mathbf{X}^{(T - 1)}$ , where  $\mathbf{X}^{(T - 1)}$  corresponds to the previous-fidelity outputs  $\mathbf{Y}^{(T - 1)})$ ). The joint likelihood can then be decomposed,
$$
\begin{array}{l} \mathcal {L} = \log p (\mathbf {Y} ^ {(0: T)}) = \log p (\mathbf {Y} ^ {(0: T - 1)}, \mathbf {Y} ^ {(T)}) \\ = \log p (\mathbf {Y} ^ {(0: T - 1)}) + \log \int \left[ p (\mathbf {Y} ^ {(T)} | \mathbf {Y} ^ {(T - 1)}, \hat {\mathbf {Y}} ^ {(T - 1)}) p (\hat {\mathbf {Y}} ^ {(T - 1)} | \mathbf {Y} ^ {(T - 1)}) \right] d \hat {\mathbf {Y}} ^ {(T - 1)} \\ = \log p \left(\mathbf {Y} ^ {(0: T - 1)}\right) - \frac {D N ^ {(T)}}{2} \log (2 \pi) - \frac {D}{2} \log \left| \tilde {\mathbf {K}} _ {a} ^ {(T T)} \right| - \frac {1}{2} \left[ \left(\mathbf {Y} _ {a} ^ {(T)}\right) ^ {\top} \left(\tilde {\mathbf {K}} _ {a} ^ {(T T)}\right) ^ {- 1} \mathbf {Y} _ {a} ^ {(T)} \right], \tag {19} \\ \end{array}
$$
$$
\tilde {\mathbf {K}} _ {a} ^ {(T T)} = \mathbf {K} _ {a} ^ {(T T)} + \hat {\mathbf {E}} ^ {(T)} \hat {\mathbf {\Sigma}} ^ {(T)} \left(\hat {\mathbf {E}} ^ {(T)}\right) ^ {\top}, \tag {20}
$$
is the updated additive kernel with the uncertainty of the predictive variance  $\hat{\Sigma}^{(T)}$  of the virtual points, whose computation details are given later in Section 4.1;