SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left[ \mathbf {K} _ {a} ^ {(T T)} \right] _ {i j} = \left[ \mathbf {K} ^ {(T T)} \right] _ {i j} - \left[ \mathbf {K} ^ {(T - 1, T - 1)} \right] _ {i j} \tag {21}
$$
is the additive/residual part of the kernel from  $(T - 1)$  to  $T$ ;
$$
\mathbf {Y} _ {a} ^ {(T)} = \left( \begin{array}{c} \breve {\mathbf {Y}} ^ {(T)} \\ \hat {\mathbf {Y}} ^ {(T)} \end{array} \right) - e ^ {- \beta \Delta_ {T}} \times \left( \begin{array}{c} \breve {\mathbf {Y}} ^ {(T - 1)} \\ \bar {\mathbf {Y}} ^ {(T - 1)} \end{array} \right) \tag {22}
$$
is the additive part for the completed outputs from  $(T - 1)$  to  $T$ ;  $\Delta_T$  is the time interval between  $t_{T-1}$  and  $t_T$ ;  $\log p(\mathbf{Y}^{(0:T-1)})$  is the log likelihood for the previous  $T$  fidelities, which is obtained by calling Eq. (19) recursively. The detailed derivation is preserved in Appendix due to the space limitation. Through the decomposition of Eq. (19), the computation complexity is reduced from  $\mathcal{O}((D\sum_{i=0}^{T}N^{t_i})^3)$  to  $\mathcal{O}(D\sum_{i=1}^{T}(N^{t_i} + N^{t_{i-1}} - |\mathbf{X}^{(t_i)}\cap \mathbf{X}^{(t_{i-1})}|_n)^3)$ . Here,  $|\cdot|_n$  indicates the number of samples. Furthermore, when data at a new fidelity is obtained, we only need to modify the joint likelihood slightly by adding new blocks, which will be handy for active learning or Bayesian optimization. Model training is conducted easily by maximizing the joint likelihood Eq. (19) with respect to the hyperparameters using gradient-based optimizations.
4.1 Predictive Posterior
Since the joint model (1) is a Gaussian, the predictive posterior for the highest fidelity can be derived as in standard GP with a large covariance matrix that requires inversion for once. Similar to the derivation of an efficient joint likelihood in the previous section, we drive an efficient predictive posterior  $\mathbf{y}(\mathbf{x}_*,T)\sim \mathcal{N}(\bar{\mathbf{y}} (\mathbf{x}_*,T),\pmb {\Sigma}(\mathbf{x}_*,T))$  (see Appendix H),
$$
\bar {\mathbf {y}} \left(\mathbf {x} _ {*}, T\right) = e ^ {- \beta \Delta_ {T}} \times \bar {\mathbf {y}} \left(\mathbf {x} _ {*}, T - 1\right) + \left(\mathbf {k} _ {a *} ^ {(T T)}\right) ^ {\top} \left(\mathbf {K} _ {a} ^ {(T T)}\right) ^ {- 1} \mathbf {Y} _ {a} ^ {(T)} \tag {23}
$$
$$
\pmb {\Sigma} (\mathbf {x} _ {*}, T) = e ^ {- 2 \beta \Delta_ {T}} \times \pmb {\Sigma} (\mathbf {x} _ {*}, T - 1) + \tilde {\pmb {\Sigma}} ^ {(T)} + \pmb {\Gamma} ^ {(T)} \hat {\pmb {\Sigma}} ^ {(T)} \bigl (\pmb {\Gamma} ^ {(T)} \bigr) ^ {\top}
$$
$$
\tilde {\boldsymbol {\Sigma}} ^ {(t _ {T})} = \mathbf {I} \left(\tilde {\mathbf {k}} _ {a *} ^ {(T T)}\right) ^ {\top} \left(\tilde {\mathbf {k}} _ {a} ^ {(T T)}\right) ^ {- 1} \tilde {\mathbf {k}} _ {a *} ^ {(T T)}, \quad \hat {\boldsymbol {\Sigma}} ^ {(t _ {T})} = \mathbf {I} \left(\hat {\mathbf {k}} _ {a *} ^ {(T T)}\right) ^ {\top} \left(\hat {\mathbf {K}} _ {a} ^ {(T T)}\right) ^ {- 1} \hat {\mathbf {k}} _ {a *} ^ {(T T)},
$$
$$
\boldsymbol {\Gamma} ^ {(t _ {T})} = \left[ \mathbf {k} _ {a *} ^ {(T, T)} \left(\mathbf {E} _ {n} ^ {(T)}\right) ^ {\top} \tilde {\mathbf {k}} _ {a *} ^ {(T - 1, T - 1)} \left(\tilde {\mathbf {K}} _ {a} ^ {(T - 1, T - 1)}\right) ^ {- 1} \right] \mathbf {E} _ {m} ^ {(T)}, \tag {24}
$$
with two selection matrixes that follow:
$$
\hat {\mathbf {X}} ^ {(t _ {T})} = \left(\mathbf {E} _ {m} ^ {(T)}\right) ^ {\top} [ \mathbf {X} ^ {(T - 1)}, \hat {\mathbf {X}} ^ {(T)} ], \quad \mathbf {X} ^ {(t _ {T})} = \left(\mathbf {E} _ {n} ^ {(T)}\right) ^ {\top} [ \mathbf {X} ^ {(T - 1)}, \hat {\mathbf {X}} ^ {(T)} ]. \tag {25}
$$
In these equations,  $[\tilde{k}_{a*}^{(TT)}]_j = k^y (\mathbf{x}_*,T,\mathbf{x}_j,T) - k^y (\mathbf{x}_*,T - 1,\mathbf{x}_j,T - 1)$  for  $\mathbf{x}_j\in \{\mathbf{X}^{(t_T)}\}$  is the kernel additional part between  $T$  and  $T - 1$  for  $\mathbf{x}_*$  and  $\mathbf{X}^{(t_T)}$ ;  $\hat{\Sigma}^{(t_T)}$  is the predictive variance based on  $\{\mathbf{X}^{(T - 1)},\hat{\mathbf{X}}^{(T)}\}$  of  $T$  fidelity model;  $[\hat{k}_{a*}^{(TT)}]_j = k^y (\mathbf{x}_*,T,\mathbf{x}_j,T) - k^y (\mathbf{x}_*,T - 1,\mathbf{x}_j,T - 1)$  for  $\mathbf{x}_j\in \{\hat{\mathbf{X}}^{(t_T)}\}$  is the kernel additional part between between  $T$  and  $T - 1$  for  $\mathbf{x}_*$  and  $\hat{\mathbf{X}}^{(t_T)}$ ;  $\hat{\Sigma}^{(t_T)}$  is the predictive variance based on  $\{\hat{\mathbf{X}}^{(T)}\}$ .
As stated, FiDEs serve as a unifying framework that unifies many SOTA multi-fidelity and single-fidelity surrogates. We made connections to some popular models in Table 1.
Multi-Variate GPs are commonly used surrogates that can be recovered by setting  $t = \text{Const}$  or  $T = 0$  for the FiDEs, which gives us the fundamental semiparametric latent factor models (SLFM) [20]. SLFM is a simplified Linear model of coregionalization (LMC) [23, 29]. Based on SLFM, intrinsic model of coregionalization (IMC) reduces the computational complexity by setting a rank-1 approxi
mation  $\mathbf{u}(\mathbf{x})\sim \mathcal{N}(\mathbf{0},\mathbf{k}^u (\mathbf{x},\mathbf{x}')\otimes \mathbf{I})$  ; HOGP [24] improves IMC by letting  $[\mathbf{S}\mathbf{S}^{\top}]_{ij} = k^{d}(\mathbf{z}_{i},\mathbf{z}_{j})$  with tensor decomposition, where  $\mathbf{z}$  are latent vectors. Higdon et al. [19] further simplify SLFM using singular value decomposition (SVD) to obtain S. For a full review of the GP-based multi-variate surrogates, the readers are referred to [21] for an excellent review. To overcome the fixed bases of SLFM, GP regression network (GPRN [30, 25, 31]) introduces another GP to model a flexible S.
Model	Assumptions under FiDEs
SLFM [20]	T=0;
IMC [23]	T=0; u(x) ~ N(0, k^u(x, x') ⊗ I)
HOGP [24]	T=0; R=1; [SS^T]_{ij} = k^d(z_i, z_j)
GPRN [25]	T=0, S ← GP(x)
AR [8]	D=1; B(t) = β(t_i)
ResGP [12]	R=1, B(t) = 0, S = I,
NAR [10]	B(t) = 0, G(u(x, t)) ← GP(x, y, t_i), S = I
MF-BNN [26]	B(t) = 0, G(u(x, t)) ← BNN(x, y, t_i), S = I
SC [27]	B(t) = 0, G(u(x, t)) ← PCE(x, y, t_i), S = I
DC [28]	B(t) = 0, G(u(x, t)) ← GP(x, y, t_i)
S = ResPCA(Y) or S_i = PCA(Y(i))
IFC [8]	B(t) = 0, u(x, t), ← NN(x, y, t), S = I
ContinuAR	B(t) = βI, S = I
Table 1: FiDEs unifying GP surrogates
Tractable Fusion is the name we give to multi-fidelity models where the joint outputs form a GP. AR [B] is the most fundamental and tractable solution to FiDEs for a univariate problem, i.e.,  $D = 1$ , which allows it to take a (discrete) t-dependent  $\mathbf{B}(t) = \beta(t_i)$  without leading to an intractable solution. To deal with high-dimensional problems, ResGP [D] avoids the difficulty involved with  $\mathbf{B}(t)$  by setting it to zero and uses a conditional independent GP for  $u(\mathbf{x}, t_i)$  (equivalent to  $R = 1$ ,  $S = I$ ).
Intractable Fusion refers to methods where the joint output is no longer a GP. Normally, they assume  $\mathbf{B}(t) = 0$  to avoid the integral and replace the antiderivative  $\mathcal{G}(\mathbf{u}(\mathbf{x}, t))$  in Eq. (13) with a regression model that takes  $\mathbf{y}$  (at the previous fidelity) as model input. For instance, the popular nonlinear AR (NAR) [10] uses  $GP(\mathbf{x}, \mathbf{y}, t_i)$  to replace  $\mathcal{G}(\mathbf{u}(\mathbf{x}, t))$ ; Deep coregionalization (DC [28]) extends NAR with a residual PCA to capture S; Wang et al. [24] further introduce a fidelity vari-. ating  $\mathbf{S}(t_i)$  to increase model flexibility at the cost of significant growth in the number of model parameters and a few simplifications in the approximated inference; Li et al. [26] take the advances of recent advancement of deep learning neural network (NN) and place a Bayesian neural network (BNN) to replace  $\mathcal{G}(\mathbf{u}(\mathbf{x}, t))$ ; A similar idea is proposed by Meng and Karniadakis [22] who add a physics regularization layer to a deep NN; Li et al. [23] propose a Bayesian network approach to multi-fidelity fusion with active learning techniques for efficiency improvement. To account for the missing uncertainty propagation in intractable fusion methods, Cutajar et al. [24] use approximation inference at the price of overfitting and scalability to high-dimensional problems; In the UQ community, multi-fidelity fusion has been implemented using stochastic collocation (SC) [27], which essentially uses a polynomial chaos expansion (PCE) to approximate  $\mathcal{G}(\mathbf{u}(\mathbf{x}, t))$  under FiDEs. All the above methods do not generalize to infinite-fidelity problems. The seminal work IFC [8] uses a NeuralODE to solve general FiDEs, leading to a great challenge in model training.
Algorithm. The solutions of FiDEs are closely related to the latent force models (LFMs) [16, 35], where they focus on dynamic processes without  $\mathbf{x}$ . For tractable fusion, Le Gratiet [10] improves the efficiency of AR by decomposing the likelihood functions based on a subset multi-fidelity data structure, which is relaxed through our non-subset decomposition in Section 4.
To assess ContinuAR, we compare it with (1) AR [3], (2) ResGP [2], (3) MF-BNN [33], and (4) IFC [8], which are the most closely related SOTA methods for high-dimensional multi-fidelity fusion, particularly with infinite fidelities. Note that AR and ResGP are modification versions according to [36] instead of their original versions that cannot deal with non-subset or high-dimensional problems. ContinuAR, AR, and ResGP are implemented using Pytorch. All GP-based methods use the RBF kernel for a fair comparison. MF-BNN and IFC are conducted using default settings from their source codes. As presented in IFC's original paper, there are two variations of IFC, one with deep learning (IFC-ODE) and the other one with Gaussian process ODE (IFC-GPODE), which shows better results. Thus, we show the results of IFC-GPODE in our experiments. All GP-based models are trained with 200 iterations whereas MF-BNN and IFC with 1000 iterations to reach convergence. All experiments are run on a workstation with an AMD 5950x CPU, Nvidia RTX3090 GPU, and 32 GB RAM. All experiments are repeated five times with different random seeds, and the mean performance and its standard deviation are reported.
6.1 Multi-Fidelity Fusion for Benchmark PDEs
We first assess ContinuAR in three canonical PDE simulation benchmark problems, namely, Heat, Burgers', and Poisson's equations as in [28, 37-39], which produces high-dimensional spatial/spatial-temporal fields as model outputs. The multi-fidelity results are generated by solving the corresponding PDEs using finite difference with mesh nodes of  $\{4^2, 8^2, 16^2, 32^2, 64^2\}$ , which is also used as the fidelity index  $t$  for ContinuAR and IFC. We use interpolation to upscale the lower fidelity fields and record them at the high-fidelity grid nodes to provide uniform output across different fidelity. Equation parameters in the PDEs and parameterized initial or boundary conditions represent the corresponding inputs. Please refer to Appendix J for comprehensive details of simulation setups.
IFC has an extremely high computational cost which scales poorly to output dimension (see Table 1 for experiments with  $N^0 = 32$  and  $\eta = 0.5$ ). For Poisson's equation, considering the 5x more iterations required, IFC requires about 62,500x more training time than ContinuAR on a CPU. It
$^{2}$  https://github.com/shib0li/DNN-MFBO  ${ }^{3}$  https://github.com/shib0li/Infinite-Fidelity-Coregionalization
Figure 2: Subset Evaluation with  $\eta = 0.5$  (top row) and  $\eta = 0.75$  (bottom row): RMSE against number of training samples  $N^0$  for Heat (left), Burger's (middle), and Poisson's (right) equation.
will take 32 days just to finish the Poisson experiment in our experimental setup. Thus, we do not consider IFC a practical method for high-dimensional problems because its training cost is almost as expensive as running a high-fidelity simulation; we only apply it to relatively low-dimensional problems, namely, Heat equation and the later real-world application experiments.
IFC
(CPU/GPU)	Ours	AR	ResGP	MF-BNN
Heat	3.0/3.6	0.016	0.012	0.010	0.014
Burgers	66.9/63.9	0.010	0.009	0.024	0.026
Poisson	137/67.1	0.011	0.001	0.001	0.025
Table 2: Training time (seconds) per iteration
Figure 1: Testing RMSE against training time for Heat equation.
Classic Subset Assessment. We follow the classic experiment setup where the training samples are consistently increased; the lower-fidelity data forms a superset of the higher-fidelity data for this experiment. To deliver concrete results, the high-fidelity training samples are reduced at rate  $\eta$ , i.e.,  $\mathrm{round}(\eta |\mathbf{X}^{(t_{i + 1})}|_n) = |\mathbf{X}^{(t_i)}|_n$ , while the removed samples are randomly selected. For each experiment, we gradually increase the number of lowest-fidelity training data  $N^0$  from 32 to 128 and calculate predictive accuracy (RMSE). The statistical result for five repeated experiments (with different random seeds) is demonstrated in Fig. ②. The superiority of IFC and ContinuAR indeed highlights the benefits of harnessing the useful information hidden in the fidelity indicators  $t_i$ . All method benefits from a larger  $\eta$ ; MF-BNN is unstable due to its model complexity. ContinuAR outperforms the competitors with a large margin consistently in call cases (with up to 6.8x improvement). Averaging over all experiments, ContinuAR achieves 3.5x, 2.5x, and 8.7x accuracy improvements over the best competitor for Heat, Burger's, and Poisson's equation, respectively. Testing error against training time for Heat  $N^0 = 32$  and  $\eta = 0.5$  is shown in Fig. ③, where ContinuAR achieves 2.8x and 603x improvement in RMSE and training time. Detailed mean error fields (see Appendix for the computational details) are also provided in Fig. ④, which clearly reveals the superiority of ContinuAR by producing minimal red regions (high error) and maximal blue regions (low error).
Non-subset Assessment. We then assess the compatibility of ContinuAR for non-subset training data, which is inevitable in many-fidelity problems. The setup is similar to subset assessment with the same decreasing rate  $\eta$ , except that the training data for each fidelity is randomly selected without forcing a subset structure. The results are reported in Fig. B. The results are consistent with the subset assessment, and ContinuAR outperforms the competitors with a large margin (with up to 3x improvements), whereas MF-BNN performs poorly as usual. Averaging over all experiments, ContinuAR achieves 1.9x, 1.6x, and 1.5x accuracy improvements over the best competitor for Heat, Burger's, and Poisson's equation, respectively. The mean error fields are demonstrated in Fig. B, which draws the same conclusion as the subset assessment.
Figure 3: Non-Subset Evaluation with  $\eta = 0.5$  (top row) and  $\eta = 0.75$  (bottom row): RMSE against number of training samples  $N^0$  for Heat (left), Burger's (middle), and Poisson's (right) equation.
Figure 4: Average RMSE fields of the subset (left four columns) and non-subset (right four columns) evaluation with  $\eta = 0.5$  and  $128N^0$  training samples for Heat equation (1st row), Poisson's equation (2nd row), Burger's equation (3rd row), and TopOP (4th row).
6.2 Multi-Fidelity Fusion for Real-World Applications
Next, we assess ContinuAR with real-world applications. Particularly, we look at two practical simulation applications: topology optimization (TopOP) and Plasmonic nanoparticle arrays (PNA) simulation, both of which are known for their high computational cost and render the need for multifidelity fusion. The detailed problem descriptions and simulation setups are given in Appendix J. The TopOP data contains five fidelity solutions of dimensionality of 1600 based on simulations on a mesh concatenating \{64, 256, 576, 1024, 1600\} nodes, whereas the PNA data has five-fidelity outputs of two dimensions.
The same subset and non-subset assessments are conducted, and the results are reported in Fig. 6. We can see that the advantage of ContinuAR is clear but less pronounced due to the complexity of the data. Averaging over all experiments, ContinuAR achieves 1.1x and 1.4x improvements over the best competitor in TopOP and PNA, respectively. Although taking averagely 192s for training (140x more than ResGP and AR) for PNA, IFC only shows small improvements over ResGP and AR in non-subset settings, but not in the subset settings. In contrast, the training time of ours is only 5.7s (4.4x more than ResGP and AR) and the improvements are significant in all cases. Detailed mean error field in Fig. 4 shows a significant reduction in red region volume for ContinuAR.
(a) TopOP with  $\eta = 0.5$  (left) and  $\eta = 0.75$  (right)
(b) PNA with  $\eta = 0.5$  (left) and  $\eta = 0.75$  (right)
Figure 5: Subset (first row) and Non-Subset (second row) Evaluation.
Figure 6: Cost Evaluation for Heat equation, Burger's equation, Poisson's equation, TopOP, and PNA.
Cost Evaluation. Finally, to assess ContinuAR in a more realistic setting, we relax the requirement that low-fidelity samples are more than high-fidelity samples and randomly pick samples for each fidelity (ensuring that each fidelity has a minimal of four samples) for each models. We do not test under different seeds because the randomness of setting is already sufficient to assess the robustness of the models. The computational cost for generating the training data against accuracy results for all assessment datasets are reported in Fig. 6. The superiority of ContinuAR over the competitors is consistent with previous experiments. The average accuracy improvements over all settings of ContinuAR over the best competitors is  $4.3\mathrm{x}$ ,  $9.3\mathrm{x}$ ,  $2.7\mathrm{x}$ ,  $1.2\mathrm{x}$ , and  $1.3\mathrm{x}$  for Heat equation, Burger's equation, Poisson's equation, TopOP, and PNA, respectively.
In this work, we propose a unifying framework FiDEs to pave the way for tractable infinite-fidelity fusion and a novel solution ContinuAR, which shows a significant improvement over the SOTA competitors. We expect FiDEs and ContinuAR can lead to the development of surrogate-assisted systems to deliver cheaper, more efficient, and eco-friendly scientific computation. The limitation of this work includes the rank-1 assumptions and further examination in highly-complex real systems.
[38] Mehmet Onder Efe and Hitay Ozbay. Proper orthogonal decomposition for reduced order modeling: 2d heat flow. In Proceedings of 2003 IEEE Conference on Control Applications, 2003. CCA 2003., volume 2, pages 1273-1277. IEEE.
[39] Maziar Raissi and George Em Karniadakis. Machine Learning of Linear Differential Equations using Gaussian Processes. Journal of Computational Physics, 348:683-693, November 2017. ISSN 00219991. doi: 10/gbzfrgr.
A Gaussian process
Due to its capacity to handle complex black-box functions and quantify uncertainty, the Gaussian process (GP) is often a popular selection for the surrogate model. For now, let's contemplate a simplified scenario where we possess noise-influenced observations, denoted as  $y_{i} = f(\mathbf{x}_{i}) + \varepsilon$ ,  $i = 1,\dots ,N$ . In a GP model, a prior distribution is placed over the true underlying function of  $f(\mathbf{x})$ , whose input is  $\mathbf{x}$ :
$$
\eta (\mathbf {x}) \mid \boldsymbol {\theta} \sim \mathcal {G P} \left(m _ {0} (\mathbf {x}), k \left(\mathbf {x}, \mathbf {x} ^ {\prime} \mid \boldsymbol {\theta}\right)\right), \tag {A.1}
$$
employing mean and covariance functions,
$$
\begin{array}{l l} m _ {0} (\mathbf {x}) & = \mathbb {E} [ f (\mathbf {x}) ], \\ k (\mathbf {x}, \mathbf {x} ^ {\prime} | \boldsymbol {\theta}) & = \mathbb {E} [ (f (\mathbf {x}) - m _ {0} (\mathbf {x})) (f (\mathbf {x} ^ {\prime}) - m _ {0} (\mathbf {x} ^ {\prime})) ], \end{array} \tag {A.2}
$$
where the hyperparameters  $\theta$  control the kernel function and  $\mathbb{E}[\cdot]$  is the expectation. Centering the data allows us to assume the mean function as a constant,  $m_0(\mathbf{x}) \equiv m_0$ . Other options, like a linear function of  $\mathbf{x}$ , are possible but are seldom employed unless there is prior knowledge of the function's shape. Various forms can be adopted for the covariance function, with the ARD kernel being the most widely favored.
$$
k \left(\mathbf {x}, \mathbf {x} ^ {\prime} \mid \boldsymbol {\theta}\right) = \theta_ {0} \exp \left(- \left(\mathbf {x} - \mathbf {x} ^ {\prime}\right) ^ {T} \operatorname {d i a g} \left(\theta_ {1} ^ {- 2}, \dots , \theta_ {l} ^ {- 2}\right) \left(\mathbf {x} - \mathbf {x} ^ {\prime}\right)\right). \tag {A.3}
$$
Starting here, we remove the specific mention of  $\pmb{\theta}$  's dependence on  $k(x,x^{\prime})$  . The hyperparameters  $\theta_{1},\dots ,\theta_{l}$  are termed as length-scales in this case. When  $\mathbf{x}$  is a constant parameter,  $f(\mathbf{x})$  represents its random variable. In contrast, a set of values,  $f(\mathbf{x}_i)$  , where  $i = 1,\ldots ,N$  , constitutes a partial realization of the GP. Realizations of GPs are deterministic functions of  $x$  . The key characteristic of GPs is that the joint distribution of  $\eta (x_i)$  , for  $i = 1,\dots ,N$  , is a multivariate Gaussian.
We can derive the model likelihood by assuming the Gaussian distribution of the model deficiency  $\varepsilon \sim \mathcal{N}(0, \sigma^2)$ , along with the utilization of the prior (A1) and the existing data.
$$
\begin{array}{l} \mathcal {L} \triangleq p (\mathbf {y} | \mathbf {x}, \boldsymbol {\theta}) = \int (f (\mathbf {x}) + \varepsilon) d f = \mathcal {N} (\mathbf {y} | m _ {0} \mathbf {1}, \mathbf {K} + \sigma^ {2} \mathbf {I}) \\ = - \frac {1}{2} \left(\mathbf {y} - m _ {0} \mathbf {1}\right) ^ {T} \left(\mathbf {K} + \sigma^ {2} \mathbf {I}\right) ^ {- 1} \left(\mathbf {y} - m _ {0} \mathbf {1}\right) \tag {A.4} \\ - \frac {1}{2} \ln | \mathbf {K} + \sigma^ {2} \mathbf {I} | - \frac {N}{2} \log (2 \pi), \\ \end{array}
$$
where  $\mathbf{K} = [K_{ij}]$  is the covariance matrix, in which  $K_{ij} = k(\mathbf{x}_i,\mathbf{x}_j)$ ,  $i,j = 1,\dots ,N$ . The hyperparameters  $\pmb{\theta}$  are frequently obtained through point estimations, utilizing the maximum likelihood (MLE) of Eq. (A4) with respect to  $\pmb{\theta}$ . The joint Gaussian distribution of  $\mathbf{y}$  and  $f(\mathbf{x})$  also possesses a mean value of  $m_0\mathbf{1}$  and a covariance matrix.
$$
\mathbf {K} ^ {\prime} = \left[ \begin{array}{c c} \mathbf {K} + \sigma^ {2} \mathbf {I} & \mathbf {k} (\mathbf {x}) \\ \hline \mathbf {k} ^ {T} (\mathbf {x}) & k (\mathbf {x}, \mathbf {x}) + \sigma^ {2} \end{array} \right], \tag {A.5}
$$
where  $\mathbf{k}(\mathbf{x}) = (k(\mathbf{x}_1,\mathbf{x}),\dots ,k(\mathbf{x}_N,\mathbf{x}))^T$ . The conditional predictive distribution at  $\mathbf{x}$  is obtained by conditioning on  $\mathbf{y}$ .
$$
\hat {f} (\mathbf {x}) | \mathbf {y} \sim \mathcal {N} (\mu (\mathbf {x}), v (\mathbf {x}, \mathbf {x} ^ {\prime})),
$$
$$
\mu (\mathbf {x}) = m _ {0} \mathbf {1} + \mathbf {k} (\mathbf {x}) ^ {T} \left(\mathbf {K} + \sigma^ {2} \mathbf {I}\right) ^ {- 1} (\mathbf {y} - m _ {0} \mathbf {1}), \tag {A.6}
$$
$$
v (\mathbf {x}) = \sigma^ {2} + k (\mathbf {x}, \mathbf {x}) - \mathbf {k} ^ {T} (\mathbf {x}) \left(\mathbf {K} + \sigma^ {2} \mathbf {I}\right) ^ {- 1} \mathbf {k} (\mathbf {x}).
$$
Given by  $\mu (\mathbf{x})$ , the expected value is  $\mathbb{E}[f(\mathbf{x})]$ , and  $v(\mathbf{x})$  represents the predictive variance. The transition from Eq. (A5) to Eq. (A6) is critical, as the prediction posterior of this wake relies on a comparable block covariance matrix.
B From AR to LiFiDE
We first revisit the classic AR and reveal its connection to an ODE's forward Euler solution. Rewrite Eq. (II) as follows,
$$
y (\mathbf {x}, t _ {T}) = \gamma y (\mathbf {x}, t _ {0}) + v (\mathbf {x}, t _ {0}),
$$
$$
y (\mathbf {x}, t _ {T}) - y (\mathbf {x}, t _ {0}) = (\gamma - 1) y (\mathbf {x}, t _ {0}) + v (\mathbf {x}, t _ {0}), \tag {A.7}
$$
$$
y (\mathbf {x}, t _ {T}) - y (\mathbf {x}, t _ {0}) = \alpha y (\mathbf {x}, t _ {0}) + v (\mathbf {x}, t _ {0}),
$$
where  $\alpha \equiv \gamma - 1$ . We then divide both sides by the constant  $(t_{T} - t_{0})$  indicating the fidelity difference and absorb the constant into  $(\rho - 1)$  and  $u(\mathbf{x}, t_0)$  and write