GAR: Generalized Autoregression for Multi-Fidelity Fusion
Authors: Yuxin Wang
Zheng Xing
WEI W. XING
Venue: NeurIPS 2022 Accept (2022)
Keywords: Gaussian process
autoregression
multi fidelity
nonparametric Bayesian

Abstract: In many scientiﬁc research and engineering applications, where repeated simulations of complex systems are conducted, a surrogate is commonly adopted to quickly estimate the whole system. To reduce the expensive cost of generating training examples, it has become a promising approach to combine the results of low-ﬁdelity (fast but inaccurate) and high-ﬁdelity (slow but accurate) simulations. Despite the fast developments of multi-ﬁdelity fusion techniques, most existing methods require particular data structures and do not scale well to high-dimensional output. To resolve these issues, we generalize the classic autoregression (AR), which is wildly used due to its simplicity, robustness, accuracy, and tractability, and propose generalized autoregression (GAR) using tensor formulation and latent features. GAR can deal with arbitrary dimensional outputs and arbitrary multiﬁdelity data structure to satisfy the demand of multi-ﬁdelity fusion for complex problems; it admits a fully tractable likelihood and posterior requiring no approximate inference and scales well to high-dimensional problems. Furthermore, we prove the autokrigeability theorem based on GAR in the multi-ﬁdelity case and develop CIGAR, a simpliﬁed GAR with the same predictive mean accuracy but requires signiﬁcantly less computation. In experiments of canonical PDEs and scientiﬁc computational examples, the proposed method consistently outperforms the SOTA methods with a large margin (up to 6x improvement in RMSE) with only a few high-ﬁdelity training samples.

GAR: Generalized Autoregression for Multi-Fidelity Fusion
Yuxin Wang School of Mathematical Science Beihang University Beijing, China, 100191. WYXtt_2011@16: com
Zheng Xing Graphics&Compunee Department Rockchip Electronics Co., Ltd Fuzhou, China, 350003 zheng wwwggoiikoookkcssscooooo
Wei W. Xing
School of Mathematics and Statistics, University Sheffield, Sheffield S10 2TN, UK School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China, 100191. wayne xingle@gmail. com
Abstract
In many scientific research and engineering applications where repeated simu- lations of complex systems are conducted, a surrogate is commonly adopted to quickly estimate the whole system. To reduce the expensive cost of generating training examples, it has become a promising approach to combine the results of low-fidelity (fast but inaccurate) and high-fidelity (slow but accurate) simulations. Despite the fast developments of multi-fidelity fusion techniques, most existing methods require particular data structures and do not scale well high -imeensioii output. To resolve these issues, we generalize the classic autoregression (AR), which is wildly used due to its simplicity, robustness, accuracy, and tractability, and propose generalized autoregression (GAR) using tensor formulation and latent features. GAR can deal with arbitrary dimensional outputs and arbitrary multi- fidelity data structure to satisfy the demand of multi-fidelity fusion for complex problems; admits fully tractable likelihood and posterior requiring no approxi- mate inference and scales well to high-dimensional problems. Furthermore, we prove the autokrigeability theorem based on GAR in the multi-fidelity case and develop CIGAR, simplified GAR with the exact predictive mean accuracy with computation reduction by a factor of d', where d is the dimensionality of the output. The empirical assessment includes many canonical PDEs and real scientific examples and demonstrates that the proposed method consistently outperforms the SOTA methods with large margin (up 6x improvement in RMSE) with only couple high-fidelity training samples.
1 Introduction
The design, optimization, and control of many systems in science and engineering can rely heavily on the numerical analysis of differential equations, which is generally computationally intense. In this case, data-driven surrogate model is used to approximate the system based on the input-output data of the numerical solver and to help improve convergence efficiency where repeated simulations are required, e.g., in Bayesian optimization (BO) I and uncertainty analysis (UQ) (2).
With the surrogate model in place, the remaining challenge is that executing high-fidelity numerical simulations to generate training data can still be very expensive. To further reduce the computational
TThe authors contribute equally to this paper.
Corresponding author.
Primary
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

burden, it is possible to combine low-fidelity results to make high-fideling predictions B. More specifically, low-fidelity solvers are normally based on simplified PDEs (e.g. reducing the levels of physical detail) or simple solver setup (e-g., using coarse mesh, large time step, a lower order of an approximating basis, and a higher error tolerance). They provide fast but inaccurate solutions to the original problems whereas the high-fidelity solvers are accurate yet expensive. The multi-fidelity fusion technique works similar to transfer learning to utilize many low-fidelity observations to improve the accuracy when using only a few high-fidelity samples. In general, it involves constructions of surrogates for different fidelities and a cross-fidelity transition process. Due to its efficiency, multi-fidelityy method has attracted increasing attention in BO UQ 6 N
and
surrogate modeling 9. We refer to I0 111 for great review.
Despite the success of many state-of ofate-of-the-arr art (SOTA) approaches, they normally assume that (1) the output dimension the same and aligned across all fidelities, which generally does not hold for multi-fidellit simulation where the output are quantities at nodes that are naturally not aligned; (2) the high-fidelity samples' corresponding inputs form a subset of the low-fidelity inputs; and (3) the output dimension is small, which is not practical for scientific computing where dimension can be 1-million (for a 100 x 100 x 100 spatial-temporal field). These assumptions seriously hinder their applications for practical problems, e.g., MRI imaging and solving PDEs in scientific computation.
To resolve these challenges, previous work either uses interpolation align the dimension 1
9 or
relies on approximate inference with brutal simplification R 13 leading to inferior performance. We notice that the classic autoregression (AR), which is widely used due to its simplicity, robustness, accuracy, and tractability, consistently shows robust and top-tier performance for different datasets in the literature, despite its incapability for high-dimensional problems. Thus, instead of proposing another ad-hoe model with pre- pre-processing and simplification (leading to models that are difficult to tune and generalize poorly), we generalize AR with tensor algebra and latent features and propose generalized autoregression (GAR), which can deal with arbitrary high-dimensional problems without the subset multi-fidelity data structure. GAR is a fully tractable Bayesian model with scalability to extremely high-dimensional outputs, without requiring any approximate inference. The novelty of this work is as follows,
1. Generalization of AR for arbitrary non-structured and high-dimensional outputs. With tensor algebra and latent features, GAR allows effective knowledge transfer in closed-form and is scalable to extreme high- nigh-dimensional problems.
2. Generalization to non-subset multi-fidelity data for AR. To the best of our knowledge, are the first to generalize the closed form solution of subset data to non-subset cases for AR and the proposed GAR.
3. For the first time, we reveal the autokrigcability 14! for the multi-fidelity fusion within an AR structure, based on which we derive conditional independent GAR (CIGAR), an efficient implementation of GAR with the exact accuracy in posterior mean predictions.
2 Backgronud
2.1 Statement of the problem
Given multi-fidelity data D' = {xis y J for i = I,...,T, where x E R' denotes the system inputs vector of parameters that appear the system of equations and/or in the initial-boundary conditions for a simulation); y" E TRd denotes the corresponding outputs, where d' the dimension for i fidelity; 7 is the total number of fidelities. Generally speaking, higher-fidelity data are closer to the ground truth and are more expensive to obtain. Thus, we have fewer samples for the high fidelity, i.e., N > 2 > ... N The dimensionality is not necessary the same or aligned across different fidelities. In most work e.g., 05 16 II. the system inputs of higher-fidelty are chosen to be the subset of the lower fide-fiielity, i.e., X C... X C X!. We call this the subset structure for multi-fidelity dataset, as opposed to arbitrary data structures, which we will resolve Section B.3 with a closed-form solution and extend it to the classic AR. Our goal is to estimate the function f" R + Rd given the observation data across different fidelities { J1=1

---

2.2 Autoregression
For the sake of clarity, we consider -voffidelity case with superscript h indicating high-fidelity and I low-fidelity. Nevertheless, the formulation can be generalized to problems with multiple fidelities recursively. Considering simple scalar output for all fidelities, AR [3] assumes
f*(x) (x)
where p is factor transferring knowledge from the low fidelity in a linear fashion, whereas f (") tries to capture the residual information. we assume zero mean Gaussian process (GP) prior [7 (see Appendix A for a brief description) for fx) and f"(x), i..., f'() NN (x,xx'''x and ("(x) ~ N (0, A (x, (x,x')), x')), the high-fidelity function also follows a GP. This gives an elegant joint GP for the joint observations y y''
( yd (
N
K X PK x') p'K'(X", (") + +K x²)) )
( x²)
(2)
where y' E RR-ix1 is the low-fidelity observations corresponding to input X E RNXLL and yd E R x1 is the high-fidelity observations; [K X'), k (x,, x,) is the covariance matrix of W (x,,,,) is for the high-fidelity inputs X1, X2 E X"; [K' (X',xx)) ) (x,, x,) the cross-fidelity covariance matrix of the low-fidelity inputs x, X and high-fidelity inputs x, E X* and (' x (K' x²))². One immediate advantage of AR is that the joint Gaussian form allows not only joint training for all low- and high-fidelity data but also predictions for any given new inputs by a conditional Gaussian (as the posterior is derived in a standard GP 17). Furthermore, Le Gratiet [5 derive Lemma D to reduce the complexity from O( + +*) to O((N)) + (N*))) with a subset data structure.
the low-fidelity inputs x, X': [" (XXX))))
Lemma 1. A X C X', the joint likelihood and predictive posterior of AR can be decomposed into two independent parts corresponding the low- and high-fidelity data.
3 Generalized Autoregression
Let's now consider the more general high- dish-imnnniinal case. naive approach is to simply convert the multi-dimensional output into a scalar value by attaching a dimension index to the input. However, AR will end up with a joint GP with a covariance matrix of the size of (N'd + Nh, making infeasible for modestly high-dimensional problems.
3.1 Tensor Factorized Generalization with Latent Features
To resolve the scalability issue, we rearrange all the output into a multidimensional space (i.e., tensor space) and introduce latent coordinate features to index the outputs to capture their correlations as in HOGP [А] More specifically, we organize the low-fidelity output as a M-mode tensor, z' E T x.. x dis, where the output dimension d' = -I d',, The element Z is indexed based on (cI,..., CM C1 & C& dg for l = 1,... 1,..., M). If the original data indeed admits
its coordinates a multi-array structure, we can use their original index with actual meaning to index the coordinates. For instance, a 2D spatial dataset can use its original spatial coordinate index single location (pixel). To improve our model flexibility, we do not have to limit ourselves from using the original index, particularly for the cases where the original data does not admit a multi-array structure or the multi-array structure is of too large size. In such case, we can use arbitrary tensorization and a latent feature vector v'... (whose values are inferred in model training) for each coordinate cm in mode m. This way, the element Zt is indexed by the vector (vl (V....... vi Cer ). Following the linear transformation of Eq. ① we first introduce the tensor-matrix product 19]
Fx())
F' F(() X1 WI...., M W + (x),,
where F* (x) denotes target function f (x() with its output organized into multi-array Zh and the same concept applies to F'(x) and F" (x); Xm denotes the tensor matrix product at mode m. To give concrete example, considering an arbitrary tensor Z' E Rd x xx'x and a matrix Wm. E = the k= Wikk Zi Zi,....im -1 fmm4..... which becomes an df ... x d X X d 1 X ... d's tensor. We can further denote the group
X product is calculated as Iz' Xm W.,. Wiki..... mm......mm-- -1 J.6m4 = Dim
3

---

of M linear transformation matrixes as Tucker tensor W = [WI.... W and represent Eq.
compactly using Tucker operator
19
F (x) x W, which has an important property:
vec (xx) F" (x))
W ... W vec
(F (F
(4)
Inspired by AR of Eq. I we place a tensor-variate GP (TGP) prior 20 for the low-fidelity tensor function F' (x) and the residual tensor function Fxx):
Z' (x, x²) TGP (0, k2 (x.x') x'), s ................................ S'x) Z' (x,x') TGP (0, k (x, ) S SMM), (5)
where Sim E Ri xdm. are the output correlation matrix with [S |S = kin (y va and kú (.. be- ing the kernel function (with unknown hyperparameters). A TGP is generalization of a multivariate GP that essentially represents a joint GP prior vec(Y') ~ W (0. (X.
m- S Similar to
M
the joint probability of 2 we can derive the joint probability for for y = [vec{ Y') /eeleeeeeV vec(Y")777 based on Tucker transformation of B); we preserve the proof in the Appendix for elarity.
Lemma 2. Given the tensor GP priors for Y' (xxx') and Y" (xxxxx') and the Tucker transformation of
the joint probability for y - ( ( is y (0, E), where Σ
('X'', ) 80
Sm
K'XX' X @
Sm WT
K (' (0) @
W Sm.
K' X²) @ (@th m- W S.. W + (X"X", X")
m1 Sin
Lemma 2 admits any arbitrary outputs (living in different spaces, having different dimension and/or mode, and being unaligned) at different fidelity. Also, it does not require subset dataset to hold. Corollary 3.0.1. Lemma E can be applied to data with a different number of mode at each fidelity, i.e., M * Mh, if we add redundancy index such that all outputs have the same M number of modes.
Lemma 2 defines our GAR model, generalized AR with special tensor structures. The co- ( (x), (x))) xxxxx') IIm T (ven v²n)) cross-fidelity variance for low-fidelity is cov(
cov( cov(Z: (x), Z (x') (xxxx') x') 11 I F (v!
(emm )u (where w is the c'th element of W and high-fidelity cov (Z) (x), Z (x))
x'xxxx)) II T (v! e.. v2 (w (x( x') II kr (u uz (win Gfet )² The complex between-fidelity output correlations cap- tured using latent features {Vm UT} M
/m=1 with arbitrary kernel function 7 whereas the cross- composite manner. This combination overcomes the simple linear correlations assumed in previous work that simply decomposes the output as a dimension reduction preprocess 12 When the dimensionality aligns for Z' and Zd and thus d' d we can share the same latent features across the two fidelities by letting v u, while keeping the kernel functions diffferent This way, the latent features more resistant overfitting. For non-aligned data with explicit indexing, we can use kernel interpolation 21 for the same purpose. To further encourage sparsity in the latent feature, we impose a Laplace prior, i.e., vy" Laplace(A) xx exp(-Allve Illv)"
fidelity output correlations are captured
3.2 Efficient Model Inference for Subset Data Structure
With the model fully defined, we can now train the model to obtain all unknown model parameters.
For compactness, we use the following compact notation S' = 8 M
m=1 S!
Sh
2 Mf Sh W
mm1
W m. K' = ('XX',X)) x'), k = x'(xxx")) kh = K xx(x'x') K K x²)..
and, K' = K" (X', x²) (with slight abuse of notation).
Lemma 3. Tensor generalization of Lemma L If X C XX,, the joint likelihood L for y = [vec(Y") Veccl ("")("" admits two independent separable likelihoods L = L' + L" where
L vec (r') (K' s0 (a) s''=~vvee
(v')
1
log K' 66 S
N' D2
log(2*);
2
1 2 vec hh r' x W) T
N Dr
L =
(KTSST)- (Y* r W)
log
K ST
2 log(2.),
2
where W= = E, W Wi...., Warl is Tucker tensor with selection matrix E x = x

---

We preserve the proof in Appendix for clarity. Note that L' and L" are HOGP likelihoods for Y' and the residual Y YY' x W, respectively. Since the computational of L' and C are independent, the model training can be conducted efficiently in parallel.
Predictive posterior. Similarly, we can derive the concrete predictive posterior for the high-fideling outputs by integrating out the latent functions after some tedious linear algebra (see Appendix), which is also Gaussian, vee ( (V(cc(Z, ), ),), where
vec(Z,) (22) kg (K') 2 W veal Y') + (K') @ vec(Y"),
(6)
; Kk'. (k.) (K') k.) ws'w" (A.. (k.)' (K")-Ik') @s.
kg = k'(x,, x²) is the vector of covariance between the give input and low-fidelity observation inputs X similarly, we have k!. = k' (x....x,, x.), K. = k" (xx,,, x²), k,. = k" (x.,
3.3 Generalization for Non-subset Data: Efficient Model Inference and Prediction
In practice, it is sometimes difficult to ask the multi filei-fi data to preserve a subset structure,
particularly in the case of muult--fidelin Bayesian optimization 22 2 This presents the challenge for most SOTA multi-fidelity models e.g., NAR [6). ResGP 9 stochastic collocation [24]. In
contrast, the advantage of AR is that even if the multi-fidelity data does not admit a subset data structure, the model can still be trained using all available data based on the joint likelihood of (2) However, this method lacks scalability due to the inversion of the large joint covariance matrix E. The situation gets worse if we are dealing with multi-fidelity data with more than two fidelities. To resolve this issue, we propose a fast inference method based on imaginary subset. More specifically,
considering the missing low-fidelity data as latent variables Y the joint likelihood function is
log Doy, yh) log / p(Y', yh v'av'' = log / (P(V") (') Y p(Y') av
= log [ I Y') y' + +logp(Y'),
where p(Y Y¹) is the likelihood in Lemma 3 given the complementary imaginary subset, and PY YY) NYYY S' x S') is the imaginary posterior with the given low-fidelity observations Y'. The integral can be calculated using Gaussian quadrature or other sampling methods as in X 25 which is slow and inaccurate.
Lemma 4. The joint likelihood of GAR for non-subset (and also umaligned) data can be decomposed into two independent GPs' likelihood
logp(y Y,, Y²a = C¹ _-_ dh
log(2T))
1
log
ÉSÉ & wTsw K
2
1 vec(V") 2 P")
T
veo( V')
T
(K' K + ES'E @ w'''w "[ vec((²") vec(P") (V")
veo
vec(Y') )
W
w
(8)
where L is the likelihood for low-fidelity data Y w I @ W Y the collection of fige-fi observations corresponding the imaginary low-fideiity outputs v 4 is the complement (with selection matrix X -E X') corresponding to low-fidelity outputs Y i.e. Y Y y") and
X x are the selection matrix for Y
0 0 ( & WTS'W
We preserve the proof in the appendix. Notice that ÉS'TT & WTSW
the low-right block of the predictive variance for the missing low-fidelity observations vee YY; We can easily understand the last part of the likelihood as a GP with accumulated uncertainty(variane) added to the corresponding missing points. Lemma naturally applies to AR when the outputs is scaler, where W p. S' 1. and S = 1.
5

---

Predictive posterior. Surprisingly, the posterior also turns out to be Gaussian distribution,
p I Yh. xxx 2T $ xs5 +r + @ s r
(9)
1 (voo(Z") - voc(z)) -oocz)) T
(st +r e ('vcc'(""" -voc)))
exp
where r and the mean of the predictive posterior Z, are given as follows,
r= (KK((K")1 'ET - (((K') w) Em @ I
vecl 22) = (K((K') @ w) ( vee vec(
ve )
(KI 11) (eec(Y")) - ww(
vec((
(10)
where Em and En are the selection matrices such that X E [X X",, X²
E [ x*)
W E & W, and Ki is the covariance matrix that K
K (Xx X X x))).
3.4 Autokrigeability, Complexity, and Further Acceleration
For subset structured data, the computational complexity of GAR is decomposed into two in- dependent TGPs for likelihood and predictive posterior. Thanks to the tensor algebra (mainly K & I) the complexity of the i-fidelity kernel matrix inversion is reduced to O m=1 (d ( (N")") instead of D(NN''))) For the non-subset case, the computational com- plexity in Eq. 8 unfortunately (( ))) where Nn is the number of the imaginary low-fidelity points. Nevertheless, due to the tensor structure, we can still use conjugate gradient 2
(K
to solve the
linear system efficiently.
Notice that the mean prediction Z in Eq. 9 does not depend on any output covariance matrixes (S mi St 3 m==1- which reassemble the autokrigeability (no knowledge transfer in noiseless cases based on the GAR framework. For applications where the predictive variation is not of interest, we can introduce a conditional independent output-correlation, i.e., St = S. = I and orthogonal weight matrixes, i.e., W W.. I, to reduce the computationally complexity further down to (((N*)) (see Appendix for detailed proof). We call this CIGAR as an abbreviation for conditional independent GAR. In empirical assessment, CIGAR is slightly worse than GAR due to the difficulty of ensuring WT W,, and numerical noise.
for mean predictions) 14 [
m
Related Work
GP for high-dimensional outputs is an important model in many applications such as spatial data modeling and uncertainty quantification. For an excellent review, the readers are referred to [27]] Linear model of coregionalization (LMC) 28 29 might be the most general framework for high- dimensional GP developed in the geostatistic community. LMC assumes that the full covariance matrix as a sum of constant matrixes timing dept-dee kernels. To reduce model complexity, semiparametrie latent factor models (SLFM) (30 simplify LMC by assuming that the matrixes are rank-1 matrixes. Higdon al. 311 further simplifies SLFM using singular value decomposition (SVD) on the output collection to fix the bases for the rank-1 I matrixes. To overcome the linear assumptions of LMC, the (implicit) bases can be constructed in a nonlinear fashion using manifold learning, e.g.. KPCA 32 and IsoMap [ or process convolution 34.30 Other approaches include multi-task GP, which considers the outputs as dependent tasks B in a framework similar to LMC and GP regression network (GPRN) 44 Il which proposes products of GPs to model nonlinear outputs, leading to nontractable models. Despite their success, the complicity of the above approaches at best ( + dd)) whereas some are ((N3dd) which cannot scale well to high-dimensional outputs for scientific data where d can be, says, 1 million. This problem can be well resolved by introducing tensor algebra 42 to form HOGP [8 scalable model inference, e.g., in GPRN [43
Multi- Multi-fidelity has become a promising approach to further reduce the data demands in building a surrogate model [K] and Bayesian optimization. The seminal autoregressive (AR) model of Kennedy K introduces a linear transformation to univariate high-fidelity outputs. This model was enhanced
by Le Gratict [5] by adopting a deterministic parametric form of linear transformation for the

---

efficient training scheme as introduced previously. However, it is unclear how AR can deal with high-dimensional outputs. To overcome the linearity of AR, Perdikaris al. [6 proposes nonlinear AR (NAR). It ignores the output distributions and directly uses the low-fidelity solution as an input for the high-fidelity GP model, which essentially a concatenating GP structure known as deep GP [4.. To propagate uncertainty through the multi-fidelity model, Cutajar et al. 25] uses expensive approximation inference, which makes it prone to overfitting and incapable of dealing with very large dimensional problems. For mulli-fidelity Bayesian optimization (MFBO), Poloczek et al. E and Kandasamy et al. [5] approximate each fidelity with a GP independently: Zhang et al. 4 use convolution kernel, similar the process convolution tA 36 to learn the fidelity correlations; Song et al. [ combine all fidelity data into one single GP to reduce uncertainty. However, most MFBO surrogates do not scale to high-dimensional problems because they are designed for one target (or at most a couple).
To deal with large dimensional outputs, e.g., spatial-temporay fields, Xing et al. 9 extend AR by assuming a simple additive structure and replacing the simple GPs with scalable multi-output GPs at the cost of losing the power for capturing the output correlations, leading to inferior performance and inaccurate uncertainty estimates; Xing et al. 12 propose Deep coregionalization to extend extracted from embedding the high-dimensional outputs onto a residual latent space using a proposed residual PCA; Wang et al. [ further introduce bases propagation along with latent process propagation in a deep GP to increase model flexibility at the cost of significant growth in the number of model parameters and a few simplifications in the approximated inference. Parussini et al. K generalize NAR to sighndimensionl problems. However, these methods lack a systematic way for joint model training, leading to instability and poor fitting for small datasets. Wu et al. 7 extend GP using neural process model high gigh-imensiooal and non- subset problem effectively. In scientific computing, multi-fidelity fusion has been implemented using stochastic collocation (SC) method [24]] for nigh-dimensional problems, which provides closed-form solutions and efficient design of experiments for the multi-fidelity problem. Xing et al. 7 showed that SC is essentially a special case of AR and proposed active learning to select the best subset for the high-fidelity experiments.
NAR by learning the latent process 30 [0]
To take the advances of deep learning neural network (NN) and being compatible with arbitrary propose an NN-based multi-task method that can naturally extend to MFBO. Li al. 23 further extend it Bayesian neural network (BNN) to MFBÓ. Meng and Kariiadaki 48) add a physics regularization layer, which requires explicit form of the problem PDEs, to improve prediction accuracy. To scale for high- dimensional problems with arbitrary dimensions in each fidelity, Li et al. [3] propose a Bayesian network approach to multi-fidelity fusion with active learning techniques for efficiency improvement.
multi-fidelity data (i.e., non-subset structure), Perrone et al. (2
Except for multi-fidelis fidtii-iiility fusion, AR can be used for model multi-variate problem 40 50 where GAR can also find its applications. GAR is general framework for GP-based multi-fidelity fusion of high-dimma dimensional outputs. Specifically, AR is a special case when setting W = pI and using a separable kernel; ResGP is special case of GAR by setting W I and S I; NAR is a special case of integrating out W with a normal prior and using a separable kernel; DC is a special case of GAR if it only uses one latent process, integrating out W as in NAR with separable kernel; MF-BNN is finite case of GAR if only one hidden layer is used. See Appendix C for the comparison between SOTA methods.
5 Experimental Results
To assess GAR and CIGAR, compare with the SOTA multi-fidelity fusion methods for high-
dimensional outputs including: (1) AR B. (2) NAR [66. (3) ResGP (9) (4) D [1). and (5)
MF-BNN [K] All GPs an RBF kernel for a fair comparison. Because the ARD kernel is separable, the AR and NAR are accelerated using the Kronecker product structure as in GAR for feasible computation. The original DC with residual PCA cannot deal with unaligned outputs, but can do so by using an independent PCA, which we called DC-I. Both DCs preserve 99% energy for dimension reductions. MF-BNN is conducted using its default setting. GAR, CIGAR, AR, NAR, and ResGP are implemented using Pytorch 5 All experiments are run on workstation with an AMD 5950x CPU and 32 GB RAM.
http://giitubb.om//vaxxiiggggi 5 http://ggihub.commmssbbb///DDDD
https//ppytorch.org/
7

---

250
ResEP
0.150
200
AR
0.125
RM5B
150
MF-SMN
DC1
RMSE
RMSE
0.100
0.04
0.075
100
CAR
0.050
50
0.02
0.025
0
16
32
8
16
32
0.000
16
32
#HF Samples
#HF Samples
#HF Samples
NAR
0.08
0.150
0.125
MF-3NN CICAR
RMSE
150
RMSE
RM5
0.100
0.075
100
0.050
50
0.02
0.025
0.00
0.000
Samples
Samples
Samples
250
0.04
MF-3NN
200
CIGAR
0.03
RMSE MMS015
150
RMSE
RMSE
0.06
0.02
100
0.02
8
16 #HF Samples
32
16
32
#HF Samples
#HF Samples
(a) Poisson's 's equation
(b) Burger's
(c) Heat
Figure 1: RMSE against increasing number of high-fidelity training samples for Poisson's. Burger's and heat equations with aligned outputs (top row), non-at -e--aligned outputs (middle row), and non-subset data (bottom row).
5.1 Multi-Fidelity Fusion for Canonical PDEs
We first assess GAR in canonical PDE simulation benchmarks, which produce high-dimensional spatial/spatial-eempool fields as model outputs. Specifically, we test on Burger's, Poisson's and the heat equations commonly used in the literature [12 The high fidelity results are obtained by solving these equations using finite difference on a 32 x 32 mesh, whereas the low fidelity by an x coarse mesh. The solutions on these grid points are recorded and vectorized outputs. Because the mesh differs, the dimensionality varies. To compare with the standard multi-fidelity method that can only deal with aligned outputs, we use interpolation to upscale the low-fidelity and record them the grid nodes. The corresponding inputs are PDE parameters and parameterized initial or boundary conditions. Detailed experimental setups can be found Appendix E.
We uniformly generate 128 samples for testing and 32 for training. We increase the high-fidelity training samples to the number of low-fidelity training samples 32. The comparisons conducted five times with shuffled samples. The statistical results (mean and std) of the RMSE are reported in Fig. D GAR and CIGAR outperform the competitors with significant margin, up 6x re- duction in RMSE and also reaching the optimal performance with a maximum of high-fidelity samples, indicating successful fusion of low- and high-fidelite CIGAR is slightly worse than GAR possibility due to the lack of hard constraints on the orthogonality of its weight matrixes during implementation. As we have discovered in the literature. AR consistently performs well. With flexible linear transformation, GAR outperforms AR while inheriting its robustness, leading to the best performance. For the unaligned output, MF-NNN showed slightly worse performance than in the aligned cases, highlighting the challenges of the unaligned outputs. In contrast, GAR and CIGAR show almost identical performances for both cases. Nevertheless, MF-BNN also shows good performance compared to the rest of the other methods, which is consistent with the finding in [I interesting to see that for the non-subset data, the capable methods show better performances than in the subset cases. GAR and CIGAR still outperform the competitors with clear margin.

---

To approximately assess the performance under active learning process. We instead generate training samples in Sobol sequence [54]]. The results are shown Appendix where GAR and CIGAR also outperform the other methods by a large margin.
5.2
Multi-Fidelity Fusion for Real-World Applications
Optimal topology structure is the optimized layout of materials, e.g., alloy and concrete, given some design specifications, e.g., external foree and angle. This topology optimization a key technique in mechanical designs, but it is also known for its high computational cost, which renders the need for multi-fidelity fusion. We consider the topology optimization of cantilever beam with the location of the point load, the angle of the point load, and the filter radius 551 as system inputs. The low-fidelity use 16 x 16 regular mesh for the finite element solver, whereas the high-fidelity x 64. Please see Appendix F..3 for detailed setup.
As in the previous experiment, the RMSE statistics against an increasing number of high-fidelity training samples are shown in Fig. 2 It is clear that GAR outperforms the competitors with large
margin consistently. CIGAR can be good as GAR when the number of training samples is large.
0.5
NAR
DC-A
MF-BNN
RMSE
RMSE
CIGAR
0.3
0.2
0.1
32
0.1
0.1
Samples
ØHF Samples
#HF Samples
Figure 2: RMSE with low-fidelity training sample number fixed to (32.64,288)
Steady-state 3D solid oxide fuel cell, which solves complex coupled PDEs including Ohm's 's law, Navier-Stokes equations, Brinkman equation, Maxwell-Stefie diffusion, and convection simultane- ously, is key model for modern fuel cell optimization. The model was solved using finite elements in COMSOL. The inputs were taken to the electrode porosities, cell voltage, temperature, and pressure in the channels. The low-fidelity experiment is conducted using 3164 elements and relative tolerance 0.1, whereas the high-fidelity uses 37064 elements and relative tolerance of 0.001. The outputs are the coupled fields of electrolyte current density (ECD) and ionic potential (IP) in the x z plane located at the center of the channel.
The RMSE statistics are shown in Fig. 5a which, again, highlights the superiority of the proposed method with only four high-fin high-fideitty data training samples. further assess the model capacity for non-structured outputs, we keep only the ECD (Fig. 55 and IP (Fig. 5c) in the low-fidelity training data to rise the challenges of predicting high fidelity ECD+IP fields. We can see that removing some low-fidelity information indeed increases the difficulties, especially when removing ECD, where MF-BNN outperforms GAR and CIGAR with a small number of training data. As soon as the training number increases, GAR and CIGAR become superior again.
Plasmonic nanoparticle arrays is a complex physical simulation that calculates the extinction and scattering efficiencies Qezt and Que for plasmonic systems with varying numbers of scatterers using coupled dipole approximation (CDA), which is method for mimicking the optical response of an array of similar, non- -on-magnetic metallic nanoparticles with dimensions far smaller than the wavelength of light (here 25 nm). Qext and Qse are defined the Qols in this work. Please see Appendix E.5 for detailed experiment setup. We conducted the experiments 5 times with shuffled samples, and we fixed the number of low-fidelity training samples to 32, 64, and 128 and gradually increase the high-fidelity training data from 4 to 32, 64, and 128. We can see in Fig GAR outperforms others by a clear margin, especially when the high-fidelity data contains only 4 samples. When there is large training sample dataset, CIGAR can be as excellent as GAR.

---

0.5
0.5
MF BNN
0.5
0.4
CIGAR
RMSE
0.3
CIBAR
RMSE
0.3
GAR
RMSE
0.3-
GAR
0.2
0.2
8
16
32
8
16
32
64
4816
32
64
128
#HF Samples
#HF Samples
#HF Samples
Figure 3: RMSE against an increasing number high-fidelity training samples with low-fidelity training sample number fixed to /32,64,128) for Plasmonic nanoparticle arrays simulations.
0.36
500
0.34
200
RMAE
0.30
RMASE
300
0.28
50
100
0.24
503
300
160
280
750
#oopoch
#oopoch
#apoch
(a) Poisson's
(b) Topology optimization
(c) SOFC
Figure Testing RMSE against an increasing number of training epochs for SOFC, topology optimization, and Poisson datasets.
5.3 Stability Test
As non-parametric model, expect GAR and CIGAR to have stable performance against overfitting compared to the NN-based methods. In this section, we show the testing RMSE against the training epoch for GAR, CIGAR, and ME-BNN for the previous Poisson's 's equation, SOFC, and topology optimization. The experiments are repeated five times to ensure fairness. The results are shown in
Fig. We can see clearly that GAR and CIGAR are more stable than MF-BNN in almost all cases. The most notables are the converge rate of GAR and CIGAR, which is more than 10x faster if we look the topology optimization and SOFC cases. For Poisson's equation, the MF-BNN is not likely to match the performance of GAR and CIGAR regardless of the large number of training epochs being used. For the SOFC, MF-BNN. and topology optimization, MF-BNN might be able to match GAR given very large epoch number, making it bad choice that consumes expensive computational resources.
500
500
500
kusp
400
LX4
400
400
COAA
CAN
RMSE
300
RMSE
300
RMSE
300
100
Samples
Samples
#HF Samples
(a) ECD+IP
(b) ECD
(c) IP
Figure 5: RMSE for SOFC with low-fidelity training sample number fixed to 32.
6 Conclusion
propose GAR, the first AR generalization to arbitrary outputs and non-subset multi-fidelity data with closed-form solution, and CIGAR, an efficient implementation by revealing the autokrigeability AR. Limitation of this work is scalability w.r.t to samples, lack of active learning [3) and the applications to broader problems of time series and transfer learning 49 50 using AR-based methods.
10

---

References
[1] Bobak Shahriari, Kevin Swersky, Ziyu Wang. Ryan P. Adams, and Nando Freitas. Taking the Human Out of the Loop: Review of Bayesian Optimization. 104(1):148-175, ISSN 0018-9219, 1558-2256. doi: 10. R..110/9PRO...015.22444
URL http::////eeeeeellooeeeeee ieee. htt.:///w......ooooooooooooooaaa
[2] Apostolos F. Psaros. Xuhui Meng, Zongren Zou, Ling Guo, and George Em Karniadakis. Uncertainty Quantification Scientific Machine Learning: Methods, Metrics, and Comparisons.
[3] M. Kennedy, Predicting the output from complex computer code when fast approximations available. 87(1):1-13. ISSN 0006-3444, 1464-3510. doi: 10. 1009//iiome//87..11
[4] Matthias Poloczek, Jialei Wang. and Peter Frazier. Multi-information source optimization. Advances in neural information processing systems, 30, 2017.
[5] Jialin Song, Yuxin Chen, and Yisong Yue. A General Framework for Multi-fidelity Bayesian Optimization with Gaussian Processes. In The 22nd International Conference Artificial Intelligence and Statistics, pages 3158-3167. URL htt.:/tttp:pppee//liiatt http:/ngw....ll.eeerereeeee ppress/v8//ooggegee
.wr.
html
[6] L. Parussini, D. Venturi, P. Perdikaris, and G.E. Karniadakis. Multi-fidelity Gaussian process regression for prediction of random fields. 336(C):36 50. ISSN 0021-9991. doi: 10. wt1..jcc...11...066
[7] W. Xing, M. Razi, R. M. Kirby, K. Sun, and A. A. Shah. Greedy nonlincar autoregression for multifidelity computer models at different scales. 1:100012, ISSN 2666-5468, doi: 10. 1016 egyai. 2020. 100012. URL httpp://:://wwt httpa/pppp/ww....ccceeeerrr....o com http:////aaaattetttttaaaaaa/a///
[8] Zheng Wang, Wei Xing, Robert Kirby, and Shandian Zhe. Milti-Fidelity High-Order Gaussian Processes for Physical Simulation. In International Conference on Artificial Intelligence and Statistics, pages 847- 855, PMLR. URL http/tttp//ppoceeediiiii
mlr www.ppressss... v100 htt1:///waaaeee wwtp//aannnn.....e html
[9] W. W. Xing, A. A. Shah, P. Wang, S. 7he, Q Fu, and R. M. Kirby. Residual gaussian process: A tractable nonparametric bayesian emulator for multi-fidelity simulations. 97:36-56, ISSN 0307-904X. doi: w..101666..pm.2201 2021.03.21... http::/:: wwwp::/weeeeeccccccc......eeee.. 50307904X21001722
[10] Giselle Monández-Goding Chanyoung Park, Nam-Ho Kim, and Raphael multi-fideling fidelity models.
Haftka. Review of
[11] B. Peherstorfer, K. Willeox, and M. Gunzburger. Survey of Multifidelity Methods Uncertainty Propaga-
tion, Inference, and Optimization. 60(3):550-511, ISSN 0036-1445. doi: 10. 10.1137/16M1082466
[12] Wei W. Xing. Robert M. Kirby, and Shandian Zhe. Deep coregionalization for the emulation of simulation- based seaporlll fields. 109944, ISSN 0021-1991111 doi: 10.1016/j.jcp jc..l URL https://Linkinghut
elsevier com
[13] Shibo Li, Robert M Kirhy, and Shandian Zhe. Deep multi-iidelity active learning of high-dimensional outputs. arXiv preprint arXiv:2012.00901, 2020.
[14] Mauricio A. Alvarez, Lorenzo Rosasco, and Neil D. Lawrence. Kernels for Vector Valued Functions: A Review.
[15] Loic Le Gratiet. Bayesian Analysis of Hierarchical Multifidelity Codes. 1(1):244-269, ISSN 2166-2525. doi: 10.1 10.1137/120884122
[16] Paris Perdikaris, M Raissi, Andreas Damianou, N D. Lawrence, and George Karniadakis. Nonlinear Information Fusion Algorithms for Data- Eff-cEfficien doi: 10. (009///sa..001.007...
Modelling, volume 473. Royal Society.
[17] Carl Edward Rasmussen and Christopher K I Williams. Gaussian Processes for Machine Learning. page 266.
[18] Shandian Zhe, Wei Xing, and Robert M. Kirby. Scalable High- High-Order Gaussian Process Regression. In The 22nd International Conference Artificial Intelligence and Statisties, pages 2611-2620 PMLR. URL http://proceedlngs. . mlr (etress/v89/2he1// html
[19] Tamara Gibson Kolda. Multilinear operators for higher-order decompositions. URL http: hthp:://wwe www. osti
[20] Zenglin Xu, Feng Yan, Yuan, and Qi. Infinite Tucker Decomposition: Nonparametric Bayesian Models for Multiway Data Analysis.

---

[21] Andrew Gordon Wilson and Hannes Nickisch. Kernel interpolation for scalable structured Gaussian processes (KISS-GP). In Proceedings of the 32nd International Conference International Conference on Machine Learnin Volume 37. ICML' 15, pages 1775-1784, JMLR.org.
[22] Valerio Perrone, Rodolphe Jenatton, Matthias Seeger, and Cédric Archambeau. Scalable hyperparameter transfer learning. Advances in neural information processing systems, 31, 2018.
[23]
Shibo Li, Wei Xing, Robert Kirby, and Shandian Zhe. Fulti-Fidelity Bayesian Optimization via Deep Neural Networks. 13:8521-85311 URL https://papers nips. sttpss..c//aperr/22///ssss (00l.deeb0Sarr3dd5eaeeeeeeeeee
Abstract. html
[24] Akil Narayan, Claude Gittelson, and Dongbin Xiu. A Stochastic Collocation Algorithm with Multifidelity Models. (695):4495-5222 ISSN 1064-8275. doi: 10.1137/130929461
[25] Kurt Cutajar, Mark Pullin, Andreas Damianou, Neil Lawrence, and Javier González. Deep Gaussian Processes for Multi-fidelity Modeling.
[26] Andrew Wilson and Ryan Adams. Gaussian process kernels for pattern discovery and extrapolation. In International conference on machine learning, pages 1067-1075. 1075. PMLR, 2013.
[27] Mauricio A. Álvarez, Lorenzo Rosasco, and Neil D. Lawrence. Kernels for Vector- Valued Functions:
A Review. 4(3):195 (43):195-266. 266. ISSN 1935-8237, 1935-8245. doi: 10.1561/220000033. URL http:
//wuw. www.nnooosbbbeee....eee com htmmm/aricce//eealll//// wtt.D/DettaaisMMM------ MMLL-
[28] Michel Goulard and Mare Voltz. Linear coregionalization model: tools for estimation and choice of variooaa matrix. Mathematical Geology, 24(3):269-285 1992.
[29] Pierre Goovaerts al. Geostatistics for natural resources evaluation. Oxford University Press Demand, 1997.
[30] Yee Whye Teh, Matthias Seeger, and Michael Jordan. Semiparametric latent factor models. In International Workshop on Artificial Intelligence and Statistics, pages 333-340. PMLR, 2005.
[31] Dave Higdon, James Gattiker, Brian Williams, and Maria Rightley. Computer Model Calibration Using High-Ding Bige--Dinnsional Output. 103(482):570-588 ISSN 00122-145, 1537-274X. doi: 10.1198/ 01621450700000088..
[32] W.W. Xing. V. Triantafyllidis A.A. Shah, P.B. Nair, and N. Zabaras. Manifold learning for the emulation of spatial fields from computational models. 326:666 690, ISSN 0021-9991, doi: 10. 60. jcp.2016.07.044 URL /ttp:::////// elsevier http://...........././/////////.
[33] Wei Xing. Akeel A. Shah, and Prasanth B. Nair. Reduced dimensional Gaussian process emulators of parametrized partial differential equations based on Isomap. 471(2174)220140699 ISSN 1364-5021, 1471-2946. doi: 10. 10. 1098/0999...20 201...0997
[34] Mauricio Álvarez, Wil Ward, and Cristian Guarnizo. Non-linear process convolutions multi-output gaussian processes. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1969-1977. PMLR, 2019.
[35] Phillip Boyle and Marcus Frean. Dependent gaussian processes. Advances in neural information processing systems, 17, 2004.
[36] Dave Higdon. Space and Space-Time Modeling using Process Convolutions. In Clive W. Ander- son. Vie Barnett, Philip C. Chatwin, and Abdel Н. El-Shaarawi, editors. Quantitative Methods for Current Environmental Issues, pages 37-56. Springer London. ISBN 978-1-4471-1171-9 17471- 0657-9. doi: 10. (0.1007/978-1-4444 -4771-----7-77
[37] Edwin V Bonilla, Felix V Agakov, and Christopher KI Williams. Kernel multi-tas learning using task spec-ipe features. In Artificial Intelligence and Statistics, pages 43 50. PMLR, 2007.
[38] Barbara Rakitsch, Christoph Lippert, Karsten Borgwardt, and Oliver Stegle. It is all in the noise: Efficient multi-task gaussian process inference with structured residuals. Advances in neural information processing systems, 26, 2013.
[39] Ping Li and Songcan Chen. Hierarchical Gaussian Processes model for multi task learning. 74:134 144, ISSN 0031 -2033 doi: 10. wtt.://....oo....20000 wtt..2001.9.00.... URL httts:://inkkii elsevier . wetrieewe/pi1//S00110000..
12

---

[40] Andrew Gordon Wilson, David A. Knowles. and Zoubin Ghahramani. Gaussian process regression networks. In Proceedings of the 29th International Coference International Conference Machine
Learning, ICML'12, page 1139-1 146, Madison, WI. USA, 2012. Omnipress. ISBN 9781450312851.
[41] Trung Nguyen and Edwin Bonilla. Efficient variational inference for gaussian process regression networks. In Artificial Intelligence and Statistics, pages 472-480. PMLR, 2013.
[42] Tamara G. Kolda and Brett W. Bader. Tensor Decompositions and Applications. 51(3):455-50 500. ISSN 0036-1445, 1095-7200.
doi: 10.1137/07070111X
[43] Shibo Li, Wei Xing, Robert M. Kirby, and Shandian Zhe. Scalable Gaussian Process Regression Nei- works. volume 3. pages 2456-2462
doi: 200000040 URL URLhhtts:://ww/ ijcai org/
seteeedingg//0020//
[44] Andreas Damianou. Deep Gaussian processes and variational propagation of uncertainty.
[45] Kirthevasan Kandasamy, Gautam Dasarathy, Junier Oliva, Jeff Schneider, and Baraháss Póczos. Gaussian process bandit optimisation with multi-fidelity evaluations. Advances in neural information processing systems, 29, 2016.
[46] Ychong Zhang, Trong Nghia Hoang, Bryan Kian Hsiang Low, and Mohan Kankanhalli. Information based multi-fidelity bayesian optimization. In NIPS Workshop on Bayesian Optimization, 2017.
[47] Dongxia Wu, Matteo Chinazzi, Alessandro Vespignani, Yi-An Ma, and Rose Yu. Multi fideliy hierarchical neural processes. arXiv preprint arXiv:2206.0443 2022.
[48] Xuhui Meng and George Em Karniadakis. composite neural network that learns from multi-fidelity data: Application to function approximation and inverse PDE problems. 401:109020,, ISSN 0021-9991 doi: je. 109000. URL http:://ww..ciiemmddreee.. htt.:/awweeeeetteooc/cc/eeeaatta 50021999119807966
[49] James Requeima, William Tebbutt, Wessel Bruinsma, and Richard E Turner. The gaussian process autoregressive regression model (gpar). In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1 1860-1869. PMLR, 2019.
[50] Rui Xia, Wessel Bruinsma, William Tebbutt, and Richard E Turner. The gaussian process latent autoregres- sive model. In Third Symposium on Advances in Approximate Bayesian Inference, 2020.
[51] Rui Tuo, C. F. Jeff Wu, Dan Yu. Surrogate Modeling of Computer Experiments With Different Mesh Densities. 56(3):372-880. ISSN 0040-1706, 1537-2723. doi: 10. 706.2003.776620.......
[52] Mehmet Onder and Hitay Ozbay. Proper orthogonal decomposition reduced order modeling: 2d heat flow. In Proceedings 2003 IEEE Conference on Control Applications, 2003. CCA 2003., volume 2, pages 1273-1277. IEEE.
[53] Maziar Raissi and George Em Kamiadakis. Machine Learning of Linear Differential Equations using Gaussian Processes. 348:683-698 693. ISSN 0021-9991. doi: 10/gbzfgr.
[54] I.M Sobol', On the distribution of points a eube and the approximatc evaluation of integrals. 7(4):86 ISSN 0041-5553, doi: 00011066600011-15555777991144440 URL httpttt//iiiiiigguuu elsevier. com/
[55] Tyler Bruns and Daniel Tortorelli. Topology optimization of non-linear clastic structures and compliant mechanisms, Computer Methods in Applied Mechanics and Engineering, 190(26):3443 3459, 2001. ISSN 0045-7825. doi: https://doi.org//// http:///....oo....00000000000/./ (000044 URL http:/ /www
sciencedirect. com wttpeeett...ccccccieeeeeeeeeeeee http://caaaacaaarttaaaaaa//////p
[56] TJ Chung. Computational fluid dynamics. Cambridge university press.
[57] N Sugimoto. Burgers equation with fractional derivative; hereditary effects nonlinear acoustic waves. 225:631-653.
[58] Kai Nagel. Particle hopping models and traffic flow theory. 53(5):4655.
[59] S Kutluay, AR Bahadir, and A Özdees. Numerical solution of one-dimensional burgers cquation: explicit and exact-explicit finite difference methods. 103(2):251-61..
[60] A. A. Shah, W. W. Xing, and V. Triantafyllidis. Reduced-order modelling of parameter dependeet, linear and nonlinear dynamic partial differential equation models. 473(2200):20160809 ISSN 1364-5021, 1471-2946. doi: 10. (098//spa.2016.08099
13

---

[61] Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations.
[62] Steven C Chapra. Raymond P Canale, et al. Numerical methods for engineers. Boston: McGraw-Hil Higher Education.
[63] S Persides. The laplace and poisson equations in schwarzschild's time. 43(3):571-578.
[64] E. Lagaris, Likas, and Fotiadis. Artificial Neural Networks for Solving Ordinary and Partial Differential Equations. 95)::877-1000, Scpt./1998. ISSN 1045-9227. doi: 10.1 109/72.712178.
[65] Frank Spitzer. Electrostatic capacity, heat flow, and brownian motion. 10-111..
[66] Krzysztof Burdzy, Zhen-Qing Chen, John Sylvester, al. The heat equation and reflected brownian motion in time-dependent domains. 32(1B):775-80.
[67] Fischer Black and Myron Scholes. The pricing of options and corporate liabilities. 81(3):637-654
[68] Wei Xing, Shireen Y. Elhabian, Vahid Keshavarzzadeh, and Robert M. Kirby. Gauss--a Process: Learning Interpretable Shared Hidden Structure Across Data Spaces for Design Space Analysis and Exploration. 142(8). ISSN 1050-0472, 1528-9000 9001. doi: 11111/144446744
[69] Erik Andreassen, Anders Clausen, Mattias Schevenels, Boyan S. Lazarov, and Ole Sigmund. Efficient topology optimization in matlab using lines of code. Structural and Multidisciplinary Optimization, 43 (1)1-11,, Jan 2011 ISSN (615-1488, 1110//00008-000-5050
doi: 10.1007/s001588 0007//000118-00-059444 URL https:://di..rg//
[70] Martin Philip Bendsoe and Ole Sigmund. Topology optimization: Theory, methods and applications. Springer, 2004.
[71] Charles aggregates, JOSA A, 23(2):349333 358, 2006.
Charles-Antoine Antoine Guérin, Pierre Mallet, and Anne Sentenac. Effective- theory for finite-size
[72] Mani Razi, Ren Wang, Yanyan He, Robert M. Kirby, and Luca Dal Negro. Optimization of Large-Scale Vogel Spiral Arrays of Plasmonic Nanoparticles. 14(1):253- 261.. ISSN 1557-1955, 1557-1963, doi: 10.1007/s1 146--018----18811 018-0799-y.
[73] Aristi Christofi, Felipe A Pinheiro, and Luca Dal Negro. Probing scattering resonances of vogel's spirals with the green's 's matrix spectral method. Optics letters, 41(9):1933-195 2016.
[74] Dongxia Wu, Liyao Gao, Xinyue Xiong, Matteo Chinazzi, Alessandro Vespignani, Yi-An Ma, and Rose Yu. Quantifying uncertainty in deep spatiotemporal forecasting. arXiv preprint arXiv:2105.11992) 2021.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how answer these questions. For cach question, change the default (TODO) to [Yes], [No] or [N/A] You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license the code and datasets? [Yes] See Section ??.
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license the code and datascts? [N/A]
Please do not modify the questions only use provided macros your answers. Note that the Checklist section does not count towards the page limit, In your paper, please delete this instructions block and only keep the Checkliss list section heading ahove along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately refiect the paper's contribu- tions and scope? [Yes] See contributions. abstract, and introduction
(b) Did you describe the limitations of your work? [Yes] See the conclusion and complexity analysis section
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We do not see an obvisou negative societal impacts as it is fundamental and quite theoretical.
14

---

(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] Please see Appendix
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (cither in the supplemental material or URL)? [Yes] Please see supplementary materials
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they chosen)? Yes] Please see the Appendix
(c) Did you report error bars (e.g., with respect the random seed after running experiments multiple times)? [Yes] Please see experimental section
(d) Did you include the total amount of compute and the type resources used (c.g., type GPUs, internal cluster, or cloud provider)? [Yes] Please see experimental section
4. If you using existing assets (e.g., code, data, models) curatingfrelcasing new assets...
(a) your work uses existing asscts, did you cite the creators? [Yes] Please see the experimental
section
(b) Did you mention the license of the assets? [Yes]
(c) you include any new assets either in the supplemental material or as URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're us- ing/curating? [Yes] see experiment senction and the Appendix
(e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content? [N/A] We do not use such data
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "xJ5gWEG1WFb", "reviewer": "Reviewer_QEGg", "summary": "The authors proposed Generalized Autoregression (GAR) to handle high-dimensional outputs for multi-fidelity fusion problems. GAR is a GP-based approach that employed tensor algebra to efficiently model covariances across the high dimensions. To handle situations where high fidelity data is not a subset of low fidelity data, they propose a latent variable approach and with a simple trick they proposed CIGAR, which is a computational more efficient version of GAR.\n\nExperiments of GAR and CIGAR showed promising results compared to SOTAs. ", "questions": "The idea of using imaginary variables to match \"missing\" data seems very similar to the use of inducing variables in sparse GPs, can you comment a bit on that? ", "limitations": "Currently only works for Gaussian likelihood. Not sure how straight forward or not it might be to generalised to arbitrary likelihood in high dimensinoal output space.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- clear and thorough comparison of GAR with existing methods in sec.4 \n- exact solutions are obtainable (when subset data exists)\n- although tensor algebra papers tend to be tedious to read, the authors defined the notations clearly and communicated effectively between equations.\n- the problem is clearly defined, and worthwhile contribution to handle high dim output multi fidelity fusion problem. \n\nWeakness:\n- there are some typos around, e.g. line 38 there is a 1. after citation [9], or in line 55 the word \"Problems\" is capitalised, but these do not affect the reading experience at all.\n- seems to only work for regression problems for now - if non-gaussian likelihood is needed, not sure how straight forward it is to translate the whole tensor algebra business to that. Worth mentioning in the paper a bit as well I guess? Nonetheless, focusing on regression data is a worthwhile contribution already in my opinion.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TnvA2uSe6Aq", "reviewer": "Reviewer_ebFJ", "summary": "This paper designs a generalized autoregression method (GAR) and its efficient implementation CIGAR to deal with non-structured high-dimensional outputs and non-subset multi-fidelity data. The result shows their method outperforms the baselines among several benchmarks at the highest fidelity level.", "questions": "The paper mentioned uncertainty quantification. But why there's no evaluation of uncertainty quantification for experiments? It will be great if the author could include other metrics for accuracy (negative log-likelihood) and uncertainty quantification (Continuous Ranked Probability Score [1] or mean interval score [1,2]).\n\n[1] Gneiting, Tilmann, and Adrian E. Raftery. \"Strictly proper scoring rules, prediction, and estimation.\" Journal of the American statistical Association 102.477 (2007): 359-378.\n[2] Wu, Dongxia, et al. \"Quantifying uncertainty in deep spatiotemporal forecasting.\" arXiv preprint arXiv:2105.11982 (2021).\n", "limitations": "No potential negative societal impact I can see.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nWell-written paper in general.\nInclude 5 experiments. 2 with real-world data.\n\nWeaknesses: \nMy main concern about the paper is its novelty. Although the author didn't overclaim their contributions \"Generalization of the AR for arbitrary non-structured high-dimensional outputs\" and \"Generalization to non-subset multi-fidelity data for the AR\", which are all for the classic autoregression model specifically. But there's existing non-AR work [1] that is able to handle both non-structured high-dimensional outputs and non-subset multi-fidelity data [1] besides MF-BNN.\n\n[1] Wu, Dongxia, et al. \"Multi-fidelity Hierarchical Neural Processes.\" arXiv preprint arXiv:2206.04872 (2022).\n\nBesides, there's no big improvement between the proposed method and baselines (MF-BNN, ResGP) for some of the experiments. Only one metric RMSE is evaluated. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JcKSMnuDeLg", "reviewer": "Reviewer_UYDR", "summary": "The authors here present a generalized autoregressive model to conduct multi-fidelity prediction on data with high-dimensional outputs via tensor decomposition. A different non-subset multi-fidelity data setting is explored. Based on autokrigeability, a more efficient implementation is proposed.", "questions": "1. How do you decide the size of the tensor? ", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1)\tProblems are clearly stated and addressed. \n2)\tThe non-subset data scenario is addressed here for the first time.\n3)\tTensor GP is applied here to reflect possible underlying structures of the data.\nWeakness:\n1)\tThe notations are a bit confusing, like Z, F and their subscripts.\n2)\tSparse variational inference might further improve the inference efficiency here.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept