Title: Bayesian Kernelized Tensor Factorization as Surrogate for Bayesian Optimization
Venue: NeurIPS 2024
Accepted: no
Keywords: Bayesian Optimization

Kernelized Tensor Factorization

Markov chain Monte Carlo

Surrogate Model
Abstract: Bayesian optimization (BO) mainly uses Gaussian processes (GP) with a stationary and separable kernel function (e.g., the squared-exponential kernel with automatic relevance determination [SE-ARD]) as the surrogate model. However, such localized kernel specifications are deficient in learning complex functions that are non-stationary, non-separable and multi-modal. In this paper, we propose using Bayesian Kernelized Tensor Factorization (BKTF) as a new surrogate model for Bayesian optimization (BO) in a $D$-dimensional grid with both continuous and categorical variables. Our key idea is to approximate the underlying $D$-dimensional solid with a fully Bayesian low-rank tensor CP decomposition, in which we place GP priors on the latent basis functions for each dimension to encode local consistency and smoothness. With this formulation, the information from each sample can be shared not only with neighbors but also across dimensions, thus fostering a more global search strategy. Although BKTF no longer has an analytical posterior, we efficiently approximate the posterior distribution through Markov chain Monte Carlo (MCMC). We conduct numerical experiments on several test functions with continuous variables and two machine learning hyperparameter tuning problems with mixed variables. The results show that BKTF offers a flexible and highly effective approach to characterizing and optimizing complex functions, especially in cases where the initial sample size and budget are severely limited.

Abstract

Bayesian optimization (BO) mainly uses Gaussian processes (GP) with a stationary and separable kernel function (e.g., the squared-exponential kernel with automatic relevance determination [SE-ARD]) as the surrogate model. However, such localized kernel specifications are deficient in learning complex functions that are non-stationary, non-separable and multi-modal. In this paper, we propose using Bayesian Kernelized Tensor Factorization (BKTF) as a new surrogate model for Bayesian optimization (BO) in a D -dimensional grid with both continuous and categorical variables. Our key idea is to approximate the underlying D -dimensional solid with a fully Bayesian low-rank tensor CP decomposition, in which we place GP priors on the latent basis functions for each dimension to encode local consistency and smoothness. With this formulation, the information from each sample can be shared not only with neighbors but also across dimensions, thus fostering a more global search strategy. Although BKTF no longer has an analytical posterior, we efficiently approximate the posterior distribution through Markov chain Monte Carlo (MCMC). We conduct numerical experiments on several test functions with continuous variables and two machine learning hyperparameter tuning problems with mixed variables. The results show that BKTF offers a flexible and highly effective approach to characterizing and optimizing complex functions, especially in cases where the initial sample size and budget are severely limited.

1 Introduction

For many applications in science and engineering, such as emulation-based studies, experiment 21 design, and automated machine learning, the goal is to optimize a complex black-box function f ( x ) 22 in a D -dimensional space, for which we have limited prior knowledge. The main challenge in 23 such optimization problems is that we aim to efficiently find the global optima, while the objective 24 function f is often gradient-free, multimodal and computationally expensive to evaluate. Bayesian 25 optimization (BO) offers a powerful statistical approach to these problems, particularly when the 26 observation budgets are limited [1, 2, 3]. A typical BO framework consists of two components-a 27 surrogate model and an acquisition function (AF)-to balance exploitation and exploration. The 28 surrogate is a probabilistic model that allows us to estimate f ( x ) with uncertainty at a new location 29 x , and the AF is used to determine which location to query next. 30

Gaussian process (GP) regression is the most widely used surrogate for BO [3, 4], thanks to its 31 appealing properties in providing analytical derivations and uncertainty quantification (UQ). The 32 choice of kernel/covariance function is a critical decision in GP models; for multidimensional 33

BO problems, perhaps the most popular kernel is the ARD (automatic relevance determination)34

Squared-Exponential (SE) or Matérn kernel [4]. Although this specification has certain numerical 35

advantages and can help automatically learn the importance of input variables, a key limitation is 36

、

(a)

truefunction

(c)

GPαEI

GPαUCB

BKTF

estimationsurface

0.5

D

0

0

-1

0

0

02

0

-0.5

1

-0.5

1

-0.5

1

-0.5

1

-0.5

-1

1.5

2

1.5

1.5

1.5

-1.5

C1

2-1

C2

I

C1

2-1

C2

C1

2 -1

C2

C1

2-1

C2

(b)

0.15

GPαUCB

GP αEI

I

surface

X10-3

2

0.5

regret

0.05

0.1

BKTF

x10-3

210

1

-0.5

0

0.5

1.5

1

1

-0.5

0

0

-0.5

-1

1

-0.5

0

0.5

0

0

1.5

0

1.5

1.5

-0.5

0

1020304050

C1

2 -1

C2

C1

2 -1

C2

C1

2 -1

C2

no.ofevaluations

that it implies/assumes that the underlying stochastic process is stationary and separable, and the 37 value of the covariance function between two random points quickly goes to zero with the increase 38 of input dimensionality. These assumptions can be problematic for complex real-world processes 39 with long-range dependencies, because estimating the underlying function with a simple ARD kernel 40 would require a large number of observations. A potential solution to address this issue is to use 41 more flexible kernel structures. The additive kernel, for example, is designed to characterize a more 42 'global' structure by restricting variable interactions [5]. However, in practice using additive kernels 43 requires strong prior knowledge to determine the proper interactions and involves a large number 44 of kernel hyperparameters to learn [6]. Another emerging solution is to use deep GP [7], such as in 45 [8, 9]; however, learning deep GP often becomes a more challenging task due to the inference of 46 latent layers. In addition, these GP related surrogates can be more deficient to tune when taking into 47 account both continuous and categorical inputs. 48

49

50

51

52

53

54

55

56

57

58

59

60

61

In this paper, we propose using Bayesian Kernelized Tensor Factorization (BKTF) as a flexible and adaptive surrogate model for BO in a D -dimensional Cartesian product space (i.e., grid) when D is relatively small (say D ≤ 10 ). BKTF is initially developed for modeling multidimensional spatiotemporal data with UQ, for tasks such as spatiotemporal kriging/cokriging [10, 11]. This paper adapts BKTF to the BO setting, and our key idea is to characterize the multivariate objective function f ( x ) = f ( x 1 , . . . , x D ) for a specific BO problem using the low-rank tensor CANDECOMP/PARAFAC (CP) factorization with random basis functions. Unlike other basis-function models that rely on known/deterministic basis functions [12], BKTF uses a hierarchical Bayesian framework to achieve high-quality UQ in a more flexible way-GP priors are used to model the basis functions, and hyperpriors are used to model kernel hyperparameters in particular for the lengthscale that characterizes the scale of variation. In addition, BKTF also provides a natural solution for categorical variables, for which we can simply introduce an inverse-Wishart prior on the covariance matrix of the basis functions.

Figure 1 shows the comparison between BKTF and GP surrogates when optimizing a bivariate 62 function ( D = 2 ) that is nonstationary, nonseparable, and multimodal. The details of this function 63 and the BO experiments are provided in Appendix C. This 2 D case clearly shows that GP surrogate 64 is limited by the local kernel and becomes ineffective in finding the global solution, while BKTF 65 offers superior flexibility and adaptability to characterize the multidimensional process from limited 66 data. Unlike GP-based surrogate models, BKTF no longer has an analytical posterior; however, 67 efficient inference and acquisition can be achieved through Markov chain Monte Carlo (MCMC) 68 in an element-wise learning way, in which we update basis functions and kernel hyperparameters 69 using Gibbs sampling and slice sampling, respectively [11]. For optimization, we first use MCMC 70 samples to approximate the posterior distribution of the entire tensor and then naturally define the 71 upper confidence bound (UCB) of the posterior as AF. This process is feasible for many real-world 72 applications that can be studied in a discretized tensor product space, such as experimental design. 73

We conduct extensive experiments on both standard optimization and ML hyperparameter tuning 74 tasks. Our results show that BKTF achieves a fast global search for optimizing complex objective 75 functions with limited initial data and budget. 76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

2 Bayesian optimization

Let f : X = X 1 × . . . ×X D → R be a black-box function that could be nonconvex, derivative-free, and expensive to evaluate. BO aims to address the global optimization problem:

BO solves this problem by first building a probabilistic model for f ( x ) (i.e., surrogate model) based on initial observations and then using the predictive distribution to decide where in X to evaluate/query next. The overall goal of BO is to find the global optimum of the objective function using as few evaluations as possible. Most BO models rely on a GP prior for f ( x ) to achieve prediction and UQ:

where k ( · , · ) is a valid kernel/covariance function and m ( · ) is a mean function that can be generally assumed to be 0. Given a finite set of observation points { x i } n i =1 with x i = ( x i 1 , . . . , x i D ) ⊤ , the vector of function values f = ( f ( x 1 ) , . . . , f ( x n )) ⊤ has a multivariate Gaussian distribution f ∼ N ( 0 , K ) , where K is the n × n covariance matrix. For a set of observed data D n = { x i , y i } n i =1 with i.i.d. Gaussian noise, i.e., y i = f ( x i ) + ϵ i where ϵ i ∼ N (0 , τ -1 ) , GP gives an analytical posterior distribution of f ( x ) at an unobserved point x ∗ :

Based on the posterior distributions of f , one can compute an AF, denoted by α : X → R , for a new candidate x ∗ and evaluate how promising x ∗ is. In BO, the next query point is often determined by maximizing a selected/predefined AF, i.e., x n +1 = arg max x ∈X α ( x | D n ) . Most AFs are based on the predictive mean and variance; for example, a commonly used AF is the expected improvement (EI) [1]:

Algorithm 1: Basic BO process. Input: Initial dataset D 0 and a trained surrogate model; total budget N . for n = 1 , . . . , N do Approximate the posterior distribution of f using the surrogate model based on D n -1 ; Find next evaluation point x n by optimizing the AF; Augment data D n = D n -1 ∪ { x n , y n } , update the surrogate model. ,

where ∆( x ) = µ ( x ) -f ⋆ n is the expected difference between the proposed point x and the current best solution, f ⋆ n = max x ∈{ x i } n i =1 f ( x ) denotes the best function value obtained so far; µ ( x ) and σ ( x ) are the predictive mean and predictive standard deviation at x , respectively; and φ ( · ) and Φ( · ) denote the probability density function (PDF) and the cumulative distribution function (CDF) of standard normal, respectively. Another widely applied AF for maximization problems is the upper confidence bound (UCB) [13]:

where β is a tunable parameter that balances exploration and exploitation. The general BO procedure can be summarized as Algorithm 1.

3 BKTF for Bayesian optimization

3.1 Bayesian hierarchical model specification

Before introducing BKTF, we first construct a D -dimensional grid space corresponding to the search space X , where X d could be continuous, integer-valued, or categorical. We define it on D sets { S 1 , . . . , S D } and denote the whole grid by ∏ D d =1 S d : S 1 × . . . × S D =

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

{ ( s 1 , . . . , s D ) | ∀ d ∈ { 1 , . . . , D } , s d ∈ S d } . For dimensions with integer-valued and categorical input, we consider S d the set of corresponding discrete values. For dimensions with continuous input, the coordinate set S d is formed by m d interpolation points c d i that are distributed over the bounded interval X d = [ a d , b d ] , i.e., S d = { c d i } m d i =1 with c d i ∈ X d . The size of S d becomes | S d | = m d , and size of the entire grid space is ∏ D d =1 | S d | . Note that we do not restrict S d to be uniformly distributed.

We assume the underlying function f as a stochastic process that is zero-centered. We randomly sample an initial dataset including n 0 input-output data pairs D 0 = { x i , y i } n 0 i =1 , where { x i } n 0 i =1 are located in ∏ D d =1 S d , and this yields an incomplete D -dimensional tensor Y ∈ R | S 1 |×···×| S D | with n 0 observed points. BKTF approximates the entire data tensor Y by a kernelized tensor CP decomposition:

where R is a pre-specified tensor CP rank, λ = ( λ 1 , . . . , λ R ) ⊤ denote weight coefficients that capture the magnitude/importance of each rank in the factorization, g r d = [ g r d ( s d ) : s d ∈ S d ] ∈ R | S d | denotes the r th latent factor for the d th dimension, entries in E are i.i.d. white noises following N (0 , τ -1 ) . It should be particularly noted that both the coefficients { λ r } R r =1 and the latent basis functions { g r 1 , . . . , g r D } R r =1 are random variables. The function approximation for x = ( x 1 , . . . , x D ) ⊤ is:

For priors, we assume λ r ∼ N (0 , 1) for r = 1 , . . . , R and use a GP prior on the latent factors for dimension d with continuous input:

d d d d d d d where k r d is a valid kernel function. In this paper, we choose a Matérn 3/2 kernel k r d ( x d , x ′ d ; l r d ) = σ 2 ( 1 + √ 3 | x d -x ′ d | l r d ) exp ( -√ 3 | x d -x ′ d | l r d ) . We fix the kernel variance of k r d as σ 2 = 1 , and only learn the lengthscale hyperparameters l r d , since the variances of the model can be captured by λ . One can also exclude λ but introduce variance σ 2 as a kernel hyperparameter on one of the basis functions; however, learning kernel hyperparameters is computationally more expensive than learning λ . For simplicity, we can also assume the lengthscale parameters to be identical, i.e., l 1 d = . . . = l R d = l d , for each dimension d . The prior distribution for the corresponding latent factor g r d is then a Gaussian distribution: g r d ∼ N ( 0 , K r d ) , where K r d is the | S d | × | S d | covariance matrix computed from k r d . We place Gaussian hyperpriors on the log-transformed kernel hyperparameters to ensure positive values, i.e., log ( l r d ) ∼ N ( µ l , τ -1 l ) . For categorical input, we assume that the corresponding latent basis functions g r d | Λ d ∼ N ( 0 , Λ -1 d ) for r = 1 , . . . , R , where the precision matrix Λ d follows a Wishart prior with an identity scale matrix and | S d | degrees of freedom, i.e., Λ d ∼ W ( I | S d | , | S d | ) . For noise precision τ , we assume a conjugate Gamma prior τ ∼ Gamma ( a 0 , b 0 ) . For dimensions with integer variables, we could model the covariance of the basis functions either using a kernel matrix or with an inverse-Wishart prior, depending on specific situations.

For observations, we assume each y i in the initial dataset D 0 follows a Gaussian distribution:

3.2 BKTF as a two-layer deep GP

Here we show the representation of BKTF as a two-layer deep GP. The first layer characterizes the generation of latent functions { g r d } R r =1 for dimension d . For the second layer, if we consider { g r 1 , . . . , g r D } R r =1 as parameters and rewrite the functional decomposition in Eq. (7) as a linear function f ( x ; { λ r } ) = ∑ R r =1 λ r ∏ D d =1 g r d ( x d ) with λ r iid ∼ N (0 , 1) , we can marginalize { λ r } and obtain a fully symmetric multilinear kernel/covariance function for any two data points x = ( x 1 , . . . , x D ) ⊤ and x ′ = ( x ′ 1 , . . . , x ′ D ) ⊤ :

As can be seen, the second layer has a multilinear product kernel function parameterized by 154 { g r 1 , . . . , g r D } R r =1 . There are some properties to highlight: (i) the kernel is nonstationary since 155 the value of g r d ( · ) is location specific, and (ii) the kernel is nonseparable when R > 1 . Therefore, 156 this specification is very different from traditional GP surrogates, such as: 157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

where all kernel functions are stationary with different hyperparameters (e.g., length scale and variance). Compared to the GP-based kernel specification, the multilinear kernel in Eq. (10) has a much larger set of hyperparameters and becomes more flexible and adaptive for the data. From a GP perspective, learning the hyperparameter in the kernel function in Eq. (10) will be computationally expensive; however, we can achieve efficient Bayesian inference of { λ r , g r 1 , . . . , g r D } R r =1 under a kernelized tensor factorization framework.

3.3 Model inference

Unlike GP, BKTF no longer enjoys an analytical posterior distribution. Based on the aforementioned prior and hyperprior settings, we adapt the MCMC updating procedure in Ref. [10, 11] to an efficient Gibbs sampling algorithm for model inference. This allows us to accommodate observations that are not located in the grid space ∏ D d =1 S d . The detailed derivation of the sampling algorithm is given in Appendix A. In terms of computational cost, we note that updating g r d and kernel hyperparameters requires min { O ( n 3 ) , O ( | S d | 3 ) } in time. Sparse GP (such as [14]) could be introduced when n, | S d | become large. See Appendix A, F for detailed discussion/comparison about computation complexity.

3.4 Prediction and AF computation

In each step of function evaluation, we run the MCMC sampling process K iterations for model inference, where the first K 0 samples are taken as burn-in and the last K -K 0 samples are used for posterior approximation. The predictive distribution for any entry f ∗ in the defined grid space conditioned on the observed dataset D n can be obtained by the Monte Carlo approximation p ( f ∗ | D n , θ 0 ) ≈ 1 K -K 0 × ∑ K k = K 0 +1 p ( f ∗ ∣ ∣ ∣ ( g r d ) ( k ) , λ ( k ) , τ ( k ) ) , where θ 0 = { µ l , τ l , a 0 , b 0 } is the set of all parameters used in hyperpriors. Although a direct analytical predictive distribution does not exist in BKTF, we can use MCMC samples to obtain the mean and variance of all points on the grid, thus offering an enumeration-based approach to define AF.

Wedefine a Bayesian variant of the UCB as the AF by approximating the predictive mean and variance (or uncertainty) in ordinary GP-based UCB with the values calculated from MCMC sampling. For every MCMC sample after burn-in, i.e., k > K 0 , we can estimate an output tensor ˜ F ( k ) over the entire grid space using the latent factors ( g r d ) ( k ) and the weight vector λ ( k ) : ˜ F ( k ) = ∑ R r =1 λ ( k ) r ( g r 1 ) ( k ) ◦ ( g r 2 ) ( k ) ◦ · · · ◦ ( g r D ) ( k ) . We can then compute the corresponding mean and variance tensors of the ( K -K 0 ) samples { ˜ F ( k ) } K k = K 0 +1 , and denote the two tensors by U and V , respectively. The approximated predictive distribution at each point x in the space becomes ˜ f ( x ) ∼ N ( u ( x ) , v ( x )) . Following the definition of UCB in Eq. (5), we define Bayesian UCB (B-UCB) at location x as α B-UCB ( x | D , β, g r d , λ ) = u ( x ) + β √ v ( x ) . The next search/query point can be determined via x next = arg max x ∈{ ∏ D d =1 S d -D n -1 } α B-UCB ( x ) .

We summarize the implementation procedure of BKTF for BO in Appendix B (see Algorithm 2). Given the sequential nature of BO, when a new data point arrives at step n , we can start the MCMC with the last iteration of the Markov chains at step n -1 to accelerate model convergence. The main computational and storage cost of BKTF is to interpolate and save the tensors ˜ F ∈ R | S 1 |×···×| S D | over ( K -K 0 ) iterations for Bayesian AF estimation. This could be prohibitive when the MCMC sample size K or the dimensionality D becomes large. To avoid saving the tensors, in practice, we can simply use the maximum values of each entry over the ( K -K 0 ) iterations through iterative pairwise comparison. The number of samples after burn-in then implies the value of β in α B-UCB. We adopt this simple AF in our numerical experiments.

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

A critical challenge in BKTF is that tensor size grows exponentially with the number of dimensions. To decrease the computational burden of enumeration-based AF, we also implement BKTF with random discretization-randomly selecting candidate points instead of reconstructing the whole space, denoted as BKTFrandom . BKTFrandom can be applied to functions with higher dimensions (e.g., D > 10 ). We discuss the comparison between BKTF and BKTFrandom in Experiments on test functions, see Section 5.1.

4 Related work

The key of BO is to effectively characterize the posterior distribution of the objective function from a limited number of observations. The most relevant work to our study is the Bayesian Kernelized Factorization (BKF) framework, which has been mainly used for modeling large-scale and multidimensional spatiotemporal data with UQ. The key idea is to parameterize the multidimensional stochastic processes using a factorization model, in which specific priors are used to encode spatial and temporal dependencies. Signature examples of BKF include spatial dynamic factor model (SDFM) [15], variational Gaussian process factor analysis (VGFA) [16], and Bayesian kernelized matrix/tensor factorization (BKMF/BKTF) [10, 11, 17]. A common solution in these models is to use GP prior to modeling the factor matrices, thus encoding spatial and temporal dependencies. In addition, for categorical dimensions, BKTF uses an inverse-Wishart prior to modeling the covariance matrix for the latent factors. A key difference among these methods is how inference is performed. SDFM and BKMF/BKTF are fully Bayesian hierarchical models and they rely on MCMC for model inference, where the factors can be updated via Gibbs sampling with conjugate priors; for learning the posterior distributions of kernel hyperparameters, SDFM uses the Metropolis-Hastings sampling, while BKMF/BKTF uses the more efficient slice sampling. On the other hand, VGFA uses variational inference to learn factor matrices, while kernel hyperparameters are learned through maximum a posteriori (MAP) estimation without UQ. Overall, BKTF has shown superior performance in modeling multidimensional spatiotemporal processes with high-quality UQ for 2D/3D spaces [11, 17].

The proposed BKTF surrogate models the objective function-as a single realization of a random process-using low-rank tensor factorization with random basis functions. This basis functionbased specification is closely related to multidimensional Karhunen-Loève (KL) expansion [18] for stochastic (spatial, temporal, and spatiotemporal) processes. Empirical analysis of KL expansion is also known as proper orthogonal decomposition (POD). With a known kernel/covariance function, truncated KL expansion allows us to approximate the underlying random process using a set of eigenvalues and eigenfunctions derived from the kernel function. Numerical KL expansion is often referred to as the Garlekin method, and in practice, the basis functions are often chosen as prespecified and deterministic functions [12, 19], such as Fourier basis, wavelet basis, orthogonal polynomials, B-splines, empirical orthogonal functions, radial basis functions (RBF), and Wendland functions (i.e., compactly supported RBF) (see, e.g., [20], [21], [22], [23]). However, the quality of UQ will be undermined as the randomness is fully attributed to the coefficients { λ r } ; in addition, these methods also require a large number of basis functions (or a large rank) to fit complex stochastic processes. Different from methods with fixed/known basis functions, BKTF uses a Bayesian hierarchical modeling framework to better capture the randomness and uncertainty in the data, in which GP priors are used to model the latent factors (i.e., basis functions are also random processes) on different dimensions, and hyperpriors are introduced on the kernel hyperparameters. Therefore, BKTF becomes a fully Bayesian version of multidimensional KL expansion for stochastic processes with unknown covariance from partially observed data, however, without imposing any orthogonal constraint on the basis functions. Following the analysis in section 3.2, BKTF is also a special case of a two-layer deep Gaussian process [24, 7], where the first layer produces latent factors for each dimension, and the second layer has a multilinear kernel parameterized by all latent factors.

5 Experiments

5.1 Optimization for benchmark test functions

We test the proposed BKTF model for BO on seven benchmark functions that are used for global optimization problems [25], with dimension D ranging from 2 to 10. All selected standard functions are multimodal; detailed descriptions are given in Appendix D. In fact, we can visually see that the standard Damavandi/Schaffer/Griewank functions in Figure 7 (see Appendix D) indeed have a

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

low-rank structure. For each function, we assume the initial dataset D 0 contains n 0 = D observed data pairs, and we set the total number of query points to N = 80 for 4D Griewank and 6D Hartmann function, N = 200 for 10D Griewank, and N = 50 for others. We rescale the input search range to [0 , 1] for all dimensions and normalize the output data using z-score normalization.

Model configuration When applying BKTF to continuous test functions, we introduce m d interpolation points for the d th dimension of the input space. The values of m d used for each benchmark function are predefined and given in Table 1 (see Appendix D). Setting the resolution grid will require certain prior knowledge (e.g., smoothness of the function); and it also depends on the available computational resources and the number of entries in the tensor, which grows exponentially with m d . In practice, we find that setting m d = 10 ∼ 100 is sufficient for most problems. We set the CP rank R = 2 , and for each BO function evaluation run 400 MCMC iterations for model inference where the first 200 iterations are taken as burn-in. We use Matérn 3/2 kernel as the covariance function for all the test functions.

Note that for the 10D Griewank function, the grid-based models do not work, and only models built in continuous space and BKTFrandom can be performed. For BKTFrandom, in each evaluation we randomly select 20k points in the search space as candidates and choose the one with the best AF as the next evaluation location.

Baselines We compare BKTF with the following BO methods that use GP as the surrogate model: (1) GP α EI and (2) GPgrid α EI: GP as the surrogate model and EI as the AF, in continuous space ∏ D d =1 X d and Cartesian grid space ∏ D d =1 S d , respectively; (3) GP α UCB and (4) GPgrid α UCB: GP as the surrogate model with UCB as the AF where β = 2 , in ∏ D d =1 X d and ∏ D d =1 S d , respectively; (5) additive GP: the sum of two 1st-order additive kernels per dimension as the surrogate with EI as the AF, in continuous space. This baseline has the same number of latent functions as BKTF ( R = 2 ) but in a sum-based manner; (6) deepGP: a two-layer fully-Bayesian deep GP with EI as the AF, implemented with the deepgp package 1 .

Similar as the setting for BKTF, we use Matérn 3/2 kernel in all GP models. Given the computational cost, we only compare deepGP on 2D functions [9]. For AF optimization in GP α EI and GP α UCB, we first use the DIRECT algorithm [26] and then apply the Nelder-Mead algorithm [27] to further search if there exist better solutions. We also compare with SAASBO (Sparse Axis-Aligned Subspace) [28] with Hamiltonian Monte Carlo sampling, implemented using BoTorch [29], on the 6D Hartmann and 10D Griewank functions.

Results To compare the optimization performance of different models on the benchmark functions, we define the absolute error between the global optimum f ⋆ and the current estimated global optimum ˆ f ⋆ , i.e., ∣ ∣ ∣ f ⋆ -ˆ f ⋆ ∣ ∣ ∣ , as the regret , and examine how regret varies with the number of function

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

evaluations. We run the optimization 10 times for every test function with a different set of initial observations. The results are summarized in Figure 2. We see that for the 2D functions Branin and Schaffer, BKTF clearly finds the global optima much faster than GP surrogate-based baselines. For Damavandi function, where the global minimum ( f ( x ⋆ ) = 0 ) is located in a small sharp area while the local optimum ( f ( x ) = 2 ) is located at a large smooth area (see Figure 7 in Appendix D), GP-based models are trapped around the local optima in most cases ( i.e., regret = 2 ) and cannot jump out. In contrast, BKTF explores the global characteristics of the objective function over the entire search space and reaches the global optimum within 10 iterations of function evaluations. For higher dimensional Griewank and Hartmann functions, BKTF successfully arrives at the global optima under the given observation budgets, while GP-based comparison methods are prone to be stuck around local optima. We compare the regret at the last iteration in Table 2 (Appendix E.2), along with the average cost of evaluations. The enumeration-based GP surrogates, i.e., GPgrid α EI and GPgrid α UCB, perform a little better than direct GP-based search, i.e., GP α EI and GP α UCB on Damavandi function, but worse on others. This means that the discretization, to some extent, offers possibilities for searching all the alternative points in the space, since in each function evaluation, every sample in the space is equally compared solely based on the predictive distribution. Additive GP is comparable with R = 2 BKTF; while the results demonstrate that BKTF can be much more flexible than additive GP. As for BKTFrandom, we see that it can alleviate the curse of dimensionality and be applied for higher-dimensional problems that may not be performed with a grid but at the cost of more evaluation budgets, particularly it costs more iterations for lower-dimensional functions compared with BKTF.

Overall, BKTF reaches the global optimum for every test function and shows superior performance for complex objective functions with a faster convergence rate. To intuitively compare the overall performance of different models across multiple experiments/functions, we further estimate performance profiles (PPs) [30] (see Appendix E.1), and compute the area under the curve (AUC) for quantitative analysis (see bottom right of Figure 2 and Table 2 in Appendix E.2). Our results show that BKTF obtains the best performance across all functions.

Interpretable latent space. The sampled latent functions are interpretable and imply the underlying correlations of the objective function. We illustrate the learned periodic (global) patterns in Appendix E.3. Effects of hyperpriors. Since we build a fully Bayesian model, the hyperparameters of the covariance functions can be updated automatically from the data likelihood and hyperprior. Note that in optimization scenarios where the observations are scarce, the prediction performance of BKTF highly depends on the hyperprior settings, i.e., θ 0 = { µ l , τ l , a 0 , b 0 } . We discuss the effects of hyperpriors in Appendix E.4. Effects of rank. The only predefined model parameter is the model rank, all other model parameters and hyperparameters are sampled with MCMC. We discuss the effects of rank on the 2D nonstationary nonseparable function defined in Introduction (see Figure 1) in Appendix E.5. We see that BKTF is robust to rank specification and can find the global solution efficiently with rank set as 2, 4, 6 and 8.

5.2 Hyperparameter tuning for machine learning

In this section, we evaluate the performance of BKTF for automatic machine learning (ML) tasks. We compare different models to optimize the hyperparameters of two ML algorithms-random forest (RF) and neural network (NN)-on classification for the MNIST database of handwritten digits 2 and regression for the Boston housing dataset 3 . The tuning tasks involve both integer-valued and categorical parameters, and the details are given in the Appendix G. We treat those integer-valued dimensions as continuous and use a Matérn 3/2 kernel for the basis functions. Given the size of the hyperparameter space, we perform BKTFrandom for classification and BKTF for regression. We assume that the number of initial observations D 0 equals the number of tuning hyperparameters. The total budget N is 20 for the classification task and 50 for the regression. We implement RF algorithms using the scikit-learn package and construct NN models by Keras with 2 hidden layers. All other model hyperparameters are set as the default values.

Model configuration We treat all the integer hyperparameters as samples from a continuous space and then generate the corresponding Cartesian product space ∏ D d =1 S d . One can interpret

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

the candidate values for each hyperparameter as the interpolation points in the corresponding input dimension. The size of the spanned space ∏ S d is 91 × 46 × 64 × 10 × 11 × 2 and 91 × 46 × 13 × 10 for RF classifier and RF regressor, respectively; 91 × 49 × 31 × 18 × 3 × 2 and 91 × 49 × 31 for NN classifier and NN regressor respectively. We set the tensor rank R = 4 for BKTF, set K = 400 and K 0 = 200 for MCMC inference, and use the Matérn 3/2 kernel for capturing correlations.

Baselines In addition to GP surrogate-based GP α EI and GP α UCB, we also compare with a baseline method: random search (RS), and two non-GP models: particle swarm optimization (PSO) [31] and Tree-structured Parzen Estimator (BO-TPE) [32], which are common methods for hyperparameter tuning. We exclude grid-based GP models as sampling the entire grid becomes infeasible.

Results We compare the accuracy for MNIST classification and MSE (mean squared error) for Boston housing regression both in terms of the number of function evaluations and still run the optimization processes ten times with different initial datasets D 0 . The results obtained by different BO models are given in Figure 3, and the final classification accuracy and regression MSE are compared in Table 5 (see Appendix H). For BKTF, we see from Figure 3 that the width between the two quartiles of accuracy and error decreases as more iterations are evaluated, and the median curves present better convergence rates compared to the baselines. Table 5 also shows that the proposed BKTF surrogate achieves the best final mean accuracy and regression error with small standard deviations. The results above demonstrate the advantage of BKTF as a surrogate.

6 Conclusion

This paper proposes to use Bayesian Kernelized Tensor Factorization (BKTF) as a new surrogate model for Bayesian optimization with mixed variables (both discrete/categorical and continuous) when the dimensionality is relatively small (e.g., say D < 10 ). Compared with traditional GP surrogates, the BKTF surrogate is more flexible and adaptive to data thanks to the Bayesian hierarchical specification, which provides high-quality UQ for BO tasks. The tensor factorization model behind BKTF offers an effective solution to capture global/long-range correlations and cross-dimension correlations. The inference of BKTF is achieved through MCMC, which provides a natural solution for acquisition. Experiments on both test function optimization and ML hyperparameter tuning confirm the superiority of BKTF as a surrogate for BO. Moreover, the tensor factorization framework makes it straightforward to adapt BKTF to handle multivariate and functional output (see e.g., [33, 34]), by directly treating the output coordinates as part of the input. A limitation of BKTF is that we restrict BO to a grid search space in order to leverage tensor factorization; however, we believe that designing a compatible grid space based on prior knowledge is not a challenging task.

There are several directions to be explored to make BKTF more scalable. Scalable GP solutions, such as sparse GP [14] and Gaussian Markov Random Field (GMRF) [35], can be introduced to reduce the inference cost when | S d | becomes large. The current MCMC-based acquisition method requires explicit reconstruction of the whole tensor, which quickly becomes infeasible when D becomes large (e.g., D > 20 ). A natural question is whether it is possible to achieve efficient acquisition directly using the basis functions and the corresponding weights without explicitly constructing the tensors. This problem corresponds to finding/locating the maximum entry of a tensor given its low-rank decomposition (see e.g., [36]).

This work aims to advance the field of probabilistic machine learning, particularly Bayesian opti379 mization. Regardless of the model limitations, it has the potential of misuse for ML algorithms. 380

381

382

References

Roman Garnett. Bayesian Optimization . Cambridge University Press, 2023.

Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking 383 the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE , 384 104(1):148-175, 2015. 385

386

387

Robert B Gramacy. Surrogates: Gaussian Process Modeling, Design, and Optimization for the Applied Sciences . Chapman and Hall/CRC, 2020.

388

389

390

391

392

393

394

395

396

Christopher KI Williams and Carl Edward Rasmussen. Gaussian Processes for Machine Learning . MIT Press, Cambridge, MA, 2006.

David K Duvenaud, Hannes Nickisch, and Carl Rasmussen. Additive Gaussian processes. Advances in neural information processing systems , 24, 2011.

Mickael Binois and Nathan Wycoff. A survey on high-dimensional Gaussian process modeling with application to Bayesian optimization. ACM Transactions on Evolutionary Learning and Optimization , 2(2):1-26, 2022.

Andreas Damianou and Neil D Lawrence. Deep Gaussian processes. In International Conference on Artificial Intelligence and Statistics , pages 207-215, 2013.

Ali Hebbal, Loïc Brevault, Mathieu Balesdent, El-Ghazali Talbi, and Nouredine Melab. 397 Bayesian optimization using deep gaussian processes with applications to aerospace system 398 design. Optimization and Engineering , 22:321-361, 2021. 399

Annie Sauer, Robert B Gramacy, and David Higdon. Active learning for deep gaussian process 400 surrogates. Technometrics , 65(1):4-18, 2023. 401

Mengying Lei, Aurelie Labbe, Yuankai Wu, and Lijun Sun. Bayesian kernelized matrix 402 factorization for spatiotemporal traffic data imputation and kriging. IEEE Transactions on 403 Intelligent Transportation Systems , 23(10):18962-18974, 2022. 404

Mengying Lei, Aurelie Labbe, and Lijun Sun. Bayesian complementary kernelized learning for 405 multidimensional spatiotemporal data. arXiv preprint arXiv:2208.09978 , 2022. 406

Noel Cressie, Matthew Sainsbury-Dale, and Andrew Zammit-Mangion. Basis-function models 407 in spatial statistics. Annual Review of Statistics and Its Application , 9:373-400, 2022. 408

409

410

Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research , 3(Nov):397-422, 2002.

Joaquin Quinonero-Candela and Carl Edward Rasmussen. A unifying view of sparse approxi411 mate Gaussian process regression. The Journal of Machine Learning Research , 6:1939-1959, 412 2005. 413

Hedibert Freitas Lopes, Esther Salazar, and Dani Gamerman. Spatial dynamic factor analysis. 414 Bayesian Analysis , 3(4):759-792, 2008. 415

Jaakko Luttinen and Alexander Ilin. Variational Gaussian-process factor analysis for modeling 416 spatio-temporal data. Advances in Neural Information Processing Systems 417

, 22:1177-1185, 2009. 418

Mengying Lei, Aurelie Labbe, and Lijun Sun. Scalable spatiotemporally varying coefficient 419 modeling with bayesian kernelized tensor regression. arXiv preprint arXiv:2109.00046 , 2021. 420

Limin Wang. Karhunen-Loeve expansions and their applications . London School of Economics 421 and Political Science (United Kingdom), 2008. 422

Holger Wendland. Scattered Data Approximation , volume 17. Cambridge university press, 423 2004. 424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

Rommel G Regis and Christine A Shoemaker. A stochastic radial basis function method for the global optimization of expensive functions. INFORMS Journal on Computing , 19(4):497-509, 2007.

Gregory Beylkin, Jochen Garcke, and Martin J Mohlenkamp. Multivariate regression and machine learning with sums of separable functions. SIAM Journal on Scientific Computing , 31(3):1840-1857, 2009.

Christopher K Wikle and Noel Cressie. A dimension-reduced approach to space-time kalman filtering. Biometrika , 86(4):815-829, 1999.

Mathilde Chevreuil, Régis Lebrun, Anthony Nouy, and Prashant Rai. A least-squares method for sparse low rank approximation of multivariate functions. SIAM/ASA Journal on Uncertainty Quantification , 3(1):897-921, 2015.

Alexandra M Schmidt and Anthony O'Hagan. Bayesian inference for non-stationary spatial covariance structure via spatial deformations. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 65(3):743-758, 2003.

Momin Jamil and Xin-She Yang. A literature survey of benchmark functions for global optimization problems. arXiv preprint arXiv:1308.4008 , 2013.

Donald R Jones, Cary D Perttunen, and Bruce E Stuckman. Lipschitzian optimization without the lipschitz constant. Journal of optimization Theory and Applications , 79(1):157-181, 1993.

John A Nelder and Roger Mead. A simplex method for function minimization. The computer journal , 7(4):308-313, 1965.

David Eriksson and Martin Jankowiak. High-dimensional bayesian optimization with sparse axis-aligned subspaces. In Uncertainty in Artificial Intelligence , pages 493-503. PMLR, 2021.

Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew G Wilson, and Eytan Bakshy. Botorch: A framework for efficient monte-carlo bayesian optimization. Advances in neural information processing systems , 33:21524-21538, 2020.

Elizabeth D Dolan and Jorge J Moré. Benchmarking optimization software with performance profiles. Mathematical programming , 91(2):201-213, 2002.

Jun Tang, Gang Liu, and Qingtao Pan. A review on representative swarm intelligence algorithms for solving optimization problems: Applications and trends. IEEE/CAA Journal of Automatica Sinica , 8(10):1627-1643, 2021.

James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyperparameter optimization. Advances in neural information processing systems , 24, 2011.

Dave Higdon, James Gattiker, Brian Williams, and Maria Rightley. Computer model calibration using high-dimensional output. Journal of the American Statistical Association , 103(482):570583, 2008.

Shandian Zhe, Wei Xing, and Robert M Kirby. Scalable high-order gaussian process regression. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 2611-2620. PMLR, 2019.

Havard Rue and Leonhard Held. Gaussian Markov random fields: theory and applications . Chapman and Hall/CRC, 2005.

Anastasiia Batsheva, Andrei Chertkov, Gleb Ryzhakov, and Ivan Oseledets. Protes: probabilistic 465 optimization with tensor sampling. Advances in Neural Information Processing Systems , 36:808466 823, 2023. 467

Appendix 468

Contents (Appendix) 469

̸

g r d is given by 500

where 501

502

̸

̸

̸

Y r ( d ) and O ( d ) are moded unfoldings of Y r and O , respectively, with the size of | S d | × ( ∏ D h =1 h = d | S h | ) . Note that the vector term a in [ µ r d ] ∗ and b in [ Λ r d ] ∗ , which are only relevant to the n observations and corresponding function values, can be computed element-wise instead of using matrix multiplication and Kronecker product. The point-wise computation can dramatically reduce the computational cost, especially for a relatively large D , since in such case the number of observations in BO can be much smaller compared to the number of samples in the entire grid space, i.e., n ≪ ∏ D d =1 | S d | .

A.2 Sampling kernel hyperparameters

We update the kernel lengthscale hyperparameters l r d from their marginal posteriors by integrating out the latent factors. Let [ y r ] d = O d vec ( Y r ( d ) ) ∈ R n , where O d ∈ R n × ( ∏ D d =1 | S d | ) is a binary matrix obtained by removing the rows corresponding to zeros in vec ( O ( d ) ) from I ∏ D d =1 | S d | . When sampling the posteriors for kernel hyperparameters under a given d and r , their marginal likelihoods only relate to [ y r ] d . The log marginal likelihood of l r d , for example, is:

̸

term c in Eq. (14) is a scalar that can be computed with u ⊤ u , where u = ( L r d ) -1 a is a vector of length | S d | ; L r d = chol ( [ Λ r d ] ∗ ) is the Cholesky factor matrix of [ Λ r d ] ∗ . This means that the complicated term c can also be calculated element-wise, and it leads to a fast learning process. With the marginal likelihood and predefined log-normal hyperpriors, we can get the marginal posteriors of the kernel hyperparameters straightforwardly; and we update them by using the slice sampling algorithm presented in [17].

A.3 Sampling Λ d for latent functions on categorical inputs

For latent factors in dimensions with categorical variables/inputs, we sample the precision hyperparameter Λ d in its prior distribution from a Wishart distribution:

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

A.4 Sampling weight vector 526

Every observed data point has the following distribution: 527

Let g ( x i ) = ( ∏ D d =1 g 1 d ( x i d ) , . . . , ∏ D d =1 g R d ( x i d ) ) ⊤ ∈ R R and y = ( y 1 , . . . , y n ) ⊤ ∈ R n ; we then 528 have y ∼ N ( ˜ G ⊤ λ , τ -1 I n ) , where ˜ G = [ g ( x 1 ) , . . . , g ( x n )] ∈ R R × n . The posterior of λ follows 529 a Gaussian distribution 530

where 531

532

A.5 Sampling model noise precision 533

For precision τ , we have a Gamma posterior 534

where 535

536

B Algorithm of BKTF for BO 537

C Optimization for the 2D nonstationary and nonseparable function 539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

C.1 Data generation

The function in Figure 1(a) of the paper (see Section 1 Introduction) is a modification of the case study used in [11]. It is a 40 × 40 2D process generated in a [1 , 2] × [ -1 , 0] square, with

where f ( x 1 ) = x 1 (sin 2 x 1 +2) , f ( x 2 ) = 0 . 2 x 2 √ 99( x 2 +1) + 4 , x 1 ∈ [1 , 2] , x 2 ∈ [ -1 , 0] . This is a nonstationary, nonseparable, and multimodel function, with the global maximum f ( x ⋆ ) = 0 . 6028 at x ⋆ = (1 . 75 , -0 . 55) .

C.2 Experimental setting

We randomly select n 0 = 30 data points as the initial data and run 50 iterations (i.e., budget) of evaluation for optimization. To compare different surrogates, we run the optimization for 20 replications with different initial datasets. For the proposed BKTF surrogate, we place Matérn 3/2 kernel functions on the latent factors, set the rank R = 4 , and run 1000 MCMC samples for model inference where the first 600 samples are burn-in. For comparison of BO methods, we consider typical GP surrogate with both EI and UCB ( β = 2 ) as the AF, denoted by GP α EI and GP α UCB respectively, and use the same Matérn 3/2 kernel for GP surrogate.

C.3 Results

Figure 1(b) in Section 1 Introduction of the paper shows the medians along with the 25% and 75% quantiles of the optimization results from 20 runs. We see that GP α EI and GP α UCB cannot find the global optimum in most cases, and they easily get stuck in the lower left flat area which contains easily find local optima. Figure 1(c) illustrates the estimation surface of the function and the estimated AF surface from one run. It is clear that BKTF can capture global correlations with limited data. The

search points contain areas of almost every peak in the true function, and the peaks of its AF surface 560 also reflect the peak area in f . 561

Samples of the latent factors in panels (a) and (b) of Figure 6 explain the uncertainty learned by 564

(a) MCMC samples for latent factors (

d

= 1

).

(b) MCMC samples for latent factors (

d

= 2

).

D Benchmark test functions 566

We summarize the characteristics of the benchmark functions tested in Table 1. Figure 7 shows 567 the functions with 2-dimensional inputs together with the 2D Griewank function. The functional 568 expressions and more detailed features of these test functions are provided in the following. 569

Schaffer

Griewank

globaloptimum

150

0

THOHE

200

100

0.5

1

0

50

比

-2

15

10

-5

C1

0

5

10

0

5

C2

10

0

0

5

10

14

5

-50

0

0

50-50

0

50

-10

0

10-10

0

(1) Branin function ( D = 2)

where x 1 ∈ [ -5 , 10] and x 2 ∈ [0 , 15] . It is a smooth but multimodal function with global minima 570 f ( x ∗ ) = 0 . 3978873 at three input points x ∗ = ( -π, 12 . 275) , ( π, 2 . 275) , (3 π, 2 . 425) . 571

(2) Damavandi function ( D = 2)

where x d ∈ [0 , 14] . This is a multimodal function with the global minimum f ( x ∗ ) = 0 at x ∗ = (2 , 2) . 572

(3) Schaffer function ( D = 2)

where x d ∈ [ -10 , 10] . The global minimum value is f ( x ∗ ) = 0 at x ∗ = (0 , 0) . One characteristic 573 of this function is that the global minimum is located very close to the local minima. 574

(4) Griewank function ( D = 3 , 4 , 10)

where x d ∈ [ -10 , 10] . This is a multimodal function with global minimum f ( x ∗ ) = 0 at x ∗ = (0 , 0) . 575

(5) Hartmann function ( D = 6) A nonseparable function with multidimensional inputs and 576 multiple local minima. 577

where 578

579

580

x d ∈ [0 , 1] . The 6-dimensional case has 6 local minima, and the global minimum is f ( x ∗ ) = -3 . 32237 at x ∗ = (0 . 20169 , 0 . 150011 , 0 . 476874 , 0 . 275332 , 0 . 311652 , 0 . 657301) .

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

Note that all these minimization problems can be easily transformed as a maximization optimization problem, i.e., max -f ( x ) .

E Supplementary results on benchmark test functions

E.1 Performance profiles

When computing the performance profiles (PPs), i.e., Dolan-Moré curves [30], we consider the number of function evaluations to find the global optimum as the performance measure. Specifically, let t p,a denote the number of evaluations used by method/solver a to reach the global solution in experiment p (a lower value is better). The value is equal to N p +100 if the method cannot find the global optimum with N p being the given observation budget for experiment p . The performance ratio

where A represents the set that includes all comparing models, and the performance profile for each method is the distribution of p ( γ p,a ≤ ρ ) in terms of a factor ρ . We set ρ = 1 : N max +1 , where N max = max N p is the largest observation budget assumed for the compared experiments. We define the problem set {P | ∀ p ∈ P} as the 10 runs for every function and draw the performance profiles of each model, also set P as the 70 experiments in the 7 tested functions and estimate the overall performance profiles.

We show the obtained PPs across test functions (i.e., overall PPs) in Figure 2 (bottom right) for illustration (see Section 5.1 of the paper); the results of all the tested functions are shown below in Figure 8. Note we only compute PPs for the methods that compared on all test functions, i.e., deepGP and SAASBO are not considered.

Table 2 gives the AUC (area under the curve) values of these curves, where the AUC of the overall performance profiles is taken as the metric to compare the overall performances. As can be seen, BKTF obtains the best performance across all the considered functions. In addition, for most of the test functions, the AUC of grid-based baseline models is comparable with those of continuous GP-based models, suggesting that discretization of the continuous space is feasible to simplify the optimization problem.

E.2 Quantitative comparison 606

We compare the last step regret , average costs of evaluations, and AUC of PPs of different methods 607 on benchmark test functions in Table 2. 608

609

610

Best results are highlighted in bold fonts. B: Branin; D: Damavandi; S: Schaffer; G: Griewank; H: Hartmann.

E.3 Interpretation of results

The basis functions learned by BKTF are interpretable. For example, Figure 9 shows the latent factors

( r = 1 ) obtained at the last iteration of function evaluation in one run on 3D Griewank function. 611

We see that BKTF can learn the periodicity (global structure) of the function benefited from the 612

613

low-rank modeling. On the other hand, a stationary and separable GP cannot, other than using a specific kernel function such as the periodic kernel, which however requires strong prior knowledge 614

to set the periodicity kernel hyperparameter. 615

616

E.4 Effects of hyperpriors

We compare the optimization performance on Branin function with different hyperprior settings in 617 Figure 10 as an example to illustrate the effects of hyperpriors. Specifically, we compare the optimiza618 tion results under several hyperprior assumptions of µ l , when τ -1 l is set as 0.5. As can be seen, BKTF 619 is not able to reach the global minimum with too small or too large mean assumptions (comparable to 620 [0 , 1] ) on the kernel lengthscales l , for example in the cases where µ l = { log (0 . 05) , log (2) } . In con621 trast, it finds the global optimum after 4 iterations of function evaluations when µ l = log (0 . 5) , see the 622 purple line. These imply the importance of hyperprior selection. The reason is that in the first several 623 evaluations, since the observations are rare, the prior basically determines the exploration-exploitation 624 balance and guides the search process. 625

Figure 11 shows the approximated posterior distributions for kernel hyperparameters and model noise 626 variance when τ -1 l = 0 . 5 , µ l = log (0 . 5) . We see that for the re-scaled input space and normalized 627 function output, the sampled length scales are around half of the input domain. Such settings are 628 reasonable to capture the correlations between the observations and are also interpretable. 629

The effects of hyper-priors on other functions are similar, and we choose an appropriate setting 630 relevant to the input range. The hyper-prior on τ impacts the uncertainty of the latent factors, for 631 example a large model noise assumption allows more variances in the factors. The role of { a 0 , b 0 } 632 becomes more important when the objective function is complex that BKTF cannot well describe the 633 function with limited observations. Generally, we select the priors that make the noise variances not 634 quite large, such as the results of τ -1 shown in Figure 4 and Figure 11. An example of the uncertainty 635 provided by BKTF is explained in Appendix C (see Figure 6). 636

637

638

639

640

641

642

643

644

645

646

647

648

649

E.5 Effects of rank

Under a fully Bayesian treatment, kernel hyperparameters of BKTF are automatically sampled by MCMCwith proper priors; the only selected parameter is the model rank. We test the effects of rank specification for the proposed BKTF surrogate on the 2D nonstationary nonseparable function defined in Section 1 Introduction (see Figure 1 and Appendix C). We use the same experiment settings as in Appendix C, i.e., 30 initial observations and 50 budget.

The results are given in Figure 12 and 13, where we compare the optimization performance of BKTF with rank R = { 2 , 4 , 6 , 8 } and two GP-based surrogate models: GP α EI and GP α UCB. To clearly illustrate the results, we only show the comparison on one run. In Figure 12, we compare the regret from different models, and in Figure 13 we compute and compare the mean CRPS (continuous ranked probability score) on the unobserved points.

CRPS is a widely applied metric for evaluating the performance of UQ for probabilistic models. With Gaussian likelihoods, CRPS can be defined as:

where n ′ is the number of unknown points in the defined space, i.e., n ′ = ∏ D d =1 m d -n , ˆ y i and 650 σ i are the approximated posterior mean and std. for the i th data point, respectively, f i denotes the 651 true value for the i th point, and ψ ( · ) and Φ( · ) are the PDF (probability density function) and CDF 652 (cumulative distribution function) of standard normal, respectively. 653

We see that BKTF successfully finds the global optimum with rank from 2 to 8, and obtains better 654 (lower) CRPS values than GP baseline surrogates during the search processes. These results indicate 655 that the proposed fully Bayesian framework is robust to the rank setting and can avoid overfitting. 656

657

658

659

660

661

F Comparison of computational complexity

We compare the computational complexity of BKTF, BKTFrandom and the baseline methods applied on test functions in Table 3, where n is the number of observations, m d is the number of interpolation points for the d th dimension, and n random denotes the number of candidates we selected in BKTFrandom, which is 20k in the conducted experiments.

As can be seen, theoretically the proposed model BKTF/BKTFrandom has the lowest computational 662 complexity. 663

G Hyperparameter tuning for machine learning 664

Table 4 lists all the hyperparameters in the tuning tasks. 665

666

H Supplementary results on ML hyperparameter tuning

We summarize the final accuracy obtained with different BO methods for MNIST classification and 667 the final MSE for the regression task in Table 5. 668

669

670

671

672

673

674

675

676

677

678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Please see Abstract and Section Introduction for more details.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations in Section 6 Conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

Justification: We illustrate methodology and technical details of the proposed model in Section 3 BKTF for Bayesian optimization and Appendix A Model inference.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We illustrate detailed information for experiment implementation in Section 5 Experiments and Appendices C-G.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the code for the 2D test function in supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specify the experimental setting and implementation details in Section 5 Experiments and Appendices C-G.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We repeat the experiments certain times and report the mean with std. results in Figure 1, 2, 3, and Table 2, 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All the experiments can be performed with a 16-core 2.40 GHz CPU and 32 GB RAM.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in this paper conform with the NeurIPS code of ethics. Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: This paper presents work whose goal is to advance the field of probabilistic Machine Learning, particularly Bayesian Optimization. We discussed such impacts in Section 6 Conclusion.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: This work has the potential of misuse for machine learning algorithms. However the current model has certain limitations on applying for high-dimensional problems, thus such risks are low. We mentioned such risks in the last paragraph in Section 6 Conclusion.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We include the URLs for the datasets we used in this work.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We submit partial of the code in supplementary material and select a license when submitting the paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

981 982 983 984 985

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes using Bayesian Kernelized Tensor Factorization (BKTF) as a new surrogate model for Bayesian optimization (BO). BKTF approximates the objective function using a low-rank tensor factorization, with Gaussian process priors placed on the latent factors to capture dependencies and enable uncertainty quantification. The key advantages are the ability to handle complex functions that are non-stationary and non-separable, and the sharing of information across dimensions to enable a more global search compared to standard GP models with local kernels. Inference is performed via MCMC sampling. Experiments on benchmark optimization functions and hyperparameter tuning tasks demonstrate improved performance over GP-based BO, especially when the initial sample size and evaluation budget are limited.\n\nSTRENGTHS:\nThe BKTF surrogate is a novel and creative approach to extend BO to handle more complex objective functions. Modeling the objective as a tensor factorization with GP priors on the factors is an elegant way to introduce non-stationarity and multi-scale correlations in a principled Bayesian framework.\n\nThe method is grounded in a clear mathematical framework, with full details of the model specification and MCMC-based inference procedure provided. Positioning BKTF as a type of deep GP offers useful insight into its expressive power.\n\nThe paper includes extensive experiments on a range of synthetic functions and real-world hyperparameter tuning tasks. The results convincingly demonstrate the advantages of BKTF over standard GP-BO in terms of optimization performance and sample efficiency, especially in the realistic setting of a very limited evaluation budget.\n\nThe paper is clearly written, with the methodology explained in detail and the experimental setup and results presented thoroughly. The authors also discuss the limitations of their work, including the scalability challenges and the restriction to a grid-based search space.\n\nWEAKNESSES:\nThe main weakness is that the proposed BKTF method is only compared against basic GP-based BO with standard kernels. To fully demonstrate the advantage of the BKTF surrogate, comparisons should be made to more advanced GP models such as deep kernel learning, deep GPs, and other scalable GP variants. Without these comparisons, it's difficult to assess how much of the performance gain is due to the specific BKTF approach vs. simply being a more flexible GP.\n\nThe BKTF model bears significant similarities to existing works on scalable GPs, such as \"Gaussian Processes for Big Data\" and \"Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)\", which also use inducing points on grids to obtain tractable approximations. The relationship of BKTF to these methods is not discussed, and it's unclear whether BKTF provides any substantial advantages over these existing scalable GP approaches.\n\nThe experiments on the synthetic test functions are somewhat limited in their dimensionality (only up to 10d). Given that BO is most useful for optimizing high-dimensional black-box functions, testing on some higher-dimensional benchmarks would be valuable. The scalability of BKTF as the dimensionality and grid size increase is not fully investigated.\n\nThe MCMC inference procedure may become prohibitively slow for high-dimensional spaces or large evaluation budgets. The paper does not report the wall-clock time of the experiments, which makes it hard to assess the computational feasibility of BKTF in practice, especially compared to alternative approaches.\n\nFor the hyperparameter tuning experiments in Section 5.2, the strong performance of BKTF with very few iterations seems counterintuitive and is not fully explained, since the BKTF model has many parameters and would be expected to require a substantial amount of data to train effectively. This seems to contradict the results on the synthetic test functions, where GP-EI performs equally well in the first few iterations.\n\nQUESTIONS:\nHow does BKTF differ from existing scalable GP methods like \"Gaussian Processes for Big Data\" and \"Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)\", which also exploit grid structure and inducing points? Does BKTF offer any advantages over these approaches in terms of performance or flexibility?\n\nBeyond standard GPs, have you considered comparing BKTF to other more advanced probabilistic regression models such as neural processes, Bayesian neural networks, or tree-based models like BART or XGBoost? These would seem to be the most relevant competitors for black-box optimization.\n\nCould you report the running time of the BKTF method, especially the cost of the MCMC inference as the dimensionality and evaluation budget grow? This is critical for assessing the practical feasibility of the approach.\n\nHave you investigated the robustness of BKTF to the choice of grid size and spacing? Does performance degrade gracefully if the grid is misspecified or can this lead to severe failure modes?\n\nThe strong performance of BKTF on the hyperparameter tuning tasks, especially with very few iterations, is quite surprising and seems to conflict with the synthetic results. Could you provide some intuition or explanation for this? Is there something fundamentally different about these tasks that enables BKTF to perform well with little data, or is it more of an artifact of the experimental setup?\n\nLIMITATIONS:\nThe relationship of BKTF to existing scalable GP methods is not thoroughly discussed. The paper does not make clear how BKTF differs from or improves upon approaches like \"Gaussian Processes for Big Data\" and \"Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)\", which also exploit grid structure. A more thorough comparison is needed to justify BKTF as a novel contribution.\n\nThe cubic scaling of the covariance matrix computations in the number of grid points, which could make BKTF infeasible for high-dimensional or very fine-grained grids. Some discussion of potential ways to scale up BKTF, e.g., by exploiting grid structure or using sparse approximations, would be valuable.\n\nThe fact that BKTF relies heavily on a sensible grid specification, and may fail badly if the grid is poorly chosen. Some experiments showing the sensitivity of the results to the grid choice would help assess this risk.\n\nThe lack of comparison to a wider range of flexible surrogate models beyond standard GPs, including more sophisticated GP models as well as other probabilistic regression approaches. The current experiments are not sufficient to establish BKTF as the best choice for BO in practice.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the Bayesian Kernelized Tensor Factorization (BKTF) as a surrogate for Bayesian optimization (BO). This model uses a CP decomposition to define a set of random basis functions drawn from a GP prior. These latent functions are then weighted by another set of random variables. This defines a probabilistic model that can perform uncertainty quantification, which can then be trained by performing MCMC sampling over the hyperparameters. The acquisition function is calculated by computing the first and second moments of the samples, and calculating the upper confidence bound (UCB).\n\nThis procedure results in a non-stationary, non-separable model that can capture complex functions. This model is tested on a range of synthetic and ML hyperparameter BO benchmarks, each of which are non-separable functions that exhibit high degrees of interaction between variables.\n\nSTRENGTHS:\n**Originality:**\nThis paper is the first to use kernelized tensors in the Bayesian optimization setting. Gaussian processes are a common surrogate in this setting, and this proposes an alternate surrogate with good arguments for its adoption.\n\n**Quality:**\nThe paper demonstrates the performance of the BKTF surrogate on a wide range of benchmarks. The performance is strong, justifying the claims in the paper. Further, many supplementary results are provided to further investigate the modelling decisions made.\n\n**Clarity:**\nThe explanation of the BKTF model and fitting process is explained well, providing a clear description of the model. Specifically, the 2D example in Figure 1 provides a clear motivation for modelling functions that have a high degree of interaction.\n\n**Significance:**\nThe proposed method is a strong surrogate for Bayesian optimization, one that can model functions with mixed input spaces and high degrees of interaction between variables. This presents a good addition to the range of available surrogates in the field.\n\nWEAKNESSES:\n**Comparison to other methods:**\nThe authors claim that a strength of their method is that their method is non-stationary and non-separable. However, I do not feel that the paper sufficiently justifies this explanation for the model's success, for the following reasons:\n\nI do not believe that the GP ARD is separable. The authors present that the ARD kernel is the product of $D$ independent kernels in 3.2. However, this is not how these kernels are implemented. Instead, (specifically for stationary kernels) they are expressed as functions of weighted distance [5], where $d=\\sqrt{\\sum_{d=1}^D (x_d - x'_d)^2/l_d}$. These kernels cannot be written as products of 1-dimensional kernels, and are non-separable. The authors also suggest that these experience the curse of dimensionality as the dimension increases, but this effect is not severe in the low-dimension regime studied in the paper.\n\nThe authors claim that the flaw of using additive GP kernels is that they:\n> [require] strong prior knowledge to determine the proper interactions and [involve] a large number of kernel hyperparameters to learn\n\nI do not find this argument convincing. For low dimensional problems, additive kernels can include all interactions up to order D, and learn the weighting of each order of interaction [1]. In fact, I believe that the number of kernel hyperparameters is of a similar order to the BKTF method. Additive kernels also work well with MAP estimation of the hyperparameters (especially for the low-dimensional problems investigated here), although I do not see why MCMC could not also be used for additive kernels if the number of kernel hyperparameters is considered too large. The additive GP baseline should therefore be order D, not order 1, to provide a fair comparison against existing non-separable modelling methods.\n\nIt is unclear why the authors compare to SaaSBO, a technique designed for high dimensional (D>100) spaces that places a strong prior on the lengthscales of the inputs (with the prior belief that few of the inputs are important, which is not the case for the test functions used).\n\nMoreover, the authors do not provide comparisons against methods that are designed for non-stationary settings e.g. [1, 2]. \n\n(Minor comment) I would like to see MCMC over the GP kernel hyperparameters to obtain a fully Bayesian acquisition function, as in [6].\n\n**Hyperparameter choices with BKTF:**\nI disagree with the authors that BKTF is robust to rank misspecification. Figure 13 shows that the CRPS doesn't converge until 30 observations for the rank 2 model. Including the initial 30 datapoints, this model is fit on 60 datapoints, for a 2D problem - the GP model provides a much better fit to the data for <60 datapoints.  Since these models are used in a BO setting where few datapoints is common, this behaviour suggests that rank *is* an important parameter, and further that the model does not fit well with few datapoints. \n\nI would also want to see this experiment repeated on higher dimension test problems, to see if the problem is exacerbated in high dimensions. Moreover, the CRPS of the rank 2 model seems to converge only to that of the GP models, suggesting the performance over GP may not be solely due to modeling quality: this should be further investigated.\n\n**Experiments:**\nIt is unclear how the authors handle categorical inputs for the baselines. The authors should be using methods designed for mixed input spaces, such as [4].\n\nFollowing from the discussion on CRPS, this paper would benefit from some experiments on the quality of the fit of the model.\n\n(Minor comment) It would be interesting to see the (arguably more popular) Matern 5/2 kernel compared to the 3/2 kernel, especially for the GP baselines.\n\n\n[1] Snoek et al. \"Input Warping for Bayesian Optimization of Non-stationary Functions\"    \n[2] Eriksson et al. \"Scalable Global Optimization via Local Bayesian Optimization\"    \n[3] Duvenaud et al. \"Additive Gaussian Processes\"    \n[4] Ru et al. \"Bayesian optimisation over multiple continuous and categorical inputs\"    \n[5] Williams and Rasmussen. \"Gaussian Processes for Machine Learning\"    \n[6] Snoek et al. \"Practical Bayesian Optimization of Machine Learning Algorithms\"\n\nQUESTIONS:\nAppendix F claims that BKTF has the lowest computational complexity. Is this for the entire MCMC chain, or just for each sample? How does the wall-clock time compare against these other methods?\n\nWhy do you use $R=2$ for synthetic experiments, and $R=4$ for the hyperparameter tuning problems? Do you observe similar results when you use $R\\in\\{6,8\\}$? What are the disadvantages of using a higher rank?\n\nIn E.3, you claim that BKTF learns periodicity of the function. However, a Matern kernel that fits to data sampled from a sine function will appear to have learned periodicity in the input domain. Has the function truly learned periodicity - if you test the function well outside the domain of the function, does it still exhibit periodic behaviour?\n\nThe authors note that the BKTF model is a special case of a deep GP. Could you provide insight as to why this model still outperforms deepgp?\n\n(Minor question) How many MCMC iterations are generally needed to converge? 400 iterations are used in 5.1, 600 in 5.2, and 1000 samples are used in C.2. Do you have any insight for the convergence of the sampling procedure? Do you run multiple chains in parallel?\n\n(Minor question) How do you select values for $\\{a_0, b_0\\}$, and what values do you use in your experiments?\n\n(Minor question) BKTFRandom selects a subset of 20,000 points from the domain. However, the search space for the synthetic functions is $(m_d)^D$, which is less than 20,000. Why does BKTFRandom then perform worse than BKTF on these functions?\n\nLIMITATIONS:\nThe authors provide good discussion on the limitations of their approach.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new surrogate model for Bayesian optimization, based on a functional tensor factorization. The approach discretizes the model to a pre-specified grid and uses MCMC sampling for inference. Bayesian optimization is carried out by selecting promising points from the pre-specified grid, as quantified by an acquisition function. The paper includes experimental results on synthetic functions as well as ML hyper-parameter tuning tasks.\n\nSTRENGTHS:\n- Nice experimental results on the ML hyper-parameter tuning task.\n- Generally well written paper.\n\nWEAKNESSES:\n- Optimization method appears to be restricted to an a-priori defined grid of possible candidate points.\n\n- I am concerned about reproducibility of the benchmarks, as the code submission not complete, e.g. appears to be missing implementations of basic functions like baseline GP fitting (`fitrgp`) and predicting (`predict`), the additive GP model mentioned in line 274 of the paper, the continuous optimization of the acquisition function. \n\n- Grid-based GP-UCB, GP-EI baselines are missing for ML tuning tasks (Figure 3). This is notable, because these experiments contain discrete variables, which requires a rounding operation if they are relaxed to a continuous space, as is done by the paper. This rounding operation is likely to degrade the performance of \"continuous\" GP-UCB and GP-EI, as it will can be prone to sampling \"between\" integers it has already seen, reducing its sample efficiency.\n\nQUESTIONS:\n1) > BKTFrandom. BKTFrandom can be applied to functions with higher dimensions (e.g., D > 10).\n\nDid you try this out, and do you have any high-dimensional benchmark results to share?\n\n2) > The predictive distribution for any entry f in the defined grid space conditioned on the observed dataset Dn can be obtained by the Monte Carlo approximation\n\nCan you clarify if the algorithm only works efficiently by amortizing MC inference to a fixed grid, and then optimizing the acquisition function on that grid? Is the implementation capable of numerical (\"continuous) optimization of the acquisition function, and did you test this in any experiments?\n\n3) > This allows us to accommodate observations that are not located in the grid space QD d=1 Sd. The detailed derivation of the sampling algorithm is given in Appendix A.\n\nIt is not clear to me how Appendix A addresses the problem of off-grid inference, since e.g. it says:\n\n> \"we define a binary tensor O with the same size of Y indicating the locations of the observation data\".\n\nwhich only seems possible when restricted to a grid.\n\n4) How is the acquisition function optimized for the methods that operate on the continuous space, like the non-grid GP EI, GP UCB, and additive GP methods (i.e. using any restarts, optimization algorithm, etc)? \n\n5) Do you have any intuition why the results in Figure 2 are more mixed compared to the baseline, whereas BKTF seems to exhibit a clearer  advantage on the the ML tuning tasks in Figure 3? In addition, please provide the results for the grid-based GP-EI and GP-UCB baselines, like for Figure 2, as this will determine if (part of) the performance gap can be explained by the continuous relaxation. Also, note that the the code for this experiment experiment is not included in the submission, in addition to the other missing code mentioned above. \n\n**Miscellaneous**\n\n1) > perhaps the most popular kernel is the ARD (automatic relevance determination)— Squared-Exponential (SE) or Matérn kernel [4]. Although this specification has certain numerical advantages and can help automatically learn the importance of input variables, a key limitation is hat it implies/assumes that the underlying stochastic process is stationary and separable\n\nThe SE kernel is equivalent to a separability (product kernel) assumption, but the Matérn kernel is not. ARD pertains to having separate lengthscales, which is not the same as an additive or multiplicative separability assumption. Please make the language more precise here. \n\n2) > Another emerging solution is to use deep GP \n\nTypo: Either \"deep GPs\" or \"a deep GP\".\n\n3) > we propose using Bayesian Kernelized Tensor Factorization (BKTF) as a flexible and adaptive surrogate model for BO in a D-dimensional Cartesian product space (i.e., grid) when D is relatively small (say D ≤ 10)\n\nThe introduction also mentions that \"the value of the covariance function between two random points quickly goes to zero with the increase of input dimensionality. These assumptions can be problematic\". Since the introduction just stressed the limitation of canonical BO methods in higher dimensions, it appears important to stress that the method is not designed to mitigate this particular limitation, if it is really limited to D < 10.\n\n4) > GP ARD: kernel function is stationary and separable\n\nCan you provide a reference for a body of work that equates GP ARD with a multiplicative separability assumption? In the GP literature, ARD is associated with having separate lengthscales for each input dimension (see e.g. Section 5.1 of https://gaussianprocess.org/gpml/chapters/RW5.pdf). This *happens* to imply separability for the SE / RBF kernel, but it does not imply separability in general, e.g. for the commonly used Matern kernel.\n\nLIMITATIONS:\n> A limitation of BKTF is that we restrict BO to a grid search space in order to leverage tensor factorization; however, we believe that designing a compatible grid space based on prior knowledge is not a challenging task.\n\nAn important limitation to highlight here once more that it goes from \"not challenging\" to prohibitively expensive as the dimension increases.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper proposes a method for Bayesian optimization where the prior is a low-rank sum of tensor products of GPs. An MCMC scheme is developed for approximate updating and UCB sampling. Several experiments show strong performance relative to baselines on artificial function optimization and ML hyperparam tuning.

STRENGTHS:
This is a seemingly new approach to BO with a more flexible type of prior, which shows good empirical performance.

WEAKNESSES:
It's not clear what is new relative to previous BKTF papers [10,11], other than the scheme for using UQ in UCB sampling.

The clearest potential drawbacks to the approach are the memory and compute costs. These should be reported for the experiments.

QUESTIONS:
Would it be possible to store the means and variances just of the tensor factors ($g$) and use a mean field approximation to more cheaply approximate posterior?

(6) and elsewhere: I think $\otimes$ would be more standard notation than $\circ$.

136: It would help to mention that each input dimension is standardized to $[0,1]$, because otherwise assuming equal length scales seems problematic.

LIMITATIONS:
The paper motivates the approach in part because standard methods assume the generating process is stationary, but the BKTF is also stationary. It’s nonstationary only after conditioning on values of $g$.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13