Title: Bayesian Kernelized Tensor Factorization as Surrogate for Bayesian Optimization
Venue: NeurIPS 2023
Accepted: no
Keywords: Bayesian optimization

surrogate model

low-rank factorization

kernelized tensor factorization

Markov chain Monte Carlo
Abstract: Bayesian optimization (BO) primarily uses Gaussian processes (GP) as the key surrogate model, mostly with a simple stationary and separable kernel function such as the squared-exponential kernel with automatic relevance determination (SE-ARD). However, such simple kernel specifications are deficient in learning functions with complex features, such as being nonstationary, nonseparable, and multimodal. Approximating such functions using a local GP, even in a low-dimensional space, requires a large number of samples, not to mention in a high-dimensional setting. In this paper, we propose to use Bayesian Kernelized Tensor Factorization (BKTF)---as a new surrogate model---for BO in a $D$-dimensional Cartesian product space. Our key idea is to approximate the underlying $D$-dimensional solid with a fully Bayesian low-rank tensor CP decomposition, in which we place GP priors on the latent basis functions for each dimension to encode local consistency and smoothness. With this formulation, information from each sample can be shared not only with neighbors but also across dimensions. Although BKTF no longer has an analytical posterior, we can still efficiently approximate the posterior distribution through Markov chain Monte Carlo (MCMC) and obtain prediction and full uncertainty quantification (UQ). We conduct numerical experiments on both standard BO test functions and machine learning hyperparameter tuning problems, and our results show that BKTF offers a flexible and highly effective approach for characterizing complex functions with UQ, especially in cases where the initial sample size and budget are severely limited. 

Abstract

Bayesian optimization (BO) primarily uses Gaussian processes (GP) as the key surrogate model, mostly with a simple stationary and separable kernel function such as the squared-exponential kernel with automatic relevance determination (SE-ARD). However, such simple kernel specifications are deficient in learning functions with complex features, such as being nonstationary, nonseparable, and multimodal. Approximating such functions using a local GP, even in a low-dimensional space, requires a large number of samples, not to mention in a high-dimensional setting. In this paper, we propose to use Bayesian Kernelized Tensor Factorization (BKTF)-as a new surrogate model-for BO in a D -dimensional Cartesian product space. Our key idea is to approximate the underlying D -dimensional solid with a fully Bayesian low-rank tensor CP decomposition, in which we place GP priors on the latent basis functions for each dimension to encode local consistency and smoothness. With this formulation, information from each sample can be shared not only with neighbors but also across dimensions. Although BKTF no longer has an analytical posterior, we can still efficiently approximate the posterior distribution through Markov chain Monte Carlo (MCMC) and obtain prediction and full uncertainty quantification (UQ). We conduct numerical experiments on both standard BO test functions and machine learning hyperparameter tuning problems, and our results show that BKTF offers a flexible and highly effective approach for characterizing complex functions with UQ, especially in cases where the initial sample size and budget are severely limited.

1 Introduction

For many applications in sciences and engineering, such as emulation-based studies, design of 23 experiments, and automated machine learning, the goal is to optimize a complex black-box function 24 f ( x ) in a D -dimensional space, for which we have limited prior knowledge. The main challenge in 25 such optimization problems is that we aim to efficiently find global optima rather than local optima, 26 while the objective function f is often gradient-free, multimodal, and computationally expensive 27 to evaluate. Bayesian optimization (BO) offers a powerful statistical approach to these problems, 28 particularly when the observation budgets are limited [1, 2, 3]. A typical BO framework consists of 29 two components to balance exploitation and exploration: the surrogate and the acquisition function 30 (AF). The surrogate is a probabilistic model that allows us to estimate f ( x ) with uncertainty at a new 31 location x , and the AF is used to determine which location to query next. 32

Gaussian process (GP) regression is the most widely used surrogate for BO [3, 4], thanks to its 33

appealing properties in providing analytical derivations and uncertainty quantification (UQ). The 34

choice of kernel/covariance function is a critical decision in GP models; for multidimensional 35

BO problems, perhaps the most popular kernel is the ARD (automatic relevance determination)36

Squared-Exponential (SE) or Matérn kernel [4]. Although this specification has certain numerical 37 advantages and can help automatically learn the importance of input variables, a key limitation is that 38 it implies/assumes that the underlying stochastic process is both stationary and separable, and the 39 value of the covariance function between two random points quickly goes to zero with the increase of 40 input dimensionality. These assumptions can be problematic for complex real-world processes that 41 are nonstationary and nonseparable, as estimating the underlying function with a simple ARD kernel 42 would require a large number of observations. A potential solution to address this issue is to use more 43 flexible kernel structures. The additive kernel, for example, is designed to characterize a more 'global' 44 and nonstationary structure by restricting variable interactions [5], and it has demonstrated great 45 success in solving high-dimensional BO problems (see, e.g., [6, 7, 8]). However, in practice using 46 additive kernels requires strong prior knowledge to determine the proper interactions and involves 47 many kernel hyperparameters to learn [9]. Another emerging solution is to use deep GP [10], such as 48 in [11]; however, for complex multidimensional functions, learning a deep GP model will require a 49 large number of samples. 50

51

52

53

54

55

56

57

58

59

60

In this paper, we propose to use Bayesian Kernelized Tensor Factorization (BKTF) [12, 13, 14] as a flexible and adaptive surrogate model for BO in a D -dimensional Cartesian product space. BKTF is initially developed for modeling multidimensional spatiotemporal data with UQ, for tasks such as spatiotemporal kriging/cokriging. This paper adapts BKTF to the BO setting, and our key idea is to characterize the multivariate objective function f ( x ) = f ( x 1 , . . . , x D ) for a specific BO problem using low-rank tensor CANDECOMP/PARAFAC (CP) factorization with random basis functions. Unlike other basis-function models that rely on known/deterministic basis functions [15], BKTF uses a hierarchical Bayesian framework to achieve high-quality UQ in a more flexible way-GP priors are used to model the basis functions, and hyperpriors are used to model kernel hyperparameters in particular for the lengthscale that characterizes the scale of variation.

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

Figure 1 shows the comparison between BKTF and GP surrogates when optimizing a 2D function that is nonstationary, nonseparable, and multimodal. The details of this function and the BO experiments are provided in Appendix 7.3, and related code is given in Supplementary material. For this case, GP becomes ineffective in finding the global solution, while BKTF offers superior flexibility and adaptability to characterize the multidimensional process from limited data. Different from GP-based surrogate models, BKTF no longer has an analytical posterior; however, efficient inference and acquisition can be achieved through Markov chain Monte Carlo (MCMC) in an element-wise learning way, in which we update basis functions and kernel hyperparameters using Gibbs sampling and slice sampling respectively [14]. For the optimization, we first use MCMC samples to approximate the posterior distribution of the whole tensor and then naturally define the upper confidence bound (UCB) as AF. This process is feasible for many real-world applications that can be studied in a discretized tensor product space, such as experimental design and automatic machine learning (ML). We conduct extensive experiments on both standard optimization and ML hyperparameter tuning tasks. Our results show that BKTF achieves a fast global search for optimizing complex objective functions under limited initial data and observation budgets.

GPαEI

GPαUCB

(a)

truefunction

(c)

BKTF

estimationsurface

0.5

0

0

0

0

0

02

0

-0.5

-1

1

-0.5

1

-0.5

1

-0.5

1

-0.5

-1

C1

1.5

2 -1

C2

2

1

1

C1

1.5

2 -1

C2

C1

1.5

2 -1

C2

C1

1.5

2-1

32

-1.5

(b)

*

0.15

0.2

-GPαEI

—GPαuCB

—BKTF αB-UCB

1

1

surface

x10~3

1.5

2

0.5

0

1

0.1

210

0

01

0

-0.5

3

0

0.5

0.05

1

1

-0.5

0.5

1

-0.5

-1

1

-0.5

0

0

1.5

0

1.5

1.5

-0.5

0

20

40

C1

2 -1

C2

C1

2-1

C2

C1

2 -1

C2

no.ofevaluations

2 Preliminaries 76

77

78

79

80

Throughout this paper, we use lowercase letters to denote scalars, e.g., x , boldface lowercase letters to denote vectors, e.g., x = ( x 1 , . . . , x D ) ⊤ ∈ R D , and boldface uppercase letters to denote matrices, e.g., X ∈ R M × N . For a matrix X , we denote its determinant by det ( X ) . We use I N to represent an identity matrix of size N . Given two matrices A ∈ R M × N and B ∈ R P × Q , the Kronecker

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

product is defined as A ⊗ B =   a 1 , 1 B · · · a 1 ,N B . . . . . . . . . a M, 1 B · · · a M,N B   ∈ R MP × NQ . The outer product of two vectors a and b is denoted by a ◦ b . The vectorization operation vec( X ) stacks all column vectors in X as a single vector. Following the tensor notation in [16], we denote a third-order tensor by X ∈ R M × N × P and its modek ( k = 1 , 2 , 3 ) unfolding by X ( k ) , which maps a tensor into a matrix. Higher-order tensors can be defined in a similar way.

Let f : X = X 1 × . . . ×X D → R be a black-box function that could be nonconvex, derivative-free, and expensive to evaluate. BO aims to address the global optimization problem:

BO solves this problem by first building a probabilistic model for f ( x ) (i.e., surrogate model) based on initial observations and then using the model to decide where in X to evaluate/query next. The overall goal of BO is to find the global optimum of the objective function through as few evaluations as possible. Most BO models rely on a GP prior for f ( x ) to achieve prediction and UQ:

where k is a valid kernel/covariance function and m is a mean function that can be generally assumed to be 0. Given a finite set of observation points { x i } n i =1 with x i = ( x i 1 , . . . , x i D ) ⊤ , the vector of function values f = ( f ( x 1 ) , . . . , f ( x n )) ⊤ has a multivariate Gaussian distribution f ∼ N ( 0 , K ) , where K denotes the n × n covariance matrix. For a set of observed data D = { x i , y i } n i =1 with i.i.d. Gaussian noise, i.e., y i = f ( x i ) + ϵ i where ϵ i ∼ N (0 , τ -1 ) , GP gives an analytical posterior distribution of f ( x ) at an unobserved point x ∗ :

where k x ∗ x ∗ , k x ∗ X ∈ R 1 × n are variance of x ∗ , covariances between x ∗ and { x i } n i =1 , respectively, and y = ( y 1 , . . . , y n ) ⊤ .

Based on the posterior distributions of f , one can compute an AF, denoted by α : X → R , for a new candidate x ∗ and evaluate how promising x ∗ is. In BO, the next query point is often determined by maximizing a selected/predefined AF, i.e., x n +1 = arg max x ∈X α ( x | D n ) . Most AFs are built on the predictive mean and variance; for example, a commonly used AF is the expected improvement (EI) [1]:

Algorithm 1: Basic BO process

Input: Initial dataset D 0 and a trained surrogate model; total budget N . for n = 1 , . . . , N do Compute the posterior distribution of f using all available data; Find next evaluation point x n ∈ R D by optimizing the AF; Augment data D n = D n -1 ∪ { x n , y n } , update surrogate model.

where ∆( x ) = µ ( x ) -f ⋆ n is the expected difference between the proposed point x and the current best solution, f ⋆ n = max x ∈{ x i } n i =1 f ( x ) denotes the best function value obtained so far; µ ( x ) and σ ( x ) are the predictive mean and predictive standard deviation at x , respectively; and φ ( · ) and Φ( · ) denote the probability density function (PDF) and the cumulative distribution function (CDF) of standard normal, respectively. Another widely applied AF for maximization problems is the upper confidence bound (UCB) [17]:

where β is a tunable parameter that balances exploration and exploitation. The general BO procedure can be summarized as Algorithm 1.

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

3 Bayesian Kernelized Tensor Factorization for BO

3.1 Bayesian Hierarchical Model Specification

Before introducing BKTF, we first construct a D -dimensional Cartesian product space corresponding to the search space X . We define it over D sets { S 1 , . . . , S D } and denote as ∏ D d =1 S d : S 1 ×··· × S D = { ( s 1 , . . . , s D ) | ∀ d ∈ { 1 , . . . , D } , s d ∈ S d } . For ∀ d ∈ [1 , D ] , the coordinates set S d is formed by m d interpolation points that are distributed over a bounded interval X d = [ a d , b d ] , represented by c d = { c d 1 , . . . , c d m d } , i.e., S d = { c d i } m d i =1 . The size of S d becomes | S d | = m d , and the entire space owns ∏ D d =1 | S d | samples. Note that S d could be either uniformly or irregularly distributed.

We randomly sample an initial dataset including n 0 input-output data pairs from the pre-defined space, D 0 = { x i , y i } n 0 i =1 where { x i } n 0 i =1 are located in ∏ D d =1 S d , and this yields an incomplete D -dimensional tensor Y ∈ R | S 1 |×···×| S D | with n 0 observed points. BKTF approximates the entire data tensor Y by a kernelized CANDECOMP/PARAFAC (CP) tensor decomposition:

where R is a pre-specified tensor CP rank, λ = ( λ 1 , . . . , λ R ) ⊤ denote weight coefficients that capture the magnitude/importance of each rank in the factorization, g r d = [ g r d ( s d ) : s d ∈ S d ] ∈ R | S d | denotes the r th latent factor for the d th dimension, entries in E are i.i.d. white noises from N (0 , τ -1 ) . It should be particularly noted that both the coefficients { λ r } R r =1 and the latent basis functions { g r 1 , . . . , g r D } R r =1 are random variables. The function approximation for x = ( x 1 , . . . , x D ) ⊤ can be written as:

For priors, we assume λ r ∼ N (0 , 1) for r = 1 , . . . , R and use a GP prior on the latent factors:

where k r d is a valid kernel function. We fix the variances of k r d as σ 2 = 1 , and only learn the length-scale hyperparameters l r d , since the variances of the model can be captured by λ . One can also exclude λ but introduce variance σ 2 as a kernel hyperparameter on one of the basis functions; however, learning kernel hyperparameter is computationally more expensive than learning λ . For simplicity, we can also assume the lengthscale parameters to be identical, i.e., l 1 d = l 2 d = . . . = l R d = l d , for each dimension d . The prior for the corresponding latent factor g r d is then a Gaussian distribution: g r d ∼ N ( 0 , K r d ) , where K r d is the | S d | × | S d | correlation matrix computed from k r d . We place Gaussian hyperpriors on the log-transformed kernel hyperparameters to ensure positive values, i.e., log ( l r d ) ∼ N ( µ l , τ -1 l ) . For noise precision τ , we assume a conjugate Gamma prior τ ∼ Gamma ( a 0 , b 0 ) .

For observations, based on Eq. (7) we assume each y i in the initial dataset D 0 to be:

3.2 BKTF as a Two-layer Deep GP

Here we show the representation of BKTF as a two-layer deep GP. The first layer characterizes the generation of latent functions { g r d } R r =1 for coordinate/dimension d and also the generation of random weights { λ r } R r =1 . For the second layer, if we consider { λ r , g r 1 , . . . , g r D } R r =1 as parameters and rewrite the functional decomposition in Eq. (7) as a linear function f ( x ; { ξ r } ) = ∑ R r =1 ξ r | λ r | ∏ D d =1 g r d ( x d ) with ξ r iid ∼ N (0 , 1) , we can marginalize { ξ r } and obtain a fully symmetric multilinear kernel/covariance function for any two data points x = ( x 1 , . . . , x D ) ⊤ and x ′ = ( x ′ 1 , . . . , x ′ D ) ⊤ :

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

As can be seen, the second layer has a multilinear product kernel function parameterized by { λ r , g r 1 , . . . , g r D } R r =1 . There are some properties to highlight: (i) the kernel is nonstationary since the value of g r d ( · ) is location-specific, and (ii) the kernel is nonseparable when R > 1 . Therefore, this specification is very different from traditional GP surrogates:

where σ 2 represents the kernel variance, and kernel functions { k d ( · ) , k 1st d ( · ) , k 2nd d ( · ) , k 2nd e ( · ) } are stationary with different hyperparameters (e.g., length scale and variance). Compared with GP-based kernel specification, the multilinear kernel in Eq. (10) has a much larger set of hyperparameters and becomes more flexible and adaptive to the data. From a GP perspective, learning the hyperparameter in the kernel function in Eq. (10) will be computationally expensive; however, we can achieve efficient inference of { λ r , g r 1 , . . . , g r D } R r =1 under a tensor factorization framework. Based on the derivation in Eq. (10), we can consider BKTF as a 'Bayesian' version of the multidimensional Karhunen-Loève (KL) expansion [18], in which the basis functions { g r d } are random processes (i.e., GPs) and { λ r } are random variables. On the other hand, we can interpret BKTF as a new class of stochastic process that is mainly parameterized by rank R and hyperparameters for those basis functions; however, BKTF does not impose any orthogonal constraints on the latent functions.

3.3 Model Inference

Unlike GP, BKTF no longer enjoys an analytical posterior distribution. Based on the aforementioned prior and hyperprior settings, we adapt the MCMC updating procedure in [12, 14] to an efficient element-wise Gibbs sampling algorithm for model inference. This allows us to accommodate observations that are not located on the grid space ∏ D d =1 S d . The detailed derivation of the sampling algorithm is given in Appendix 7.1.

3.4 Prediction and AF Computation

In each step of function evaluation, we run the MCMC sampling process K iterations for model inference, where the first K 0 samples are taken as burn-in and the last K -K 0 samples are used for posterior approximation. The predictive distribution for any entry f ∗ in the defined grid space conditioned on the observed dataset D 0 can be obtained by the Monte Carlo approximation p ( f ∗ | D 0 , θ 0 ) ≈ 1 K -K 0 × ∑ K k = K 0 +1 p ( f ∗ ∣ ∣ ∣ ( g r d ) ( k ) , λ ( k ) , τ ( k ) ) , where θ 0 = { µ l , τ l , a 0 , b 0 } is the set of all parameters used in hyperpriors. Although direct analytical predictive distribution does not exist in BKTF, the posterior mean and variance estimated from MCMC samples at each location naturally offer us a Bayesian approach to define the AFs.

BKTF provides a fully Bayesian surrogate model. We define a Bayesian variant of UCB as the AF by adapting the predictive mean and variance (or uncertainty) in ordinary GP-based UCB with the values calculated from MCMC sampling. For every MCMC sample after burn-in, i.e., k > K 0 , we can estimate a output tensor ˜ F ( k ) over the entire grid space using the latent factors ( g r d ) ( k ) and the weight vector λ ( k ) : ˜ F ( k ) = ∑ R r =1 λ ( k ) r ( g r 1 ) ( k ) ◦ ( g r 2 ) ( k ) ◦ · · · ◦ ( g r D ) ( k ) . We can then compute the corresponding mean and variance tensors of the ( K -K 0 ) samples { ˜ F ( k ) } K k = K 0 +1 , and denote the two tensors by U and V , respectively. The approximated predictive distribution at each point x in the space becomes ˜ f ( x ) ∼ N ( u ( x ) , v ( x )) . Following the definition of UCB in Eq. (5), we define Bayesian UCB (B-UCB) at location x as α B-UCB ( x | D , β, g r d , λ ) = u ( x ) + β √ v ( x ) . The next search/query point can be determined via x next = arg max x ∈{ ∏ D d =1 S d -D n -1 } α B-UCB ( x ) .

We summarize the implementation procedure of BKTF for BO in Appendix 7.2 (see Algorithm 2).

Given the sequential nature of BO, when a new data point arrives at step

n

, we can start the MCMC

with the last iteration of the Markov chains at step

n

-

1

to accelerate model convergence. The main computational and storage cost of BKTF is to interpolate and save the tensors

over

(

K

-

K

0

)

˜

F

∈

R

|

S

1

|×···×|

S

D

|

iterations for Bayesian AF estimation. This could be prohibitive when the MCMC

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

sample size or the dimensionality of input space is large. To avoid saving the tensors, in practice, we can simply use the maximum values of each entry over the ( K -K 0 ) iterations through iterative pairwise comparison. The number of samples after burn-in then implies the value of β in α B-UCB. We adopt this simple AF in our numerical experiments.

4 Related Work

The key of BO is to effectively characterize the posterior distribution of the objective function from a limited number of observations. The most relevant work to our study is the Bayesian Kernelized Factorization (BKF) framework, which has been mainly used for modeling large-scale and multidimensional spatiotemporal data with UQ. The key idea is to parameterize the multidimensional stochastic processes using a factorization model, in which specific priors are used to encode spatial and temporal dependencies. Signature examples of BKF include spatial dynamic factor model (SDFM) [19], variational Gaussian process factor analysis (VGFA) [20], and Bayesian kernelized matrix/tensor factorization (BKMF/BKTF) [12, 14, 13]. A common solution in these models is to use GP prior to modeling the factor matrices, thus encoding spatial and temporal dependencies. In addition, for multivariate data with more than one attribute, BKTF also introduces a Wishart prior to modeling the factors that encode the dependency among features. A key difference among these methods is how inference is performed. SDFM and BKMF/BKTF are fully Bayesian hierarchical models and they rely on MCMC for model inference, where the factors can be updated via Gibbs sampling with conjugate priors; for learning the posterior distributions of kernel hyperparameters, SDFM uses the Metropolis-Hastings sampling, while BKMF/BKTF uses the more efficient slice sampling. On the other hand, VGFA uses variational inference to learn factor matrices, while kernel hyperparameters are learned through maximum a posteriori (MAP) estimation without UQ. Overall, BKTF has shown superior performance in modeling multidimensional spatiotemporal processes with high-quality UQ for 2D and 3D spaces [14] and conducting tensor regression [13].

The proposed BKTF surrogate models the objective function-as a single realization of a random process-using low-rank tensor factorization with random basis functions. This basis functionbased specification is closely related to multidimensional Karhunen-Loève (KL) expansion [18] for stochastic (spatial, temporal, and spatiotemporal) processes. The empirical analysis of KL expansion is also known as proper orthogonal decomposition (POD). With a known kernel/covariance function, truncated KL expansion allows us to approximate the underlying random process using a set of eigenvalues and eigenfunctions derived from the kernel function. Numerical KL expansion is often referred to as the Garlekin method, and in practice the basis functions are often chosen as prespecified and deterministic functions [15, 21], such as Fourier basis, wavelet basis, orthogonal polynomials, B-splines, empirical orthogonal functions, radial basis functions (RBF), and Wendland functions (i.e., compactly supported RBF) (see, e.g., [22], [23], [24], [25]). However, the quality of UQ will be undermined as the randomness is fully attributed to the coefficients { λ r } ; in addition, these methods also require a large number of basis functions to fit complex stochastic processes. Different from methods with fixed/known basis functions, BKTF uses a Bayesian hierarchical modeling framework to better capture the randomness and uncertainty in the data, in which GP priors are used to model the latent factors (i.e., basis functions are also random processes) on different dimensions, and hyperpriors are introduced on the kernel hyperparameters. Therefore, BKTF becomes a fully Bayesian version of multidimensional KL expansion for stochastic processes with unknown covariance from partially observed data, however, without imposing any orthogonal constraint on the basis functions. Following the analysis in section 3.2, BKTF is also a special case of a two-layer deep Gaussian process [26, 10], where the first layer produces latent factors for each dimension, and the second layer holds a multilinear kernel parameterized by all latent factors.

5 Experiments

5.1 Optimization for Benchmark Test Functions

We test the proposed BKTF model for BO on six benchmark functions that are used for global optimization problems [27], which are summarized in Table 1. Figure 2(a) shows those functions with 2-dimensional inputs together with the 2D Griewank function. All the selected standard functions are multimodal, more detailed descriptions can be found in Appendix 7.4. In fact, we can visually see that the standard Damavandi/Schaffer/Griewank functions in Figure 2(a) indeed have a low-rank

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

structure. For each function, we assume the initial dataset D 0 contains n 0 = D observed data pairs, and we set the total number of query points to N = 80 for 4D Griewank and 6D Hartmann function and N = 50 for others. We rescale the input search range to [0 , 1] for all dimensions and normalize the output data using z-score normalization.

Model configuration. When applying BKTF on the continuous test functions, we introduce m d interpolation points c d in the d th dimension of the input space. The values of m d used for each benchmark function are predefined and given in Table 1. Setting the resolution grid will require certain prior knowledge (e.g., smoothness of the function); and it also depends on the available computational resources and the number of entries in the tensor which grows exponentially with m d . In practice, we find that setting m d = 10 ∼ 100 is sufficient for most problems. We set the CP rank R = 2 , and for each BO function evaluation run 400 MCMC iterations for model inference where the first 200 iterations are taken as burn-in. We use Matérn 3/2 kernel as the covariance function for all the test functions. Since we build a fully Bayesian model, the hyperparameters of the covariance functions can be updated automatically from the data likelihood and hyperprior.

Effects of hyperpriors. Note that in optimization scenarios where the observation data is scarce, the model performance of BKTF highly depends on the hyperprior settings on the kernel length-scales of the latent factors and the model noise precision τ when proceeding estimation for the unknown points, i.e., θ 0 = { µ l , τ l , a 0 , b 0 } . A proper hyper-prior becomes rather important. We discuss the effects of { µ l , τ l } in Appendix 7.5.1. We see that for the re-scaled input space, a reasonable setting is to suppose the mean prior of the kernel length-scales is around half of the input domain, i.e., µ l = log (0 . 5) . The hyperprior on τ impacts the uncertainty of the latent factors, for example, a large model noise assumption allows more variances in the factors. Generally, we select the priors that make the noise variances not quite large, such as the results shown in Figure 4(a) and Figure 5(b) in Appendix. An example of the uncertainty provided by BKTF is explained in Appendix 7.3.

Baselines. We compare BKTF with the following BO methods that use GP as the surrogate model. (1) GP α EI: GP as the surrogate model and EI as the AF in continuous space ∏ D d =1 X d ; (2) GP α UCB: GP as the surrogate model with UCB as the AF with β = 2 , in ∏ D d =1 X d ; (3) GPgrid α EI: GP as the surrogate model with EI as the AF, in Cartesian grid space ∏ D d =1 S d ; (4) GPgrid α UCB: GP as the surrogate model with UCB as the AF with β = 2 , in ∏ D d =1 S d . We use the Matérn 3/2 kernel for all GP surrogates. For AF optimization in GP α EI and GP α UCB, we firstly use the DIRECT algorithm [28] and then apply the Nelder-Mead algorithm [29] to further search if there exist better solutions.

Results. To compare optimization performances of different models on the benchmark functions, we consider the absolute error between the global optimum f ⋆ and the current estimated global optimum ˆ f ⋆ , i.e., ∣ ∣ ∣ f ⋆ -ˆ f ⋆ ∣ ∣ ∣ , w.r.t. the number of function evaluations. We run the optimization 10 times for every test function with a different set of initial observations. The results are summarized in Figure 2(b). We see that for the 2D functions Branin and Schaffer, BKTF clearly finds the global optima much faster than GP surrogate-based baselines. For Damavandi function, where the global minimum ( f ( x ⋆ ) = 0 ) is located at a small sharp area while the local optimum ( f ( x ) = 2 ) is located at a large smooth area (see Figure 2(a)), GP-based models are trapped around the local optima in most cases, i.e., ∣ ∣ ∣ f ⋆ -ˆ f ⋆ ∣ ∣ ∣ = 2 , and cannot jump out. On the contrary, BKTF explores the global characteristics of the objective function over the entire search space and reaches the global optimum within 10 iterations of function evaluations. For higher dimensional Griewank and Hartmann functions, BKTF successfully arrives at the global optima under the given observation budgets, while GP-based comparison methods are prone to be stuck around local optima. We illustrate the latent

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

(a)

Branin·globaloptimumDamavandi

Schaffer

Griewank

(q)

10

GP αEI

15

Schaffer

f

150

1

0

GPαuCB

10

0.4

200

100

0.5

-1

5

_GPgridae1

GPgridαucB

5

0.2

-5

0

0

Z1

5

100

5

32

10

15

50

0

0

5

1014

-50

0

0

50-50

0

50

-10

-2

0

10-10

0

10

1

0

0

20

40

50

0

0

20

4050

0

0

20

4050

(c)

Schaffer

Acrosstestfunctions;

188

Griewank(D=3)

8.8

Griewank (D=4)

2.3

Hartmann

GPaEI

2

ud)d

--GPαuCB

0.6

0.6

1.5

0.5

0.5

0.5

- -GPgrid ae1

0.4

0.4

1

--GPgriducB

0.2

0.2

0.5

0

1020304050

0

1020304050

0

1020304050

0

20

..BKTF αB-UCB

40

60

80

0

0

20

4050

0

50

0

0

50

p

p

p

p

no.ofevaluations

no.ofevaluations

no.ofevaluations

Best results are highlighted in bold fonts.

factors of BKTF for 3D Griewank function in Appendix 7.5.3, which shows the periodic (global) patterns automatically learned from the observations. We compare the absolute error between f ⋆ and the final estimated ˆ f ⋆ in Table 2. The enumeration-based GP surrogates, i.e., GPgrid α EI and GPgrid α UCB, perform a little better than direct GP-based search, i.e., GP α EI and GP α UCB on Damavandi function, but worse on others. This means that the discretization, to some extent, offers possibilities for searching all the alternative points in the space, since in each function evaluation, every sample in the space is equally compared solely based on the predictive distribution. Overall, BKTF reaches the global optimum for every test function and shows superior performance for complex objective functions with a faster convergence rate. To intuitively compare the overall performances of different models across multiple experiments/functions, we further estimate performance profiles (PPs) [30] (see Appendix 7.5.2), and compute the area under the curve (AUC) for quantitative analyses (see Figure 2(c) and Table 2). Clearly, BKTF obtains the best performance across all functions.

5.2 Hyperparameter Tuning for Machine Learning

In this section, we evaluate the performance of BKTF for automatic machine-learning tasks. Specifically, we compare different models to optimize the hyperparameters of two machine learning (ML) algorithms-random forest (RF) and neural network (NN)-on classification for the MNIST database of handwritten digits 1 and housing price regression for the Boston housing dataset 2 . The details of the hyperparameters that need to learn are given in Appendix 7.6. We assume the number of data points in the initial dataset D 0 equals the dimension of hyperparameters need to tune, i.e., n 0 = 4 and n 0 = 3 for RF and NN, respectively. The total budget is N = 50 . We implement the RF algorithms using scikit-learn package and construct NN models through Keras with 2 hidden layers. All other model hyperparameters are set as the default values.

Model configuration. We treat all the discrete hyperparameters as samples from a continuous space and then generate the corresponding Cartesian product space ∏ D d =1 S d . One can interpret the candidate values for each hyperparameter as the interpolation points in the corresponding input dimension. According to Appendix 7.6, the size of the spanned space ∏ S d is 91 × 46 × 64 × 10 and 91 × 46 × 13 × 10 for RF classifier and RF regressor, respectively; for the two NN algorithms, the size of parameter space is 91 × 49 × 31 . Similar to the settings on standard test functions, we set the tensor rank R = 2 , set K = 400 and K 0 = 200 for MCMC inference, and use the Matérn 3/2 kernel.

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

Final accuracy for (a) and MSE for (b).

Model

RF

NN

(a)

GP

α

EI

94.12

±

0.16

99.96

±

0.03

GP

α

UCB

94.07

±

0.19

99.99

±

0.02

BO-TPE

94.14

±

0.20

99.96

±

0.02

BKTF

α

B-UCB

94.44

±

0.15

100.00

±

0.00

(b)

GP

α

EI

26.19

±

0.45

38.46

±

3.31

GP

α

UCB

26.29

±

0.35

36.78

±

1.91

BO-TPE

26.27

±

0.31

36.40

±

4.72

BKTF

α

B-UCB

25.03

±

0.18

30.84

±

1.13

The values are presented as mean

±

std.

Best results are highlighted in bold fonts.

Baselines. Other than the GP surrogate-based GP α EI and GP α UCB, we also compare with Treestructured Parzen Estimator (BO-TPE) [31], which is a widely applied BO approach for hyperparameter tuning. We exclude grid-based GP models as sampling the entire grid becomes infeasible.

Results. We compare the accuracy for MNIST classification and MSE (mean squared error) for Boston housing regression both in terms of the number of function evaluations and still run the optimization processes ten times with different initial datasets D 0 . The results obtained by different BO models are given in Figure 3, and the final classification accuracy and regression MSE are compared in Table 3. For BKTF, we see from Figure 3 that the width between the two quartiles of the accuracy and error decreases as more iterations are evaluated, and the median curves present superior convergence rates compared to baselines. For example, BKTF finds the hyperparameters of NN that achieve 100% classification accuracy on MNIST using less than four function evaluations in all ten runs. Table 3 also shows that the proposed BKTF surrogate achieves the best final mean accuracy and regression error with small standard deviations. All these demonstrate the advantage of BKTF as a surrogate for black-box function optimization.

6 Conclusion and Discussions

In this paper, we propose to use Bayesian Kernelized Tensor Factorization (BKTF) as a new surrogate model for Bayesian optimization. Compared with traditional GP surrogates, the BKTF surrogate is more flexible and adaptive to data thanks to the Bayesian hierarchical specification, which provides high-quality UQ for BO tasks. The tensor factorization model behind BKTF offers an effective solution to capture global/long-range correlations and cross-dimension correlations. Therefore, it shows superior performance in characterizing complex multidimensional stochastic processes that are nonstationary, nonseparable, and multimodal. The inference of BKTF is achieved through MCMC, which provides a natural solution for acquisition. Experiments on both test function optimization and MLhyperparameter tuning confirm the superiority of BKTF as a surrogate for BO. A limitation of BKTF is that we restrict BO to Cartesian grid space to leverage tensor factorization; however, we believe designing a compatible grid space based on prior knowledge is not a challenging task.

There are several directions to be explored for future research. A key computational issue of BKTF is that we need to reconstruct the posterior distribution for the whole tensor to obtain the AF. This could be problematic for high-dimensional problems due to the curse of dimensionality. It would be interesting to see whether we can achieve efficient acquisition directly using the basis functions and corresponding weights without constructing the tensors explicitly. In terms of rank determination, we can introduce the multiplicative gamma process prior to learn the rank; this will create a Bayesian nonparametric model that can automatically adapt to the data. In terms of surrogate modeling, we can further integrate a local (short-scale) GP component to construct a more precise surrogate model, as presented in [14]. The combined framework would be more expensive in computation, but we expect the combination to provide better UQ performance. In terms of parameterization, we also expect that introducing orthogonality prior to the latent factors (basis functions) will improve the inference. This can be potentially achieved through more advanced prior specifications such as the Matrix angular central Gaussian [32]. In addition, for the tensor factorization framework, it is straightforward to adapt the model to handle categorical variables as input and multivariate output by placing a Wishart prior to the latent factors for the categorical/output dimension.

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

References

Roman Garnett. Bayesian Optimization . Cambridge University Press, 2023.

Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE , 104(1):148-175, 2015.

Robert B Gramacy. Surrogates: Gaussian Process Modeling, Design, and Optimization for the Applied Sciences . Chapman and Hall/CRC, 2020.

Christopher KI Williams and Carl Edward Rasmussen. Gaussian Processes for Machine Learning . MIT Press, Cambridge, MA, 2006.

David K Duvenaud, Hannes Nickisch, and Carl Rasmussen. Additive Gaussian processes. Advances in neural information processing systems , 24, 2011.

Kirthevasan Kandasamy, Jeff Schneider, and Barnabás Póczos. High dimensional Bayesian optimisation and bandits via additive models. In International conference on machine learning , pages 295-304. PMLR, 2015.

Chun-Liang Li, Kirthevasan Kandasamy, Barnabás Póczos, and Jeff Schneider. High dimensional Bayesian optimization via restricted projection pursuit models. In Artificial Intelligence and Statistics , pages 884-892. PMLR, 2016.

Paul Rolland, Jonathan Scarlett, Ilija Bogunovic, and Volkan Cevher. High-dimensional Bayesian optimization via additive models with overlapping groups. In International conference on artificial intelligence and statistics , pages 298-307. PMLR, 2018.

Mickael Binois and Nathan Wycoff. A survey on high-dimensional Gaussian process modeling 386 with application to Bayesian optimization. ACM Transactions on Evolutionary Learning and 387 Optimization , 2(2):1-26, 2022. 388

Andreas Damianou and Neil D Lawrence. Deep Gaussian processes. In International Conference 389 on Artificial Intelligence and Statistics , pages 207-215, 2013. 390

Annie Sauer, Robert B Gramacy, and David Higdon. Active learning for deep gaussian process 391 surrogates. Technometrics , 65(1):4-18, 2023. 392

Mengying Lei, Aurelie Labbe, Yuankai Wu, and Lijun Sun. Bayesian kernelized matrix 393 factorization for spatiotemporal traffic data imputation and kriging. IEEE Transactions on 394 Intelligent Transportation Systems , 23(10):18962-18974, 2022. 395

Mengying Lei, Aurelie Labbe, and Lijun Sun. Scalable spatiotemporally varying coefficient 396 modeling with bayesian kernelized tensor regression. arXiv preprint arXiv:2109.00046 , 2021. 397

Mengying Lei, Aurelie Labbe, and Lijun Sun. Bayesian complementary kernelized learning for 398 multidimensional spatiotemporal data. arXiv preprint arXiv:2208.09978 , 2022. 399

Noel Cressie, Matthew Sainsbury-Dale, and Andrew Zammit-Mangion. Basis-function models 400 in spatial statistics. Annual Review of Statistics and Its Application , 9:373-400, 2022. 401

Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM Review , 402 51(3):455-500, 2009. 403

Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine 404 Learning Research , 3(Nov):397-422, 2002. 405

Limin Wang. Karhunen-Loeve expansions and their applications . London School of Economics 406 and Political Science (United Kingdom), 2008. 407

Hedibert Freitas Lopes, Esther Salazar, and Dani Gamerman. Spatial dynamic factor analysis. 408 Bayesian Analysis , 3(4):759-792, 2008. 409

Jaakko Luttinen and Alexander Ilin. Variational Gaussian-process factor analysis for modeling 410 spatio-temporal data. Advances in Neural Information Processing Systems , 22:1177-1185, 411 2009. 412

413

414

Holger Wendland. Scattered Data Approximation , volume 17. Cambridge university press, 2004.

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

Rommel G Regis and Christine A Shoemaker. A stochastic radial basis function method for the global optimization of expensive functions. INFORMS Journal on Computing , 19(4):497-509, 2007.

Gregory Beylkin, Jochen Garcke, and Martin J Mohlenkamp. Multivariate regression and machine learning with sums of separable functions. SIAM Journal on Scientific Computing , 31(3):1840-1857, 2009.

Christopher K Wikle and Noel Cressie. A dimension-reduced approach to space-time kalman filtering. Biometrika , 86(4):815-829, 1999.

Mathilde Chevreuil, Régis Lebrun, Anthony Nouy, and Prashant Rai. A least-squares method for sparse low rank approximation of multivariate functions. SIAM/ASA Journal on Uncertainty Quantification , 3(1):897-921, 2015.

Alexandra M Schmidt and Anthony O'Hagan. Bayesian inference for non-stationary spatial covariance structure via spatial deformations. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 65(3):743-758, 2003.

Momin Jamil and Xin-She Yang. A literature survey of benchmark functions for global optimization problems. arXiv preprint arXiv:1308.4008 , 2013.

Donald R Jones, Cary D Perttunen, and Bruce E Stuckman. Lipschitzian optimization without the lipschitz constant. Journal of optimization Theory and Applications , 79(1):157-181, 1993.

John A Nelder and Roger Mead. A simplex method for function minimization. The computer journal , 7(4):308-313, 1965.

Elizabeth D Dolan and Jorge J Moré. Benchmarking optimization software with performance profiles. Mathematical programming , 91(2):201-213, 2002.

James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyperparameter optimization. Advances in neural information processing systems , 24, 2011.

Michael Jauch, Peter D Hoff, and David B Dunson. Monte carlo simulation on the stiefel manifold via polar expansion. Journal of Computational and Graphical Statistics , 30(3):622631, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis proposes a novel surrogate for Bayesian optimization (BO), kernelized tensor factorization (BKTF). The authors claim that BKTF is able to model more complex functions (nonstationary, nonseparable) compared to additive and product kernel Gaussian processes. For inference, they leverage Gibbs sampling to do full Bayesian inference. They compare against BO with the regular Gaussian process surrogate and tree-structured Parzen estimators.\n\nSTRENGTHS:\n* The papers proposes a novel surrogate model, BKTF, for BO. I believe this is new, and as long as the authors can demonstrate the utility of BKTF, this will be a valuable contribution to the BO community.\n\nWEAKNESSES:\n* The proposed strategy uses Gibbs sampling, which is well known scale poorly with the number of parameters and correlations. Thus, I am concerned that this method's performance will fair poorly at even moderately higher dimensions and number of observations than those considered in this paper.\n* On a similar note, unless I'm not mistaken, the method requires to infer the latent functions (or bases) $g_d^D$. This means inference needs to be performed at every BO steps. This contrast to GPs, where, even if one decides to do fully Bayesian inference, he does not to run MCMC at every step. Thus, the method comes with a reduction in flexibility. If the authors believe that their method can work with less expensive inference strategies, say, VI or MAP, then this should be demonstrated and evaluated.\n* The paper claims that the experiments are \"extensive\" (line 73), but unfortunately, I find that the experiments conducted in this paper cannot be considered to be extensive in today's standards. See for example [1,2], which I would consider extensive. Furthermore, at this small scale / low budget applications, noise can very easily swamp the effects. Therefore, I would expect a lot more runs. Moreover, the hyperparameter tuning experiments in Section 5.2 are not reflective of real-world use cases. So these are gain inadequate to evaluate the real world performance of BKTF.\n* Furthermore, the baselines are not enough. The research space for alternatives to BO surrogates has certainly been active, but here only the tree-structured Parzen estimator is considered. In fact, the paper mentions that BKTF here corresponds to a two-layer deep GP. Then, they should compare against deep GPs for an apple-to-apple comparison. The computational costs/scalability of DGPs would probably be comparable so this would be a more appropriate comparison.\n\nQUESTIONS:\n### Additional Major Comments\n* The paper repeatedly uses the term \"separable\" to characterize the SE-ARD kernel. I have never seen \"separable\" used in the context other than \"additively separable.\" Multiplicative kernels model the correlations between dimensions, thus I'm not sure what BKTF is doing more than SE-ARD. \n* Line 49-50: The paper claims that deep GPs require a large number of samples. I presume this statement is relative to BKTF, but I do not find evidence in this paper that BKTF require less samples.\n* Line 251: What does \"low-rank\" mean in this context?\n* The paper does not cite the original source when referring to existing methods. For example:\n  * Line 56: CANDECOMP, PARAFAC\n  * Line 69: Slice sampling\n  * Line 117: I think the GP-UCB paper [3] should be cited here.\n* Line 115: $\\beta$ in GP-UCB is not necessarily a tunable parameter in the sense that, the optimal configuration, under assumptions, is very specific. See [3].\n* The papers applied UCB acquisition functions to their method, but UCB is known to be conservative, and not that competitive among acquisition functions. Thus, I recommend using other acquisition functions.\n* Section 4: Considering that this paper proposes for an alternative surrogate, the related work section should put the proposed method in context of alternative surrogates. Unfortunately, the current form mostly discusses kernel factorization, which I think is less useful for the BO community. After all, the papers on BKTF referenced herein could be referred to obtain context.\n\n### Minor Comments\n* Above Line 158: kerenl $\\to$ kernel.\n* Between Line 87 and 88: Normally, we denote that the expectation of the noisy version of $f$, for example, $\\hat{f} = f + \\epsilon$, is minimized instead of saying that we optimize $f$ directly.\n\n\n### References\nI am not affiliated with any of the papers and authors mentioned in this review.\n1. [1] Bodin, Erik, et al. \"Modulating surrogates for Bayesian optimization.\" International Conference on Machine Learning. PMLR, 2020.\n2. [2] Malkomes, Gustavo, and Roman Garnett. \"Automating Bayesian optimization with Bayesian optimization.\" Advances in Neural Information Processing Systems 31 (2018).\n3. [3] Srinivas, Niranjan, et al. \"Information-theoretic regret bounds for gaussian process optimization in the bandit setting.\" IEEE transactions on information theory 58.5 (2012): 3250-3265.\n\nLIMITATIONS:\nYes, in Section 6. However, I think the limitations I've discussed above could also be included.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBayesian optimisation most commonly uses Gaussian Processes with the Squared exponential or Matern kernel as the surrogate model. The authors propose a new type of surrogate model, \"Bayesian Kernelized Tensor Factorization\" which introduces some advantages and disadvantages over Gaussian Processes. There seems to be prior work investigating these models for surrogate modding in general, and this paper is a followup applying these models within Bayesian optimisation in particular.\n\nThe papers introduces the model which models the data $\\{x_i, y_i}$ from a black box function $y=f(x)$ as a sum of functions where each function is a product of 1 dimensional GPs, e.g. in 2D, leteach $g()$ be a 1D GP then \n$$\n\\hat{f}(x_1, x_2) = \\sum_i g^i_1(x_1)g^i_2(x_2)\n$$\nwhich is a continuous analogue of how a matrix can be represented by it's SVD or eigen decomposition. This concept generalizes for multiple input dimensions (e.g. $g^i_1(x_1)g^i_2(x_2)g^i_3(x_3)g^i_4(x_4).....$) and the authors discretize the search space into a grid hence the implementation uses tensors.\n and it positive properties,\n- to be able to model function with separability (where variables do not interact like in additive kernels) and\n- non-stationarity.\n\nIn my interpretation, the thesis of the paper is that these properties are significant disdvantes and using a model that has these properties enables a performance improvement.\n\nThe disadvantage of the proposed model is that inference is no longer in closed form (a product of Gaussian random variables is not another Gaussian) hence an MCMC method is proposed to sample function values at points across the input space. While one could use a random discretization, (e.g. a latin hypercube, or a cluster around the current best point)  given the product structure of the surrogate model, there appear to be implementation benefits using tensor and matrix Kronecker products if the discretization is a fixed grid, discretize each dimension and build a a Cartesian product of each dimension to have a full grid.\n\nA range of synthetic and hyperparameter tuning benchmarks show the new model performing favourably with standard GP using SE-ARD kernel.\n\nSTRENGTHS:\n- in theory, I really like the idea of the model, in particular, that any matrix can be decomposed by SVD, hints that any function can be decomposed into a sum and product over functions of each dimension, i.e. the proposed model is, in theory, a universal approximator? Although intuitively, both BKTF and SE-ARD can model any smooth non-stationary surface, discontinuities and kinks are not modellable.\n\n- the inclusion of grid based GP methods is nice to see, and shows how much the grid decays performance compared to using the full unrestricted continuous space\n\nWEAKNESSES:\n# Technical\n- the proposed Cartesian discretization, $S_D$, scales exponentially with input dimension, and presumably contains _a lot_ of useless points in empty parts of the search space would a random discretization (LHC or Gaussian around current best $x$) be so much worse? Given a random set of points $X_D$, it is trivial to compute a the joint prior density $P[f(X_D)]$ density and the likelihood is just Gaussian $P[y_i|f(x_i)]$, sampling function values can be done with any off-the-shelf MCMC method.\n\n- I believe at least an additive GP should be a baseline. If non-stationary and non-separability are the main advantages of the BKTF model, presumably an additive GP with 2 kernels per dimension (matching CP rank=2 for BKTF) is an obvious baseline that has separability, such a baseline is exactly equation (7) but with sum-sum instead of sum-product. From this perspective, BKTF is simply an additive GP (that can only model separable variables) with a product over dimensions instead of a sum and this one change introduces a lot of engineering overhead (MCMC inference vs closed form inference) but also introduces more modelling power (separability can be modelled), given a high enough CP rank (CP rank =1 is just a product of 1D funs and is not separable).\n\n\n- the related work consists of two paragraphs, the first is discusses prior work on  BKTF (and feels a bit repetitive), the second focuses on stochastic process models. I feel the novelty of this paper is in using another surrogate model inside BO methods, and given the large body of BO work, there have been many works acknowledginbg the limitations of SE-ARD and proposing alternative models that are not cited or empirically compared to\n  - [Bayesian Neural networks](https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=bayesian+optimization+neural+networks&btnG=): \n    - Bayesian optimization with robust Bayesian neural networks, NeurIPS 2016\n    - Scalable bayesian optimization using deep neural networks, ICML 2015\n    - Multi-fidelity Bayesian optimization via deep neural networks: NeurIPS 2020\n  - [Deep GPs](https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=bayesian+optimization+deep+gaussian+&btnG=)\n    - Bayesian optimization using deep Gaussian processes, Arxiv\n\n# Presentation\n\n(in my personal subjective view) some changes to presentation would have made the paper far more accessible to me\n- can \"CANDECOMP/PARAFAC\" be simply described as a tensor generalization of SVD to make it easier for readers?\n- L119: \"we construct a D-dimensinoal Cartesian product Space\", can we just say \"grid\" like the authors do for the rest of the paper?\n- L81: kronecker product is introduced and never used again in the main paper\n- Section 3.1 would be much easier for me to understand if Eq (7) and (8) are introduced first, then next Section 3.3 (model inference) describes the grid and Equation (6) and MCMC details and the justification for the grid.\n- Section 3.2 is nice to mention but for me distracts from the main paper hence would be much better suited to the appendix.\n- L192: given a mean and uncertainty, this seems to be standard UCB, why is \"Bayesian-UCB\" defined?\n\nQUESTIONS:\n- the main body of the paper as it is makes the grid feel unnecessary in my view, the paper lacks a justification. Is the grid discretization _really_ required? L172 acknowledges a grid is not required, it is not _required_ for MCMC, the model structure, or for BO. It seems the grid is an implementation choice that does make nice use of the kronecker structure in the model but also introduces a exponential scaling limitations (12**6  = 3m points in the Hartmann experiment). Would the authors mind adding a \"memory used\" or \"time consumed\" column to Tables 2 and 3 top show practical implications of the discretization? Or add baseline with a randomized discretizations instead?\n\n- can the authors include 1D additive GPs as a baseline, even better adding two kernels (with independent hyperparameters) per dimension would be a very close model the BKTF with CP rank=2\n\n- at least one Bayesian neural network baseline model would compare this surrogate model with other well studied non-GP surrogate models, e.g. https://github.com/automl/RoBO, (though I am sure there are newer implementations)\n\nLIMITATIONS:\n- as above mentioned comment, discretizations in higher dimensions are generally considered bad practice, in particular, ungioded/naive grid discretizations that include many dead points.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new surrogate model called Bayesian Kernelized Tensor Factorization (BKTF) for Bayesian Optimization (BO).The BKTF model approximates the solid in the D-dimensional space using a fully Bayesian low-rank tensor CP decomposition. It uses Gaussian process (GP) priors on the latent basis functions for each dimension to capture local consistency and smoothness. This formulation allows sharing of information not only among neighboring samples but also across dimensions. The paper proposes using Markov chain Monte Carlo (MCMC) to efficiently approximate the posterior distribution. ). The paper demonstrates the effectiveness of BKTF through numerical experiments on standard BO test functions and machine learning hyperparameter tuning problems.\n\nSTRENGTHS:\n1. One of the significant strengths of the paper is the novel and reasonable solution of incorporating the idea of tensor decomposition into Bayesian Optimization (BO). This approach allows for a more efficient and effective representation of the D-dimensional Cartesian product space, enhancing the performance of BO. The adoption of tensor decomposition represents a significant advancement in the field and demonstrates the authors' innovative thinking.\n\n2. The usage of two-layer GPs is impressive. This approach is clever as it allows for the sharing of information among neighboring samples and across dimensions, enhancing the model's ability to capture local consistency and smoothness.\n\nWEAKNESSES:\n1. The cost of several cascaded full GPs may be high, especially for cases with large nodes (refer to \"coordinate\" in paper) at some dimension. More discussions are encouraged on the scalability analysis or the possible solutions, such as sparse GP,  to reduce the cost. \n \n2. As the tensor rank R  is always a crucial hyperparameter for tensor decomposition. I'm curious about how the rank setting could influence the BO. It will be great if the authors could give some comments or results on why R=2 is sufficient for the model setting.\n\nQUESTIONS:\nSee Weakness\n\nLIMITATIONS:\nSee Weakness", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a surrogate based on tensor decompositions for approximating complex functions, allowing for Bayesian-style maximization.\nNumerical experiments show the (slight) superiority of this model over classical Bayesian approaches. However, a limitation of this approach is the small dimensionality of the target functions and the need to use a discrete grid.\n\nSTRENGTHS:\n- The proposed algorithm uses a very small budget to find the maximum of complex functions (gradient-free, multimodal).\n\n- A good potential for expanding and improving the proposed algorithm.\n\n- This article uses a tensor approach for machine learning problems\n\n- The presented new algorithm, in my opinion, has quite a lot of possibilities for improvement, and the article itself is complete.\n\nWEAKNESSES:\n- A small number of numerical examples.\n\n- Final accuracy in Fig. 3b better, but very close to the accuracy of the other methods with which the comparison is made.\n\n- No comparison with non-Bayesian methods of finding the maximum.\n\nQUESTIONS:\nIn the line 96 of the text, a random noise $\\epsilon_i$ is added to the real values of the approximating function. Did you add the noise during numerical experiments to the model functions (Branin, Schaffer, Griewank, etc). If so, what was the variance of the noise?\n\nDid you try to run you algorithm on higher dimension (with less $m_d$ values), or more complex functions which need more budget for convergence?\n\nWhat is the characteristic running time of the proposed algorithm compared to the others mentioned in the article?\n\nLIMITATIONS:\nSmall dimension of functions for which the maximum is searched for. This is due to the fact that AF has to be found by unrolling the tensor from CP to the full format.\nThus, one of the main advantages of the CP tensor format, related to overcoming the curse of dimensionality, is not used.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}