Gaussian process methods for high dimensional learning
Tokyo Institute of Technology Taiji Suzuki
In this presentation, we discuss the statistical performances of some Gaussian process methods for highdimensional learning problems. The main part of the talk consists of two parts: the first part is about aBayes estimator with a Gaussian process prior for the sparse additive model and the second part is abouta Gaussian process estimator for high dimensional low rank tensor. Finally, we discuss a combination ofthese two learning methods.
(1) Sparse additive model. The sparse additive model is a useful nonparametric model for highdimensional regression. We suppose that the covariate is composed of M variables (x(1) , . . . , x(M ) ) (theycould be mutually dependent, actually, they could be copies of each other). Then the model assumes
samples Dn = {(xi , yi )}ni=1 are generated from the following model:
yi =M
∑∗
fm
(xi(m)) + ξi(i = 1, . . . , n),m=1∗
̸= 0} ofwhere {ξi }ni=1 is an i.i.d. sequence of nose. We assume that the nonzero elements I0 := {m | fm∗
f is small (sparse).
We can show that, by employing a Gaussian process prior with a sparsity inducing prior, the minimax
optimal convergence rate is attained by the Bayes estimator associated with the prior. The convergence
rate is characterized by the covering number of the unit ball of the Reproducing Kernel Hilbert Space(RKHS) corresponding to the Gaussian process prior [1].
(2) High dimensional low rank tensor estimator. Low rank tensor estimation is useful in several
practical applications such as recommendation system, image-movie processing, multi-task learning, andreduced rank regression. Basically, the method extract higher order relations among multi-modal data.
The model is described as follows. The true tensor A∗ ∈ RM1 ×···×MK is a tensor of degree K, and thesamples Dn = {(Yi , Xi )}ni=1 is generated by

Yi = ⟨A∗ , Xi ⟩ + ϵi
∑M1 ,...,MK ∗where Xi ∈ RM1 ×···×MK , ⟨A∗ , Xi ⟩ =i1 ,...,iK =1 Ai1 ,...,iK Xi,(i1 ,...,iK ) , and ϵi is a Gaussian noise with22mean 0 and variance σ (N (0, σ )). Here we assume that the true tensor A∗ is low rank in terms ofCP-rank.
To estimate A∗ , we employ a Gaussian process prior on the set of low rank tensors (more precisely, Gaussian chaos prior). That is, for the decomposition of a rank d′ tensor Ai1 ,...,iK = [[U (1) , U (2) , . . . , U (K) ]] =:∑d′
r=1 Ur,i1 Ur,i2 . . . Ur,iK , we put a prior defined as{}∑K(1)(K) ′d′(k) ⊤ (k)π(U , . . . , U|d ) ∝ exp − 2σ2 k=1 Tr[UU ] .
pMoreover, we put a prior on the rank d′ as π(d′ ) = exp(−ζd′ ) for arbitrary ζ > 0. Then, it is shown thatthe Bayes estimator corresponding to the prior achieves the minimax optimal rate up to a logarithmic
factor [2]. We will show that the Bayes estimator shows nice performances compared with existing convex
tensor estimators in numerical experiments.
(3) Finally, we combine these two notions. Suppose that the covariate forms the following composition x =∑r∏K(k)∗ (k,m)∗(x(k,m) )K,M) from the samples generated
m=1
k=1 fm (xk=1,m=1 . We estimate a function f (x) =asyi = f ∗ (xi ) + ϵi , (i = 1, . . . , n).
(k)This function can also be estimated by employing Gaussian process prior on each fm . The statistical
performance of the Bayes estimator is also derived by combining the techniques developed in (1) and (2).

References[1] T. Suzuki. Pac-bayesian bound for gaussian process regression and multiple kernel additive model. In Conference on Learning Theory (COLT2012), pages 8.1–8.20, 2012.
[2] T. Suzuki. Convergence rate of Bayesian tensor estimator: Optimal rate without restricted strong convexity.
In International Conference on Machine Learning (ICML2015), pages 1273–1282, 2015.