Title: Contextual Gaussian Process Bandits with Neural Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: contextual bandit

Gaussian process

neural network
Abstract: Contextual decision-making problems have witnessed extensive applications in various fields such as online content recommendation, personalized healthcare, and autonomous vehicles, where a core practical challenge is to select a suitable surrogate model for capturing unknown complicated reward functions. It is often the case that both high approximation accuracy and explicit uncertainty quantification are desired. In this work, we propose a neural network-accompanied Gaussian process (NN-AGP) model, which leverages neural networks to approximate the unknown and potentially complicated reward function regarding the contextual variable, and maintains a Gaussian process surrogate model with respect to the decision variable. Our model is shown to outperform existing approaches by offering better approximation accuracy thanks to the use of neural networks and possessing explicit uncertainty quantification from the Gaussian process. We also analyze the maximum information gain of the NN-AGP model and prove regret bounds for the corresponding algorithms. Moreover, we conduct experiments on both synthetic and practical problems, illustrating the effectiveness of our approach.

Contextual Gaussian Process Bandits with Neural Networks

Abstract

Contextual decision-making problems have witnessed extensive applications in various fields such as online content recommendation, personalized healthcare, and autonomous vehicles, where a core practical challenge is to select a suitable surrogate model for capturing unknown complicated reward functions. It is often the case that both high approximation accuracy and explicit uncertainty quantification are desired. In this work, we propose a neural network-accompanied Gaussian process (NN-AGP) model, which leverages neural networks to approximate the unknown and potentially complicated reward function regarding the contextual variable, and maintains a Gaussian process metamodel with respect to the decision variable. Our model is shown to outperform existing approaches by offering better approximation accuracy thanks to the use of neural networks and possessing explicit uncertainty quantification from the Gaussian process. We also analyze the maximum information gain of the NN-AGP model and prove regret bounds for the corresponding algorithms. Moreover, we conduct experiments on both synthetic and practical problems, illustrating the effectiveness of our approach.

1 Introduction

Various applications, including online content recommendation [1], healthcare [37, 15, 36], and autonomous vehicles [7], demand the sequential selection of a decision variable, conditional on the observed contextual variable representing the state of the environment in each round. These applications can generally be framed as contextual bandit problems [5, 59, 63, 2], especially when the reward function associated with each pair of decision and contextual variables is unknown. When both the decision and contextual variables are drawn from continuous sets, a significant challenge is selecting an appropriate surrogate model to approximate the reward function, considering both approximation accuracy and uncertainty quantification. A common approach to alleviate this issue is to employ a Gaussian process (GP) to model the reward function [61, 46], yielding the GP bandit method [80, 81]. Indeed, GP has proven to be an effective surrogate model to address the exploration-exploitation trade-off in estimating the unknown function while optimizing over it; see [88, 75, 90, 43, 51]. On the other hand, most of the existing GP bandit literature does not take the exogenous contextual variable into consideration, despite its critical role in capturing effects beyond the decision variable that influence the reward - effects that are integral to many of the applications previously mentioned [50, 62]. When the contextual variable is included in GP bandit problems, previous work largely adopts a GP to jointly model the reward function with both contextual and decision variables, employing a composite kernel that is either the sum or product of two separate kernels; see [57, 9].

While GP-based bandit methods have proven effective in various applications [3, 8, 92, 93, 86, 6], they may fall short in scenarios where the reward function exhibits intricate dependence on complex

contextual variables, for example, time-varying rewards [13, 31] and graph-structured contextual variables [71]. Specifically, it is a challenge to pre-define an appropriate composite kernel function for the joint GP, which is critical for the performance of the corresponding bandit algorithms, as documented by [21, 82]. Neural networks (NN), on the other hand, have been utilized elsewhere as surrogate models for the reward function in bandit problems [16, 54, 55], thanks to their flexibility and strong approximation power. However, they bring their own set of challenges. The 'black-box" nature of the neural network hinders explicit uncertainty quantification and complicates theoretical analysis of the associated algorithms. In particular, the acquisition functions guiding point selection in these algorithms necessitate an approximation of uncertainty [99, 54]. Although this approximation tends to be accurate when the NN's width is large, this can also lead to overparameterization.

Contribution. This paper proposes a neural network-accompanied Gaussian process (NN-AGP) model for solving contextual bandit problems, especially when the reward functions have intricate dependence on complex contextual variables. The proposed model is an inner product of a neural network and a multi-output GP, where the neural network captures the dependence of the reward function on the contextual variables, and the GP is employed to model the mapping from the decision variable to the reward. Our model generates a joint GP with both contextual and decision variables, which outperforms the existing GP-based bandit methods by specifying the data-driven kernel function through the lens of neural networks, thereby leading to an accurate approximation for the reward function. Moreover, compared with entirely relying on NN's, our model stands out due to the explicit GP expression with respect to the decision variable. This feature enables bandit algorithms with NN-AGP to be implemented efficiently with existing GP-based acquisition functions and provides a theoretical guarantee of the regret bounds. Our main contributions can be summarized as follows:

We propose an NN-AGP model and its upper confidence bound (UCB) algorithm for contextual bandit problems, referred to as NN-AGP-UCB. Our algorithm offers a datadriven procedure to specify the kernel functions of the joint GP, thereby achieving superior accuracy and model flexibility. We also prove the upper bound for both the maximum information gain of the NN-AGP model and the regret of the NN-AGP-UCB algorithm.

We conduct the experiments to evaluate our approach for complex reward functions, including those with time-varying dependence on sequenced and graph-structured contextual variables. Experimental results demonstrate the superiority of our approach over existing approaches that entirely rely on either GP or NN.

1.1 Related work

Since the seminal work by [80], the Gaussian process (GP) bandit problem has been extensively studied, where the bandit feedback is modeled as a GP regarding the decision variable (arms to be pulled). Some recent work includes [35, 32, 13, 12, 11, 18, 65]. In addition, GP bandits are also related to Bayesian optimization (BO) problems [38, 19, 94, 39, 34, 28, 85, 52, 23], where both problems consider optimizing black-box functions and therefore require surrogate models (GP in particular). When the number of decision variables is finite, the black-box optimization problem is also known as Ranking & Selection (R&S) [44, 73, 70, 4, 87], where GP models are widely employed as well; see [20, 58, 79, 64]. Our NN-AGP model can also be employed in (contextual) BO or R&S, but the discussion is beyond the scope of this work.

In this work, we specifically take the exogenous contextual variables into consideration. Previous work employs multiplicative and additive kernels to incorporate continuous context spaces into the scalar GP; see [57]. Other work considers safe contextual Bayesian optimization, employing a similar strategy to construct composite kernels; see [41, 9]. Another line of research explores distributionally robust BO [56, 83, 53], where the contextual variable distribution is selected from an ambiguity set. The methodology of representing the objective function using a joint GP has also been widely used in contextual policy search; see [72, 21, 40].

The connection between GP and NN has been explored in [60, 68], documenting that NN's with infinite width approach a GP model when the weight parameters are assigned with Gaussian priors. In addition, deep GP's have been proposed to enhance the model flexibility of neural networks where variational inference is employed; see [27, 26, 91]. GP models in which the parameters are represented by neural networks are studied in [98, 97].

2 Main procedure

We consider a problem of sequentially selecting a system's input variable (decision variable) for T (not necessarily known a priori) rounds. In each round, we receive a contextual variable θ t ∈ Θ ⊂ R d ′ from a set Θ , and select a decision variable x t ∈ X ⊂ R d from a set X of decisions. We then receive an observation

where f ( x ; θ ) is the reward (objective) function and ϵ t i.i.d. ∼ N (0 , σ 2 ϵ ) denotes the noise that is independent of both the contextual variables and decision variables. We consider the scenarios when both Θ and X are continuous and θ t 's are fully exogenous. That is, the selection of the decision variable in each round does not influence the future contextual variables. Although we focus on unconstrained problems in this work, our proposed model can be employed to approximate the constraints in optimization problems as well; see [9]. We also note that, for the description and discussion of our approach, the contextual variable is represented by a vector. However, we show through experiments in Section 4 that our approach is applicable to contextual variables with other structures.

Since f ( x , θ ) is unknown, we will not generally be able to choose the optimal action, and will thus incur regret r t = sup x ′ ∈X f ( x ′ , θ t ) -f ( x t , θ t ) , indicating the difference between the optimal reward and the reward we actually receive in each round. After T rounds, the cumulative regret is R T = ∑ T t =1 r t . Our goal is to develop an algorithm that achieves sub-linear contextual regret, i.e., R T /T → 0 for T →∞ , which requires a statistical model of the reward function with respect to both context variables and decision variables.

We specifically consider a reward function in the form

where g ( θ ) and p ( x ) are both m -dimensional vector-valued (unknown) functions, and m ∈ N is a user-selected quantity to indicate the complexity of the function. There are two main reasons for considering this reward function. First, this formulation is a generalization of linear contextual bandits, where the reward function is the inner product of the contextual variables and the unknown parameters. Here, we assume that the inner product is taken with two vector-valued functions with respect to decision variables and the contextual variable, which is similar in spirit to [24, 96, 42, 100]. Second, this reward function is consistent with the tensor-product approximation of a general function; see [29, 30, 47]. Therefore, further analysis on model mis-specification of the reward function can be supported by existing results of tensor-product approximation.

In this work, we specifically assume that g ( θ ) is a vector-valued deterministic mapping from R d ′ to R m , represented by a neural network with a given structure and some weight parameter W . In addition, p ( x ) is a multi-output Gaussian process (MGP) defined on X ⊂ R d . The MGP model is a generalization of the scalar-valued GP, where the output p ( x ) at each x is an m -dimensional vector. The MGP model captures not only the dependence between two outputs but also the dependence between different entries of each output. Thus, the covariance of an MGP p ( x ) is represented by a matrix-valued covariance function, denoted by K ( x , x ′ ) , and the vector of parameters involved in the MGP is denoted by Φ . We postpone the detailed description of the NN-AGP model to Section 3.1 and conclude our brief introduction of NN-AGP with an associated proposition, which follows easily from the fact that the normal distribution is preserved under linear transformations.

Proposition 1. The reward function f ( x ; θ ) is a scalar-valued mean-zero Gaussian process with respect to x and θ . The kernel function of this Gaussian process is

where K ( x , x ′ ) is the covariance of the MGP.

Next, we provide a bandit algorithm with the NN-AGP model, during which the surrogate model is sequentially learned from data. We name the algorithm neural network-accompanied Gaussian process upper confidence bound (NN-AGP-UCB). Suppose we are now in round t and observe the contextual variable θ t . In addition, we also have the historic data D t -1 = { ( θ 1 , x 1 , y 1 ) , ( θ 2 , x 2 , y 2 ) , . . . , ( θ t -1 , x t -1 , y t -1 ) } in hand. Denote by y t -1 = ( y 1 , y 2 , . . . , y t -1 )

the vector of observations. The selection of the next decision variable x t depends on the posterior distribution of the reward function, which is f ( x ; θ t ) | D t -1 ∼ N ( µ t -1 ( x ; θ t ) , σ 2 t -1 ( x ; θ t ) ) . Here

where ˜ K ( x ; θ t ) denotes the covariance vector between f ( x ; θ t ) and { f ( x τ ; θ τ ) } t -1 τ =1 . In addition, for the ( t -1) × ( t -1) -dimensional covariance matrix for historical data ˜ K D t -1 , the ( i, j ) -th entry is g ( θ i ) ⊤ K ( x i , x j ) g ( θ j ) as in Proposition 1 . The required parameters in (2), including the weight parameters W of the neural network g ( θ ) , the parameters Φ involved in the MGP p ( x ) and the variance σ 2 ϵ of the noise ϵ t , are all learned and updated with the observations through (5), which we will discuss in Section 3.1.

In terms of the acquisition function (the function that decides the decision variable in the following iteration), we employ the contextual Gaussian process-upper confidence bound (CGP-UCB) introduced in [57]. That is, we decide x t as

where µ t -1 ( x ; θ t ) and σ t -1 ( x ; θ t ) are the posterior mean and standard deviation of f ( x ; θ t ) as calculated in (2). The optimization problem (3) can be solved efficiently by global search heuristics, as suggested in [17]. In addition, β t is a user-selected hyper-parameter in each round, addressing the exploration-exploitation trade-off; see discussions in Section 3.2. The procedure for NN-AGP-UCB is summarized in Algorithm 1 . We note that other commonly selected acquisition functions for GP bandit problems or Bayesian optimization can be employed with NN-AGP as well, including knowledge gradient [76, 89, 33] and Thompson sampling [25, 74]. We postpone the discussion of these acquisition functions to the supplements.

Algorithm 1 NN-AGP-UCB

3 Statistical properties

3.1 Specification of NN-AGP

We describe the NN-AGP model here, which is employed as the surrogate for the reward function. Recall that the reward function with the pair of contextual and decision variables ( θ , x ) is f ( x ; θ ) = g ( θ ) ⊤ p ( x ) , where g ( θ ) is a vector-valued neural network (with weight parameters W ) from R d ′ to R m and p ( x ) is an m -dimensional output Gaussian process defined on X ⊂ R d . To be more specific, the m outputs p = ( p 1 , . . . , p m ) ⊤ are assumed to follow a multi-output Gaussian process (MGP) as

Here, K ( x , x ′ ) denotes the covariance matrix of p ( x ) and p ( x ′ ) , defined as K ( x , x ′ ) =    K 11 ( x , x ′ ) · · · K 1 m ( x , x ′ ) . . . . . . . . . K m 1 ( x , x ′ ) · · · K mm ( x , x ′ )    which is positive and semi-definite. The ( l, l ′ ) -th entry ′ ′

.

K ll ′ ( x , x ) represents the covariance (similarity) between outputs p l ( x ) and p l ′ ( x ) . The NN-AGP

model results in a scalar-valued Gaussian process with both the contextual and decision variables and therefore facilitates explicit acquisition functions and theoretical analysis. This GP representation arises from the linear structure between the MGP and the mapping g ( θ ) .

To specify the covariance matrix, we adopt the collaborative multi-output Gaussian process model [69] as a representative, which generalizes the commonly-used linear model of coregionalization (LMC) and the intrinsic coregionalization model (ICM); see [67]. That is, the MGP is determined by the linear transformation of multiple independent scalar-valued Gaussian processes as

Here, p l ( x ) is the l -th element of p ( x ) , Q is the number of involved GP's, { u q ( x ) } Q q =1 and { v l ( x ) } m l =1 are independent scalar-valued GP's, and the a l,q 's are coefficient parameters. In this way, the correlation between different entries in the MGP p ( x ) is captured by { u q ( x ) } Q q =1 through the a l,q 's. Morever, v l ( x ) represents specific independent features of p l ( x ) itself, for l = 1 , 2 , . . . , m . Suppose the kernel functions of the u q ( x ) 's and v l ( x ) 's are k q ( x , x ′ ) 's and ˜ k l ( x , x ′ ) 's and all these kernel functions are less than or equal to one, as is regularly assumed [80, 81, 57]. Then the matrix-valued kernel function of p ( x ) is K ( x , x ′ ) = ∑ Q q =1 A q k q ( x , x ′ ) + Diag { ˜ k 1 ( x , x ′ ) , . . . , ˜ k m ( x , x ′ ) } . Here, A q denotes the semi-definite matrix in which the ( l, l ′ ) -th entry is a l,q a l ′ ,q . In some applications, the parameters involved in the kernel functions k q ( x , x ′ ) and ˜ k l ( x , x ′ ) , and the coefficients a l,q 's are not known in advance. We denote these unknown parameters and coefficients in the MGP as Φ . In addition to the model (4), other types of MGP's can be employed in our methodology as well [14], while the selection of model (4) enables the theoretical analysis of our approach.

In terms of the mapping g ( θ ) , since we have no prior knowledge, we select the neural network as the surrogate model, due to 1) its strong approximation power for intricate dependence on complex variables; 2) its flexibility of adaptation to different application scenarios (e.g. time-series or graph-structured contextual variables) and 3) the availability of fruitful methods and tools for the training procedure. Given the data D t = { ( θ 1 , x 1 , y 1 ) , ( θ 2 , x 2 , y 2 ) , . . . , ( θ t , x t , y t ) } , the learning of unknown parameters in the MGP and the weight parameters in the neural network (as well as the noise variance) is through maximum likelihood estimation (MLE). That is

where the (normalized) likelihood function is L t = -ln [∣ ∣ ∣ ˜ K D t + σ 2 ϵ I t ∣ ∣ ∣ ] -y ⊤ t [ ˜ K D t + σ 2 ϵ I t ] -1 y t .

Here, y t = ( y 1 , y 2 , . . . , y t ) is the vector of observations and ˜ K D t is the covariance matrix of the data D t . The parameters W and Φ are contained in this covariance matrix. That is, instead of pre-defining a kernel function of the GP, the kernel function of NN-AGP is specified through learning the neural network from the data, yielding better approximation accuracy. We include a discussion of the consistency of training NN-AGP in the supplements.

3.2 Cumulative regret

Recall that the cumulative regret is defined as R T = ∑ T t =1 { sup x ′ ∈X f ( x ′ , θ t ) -f ( x t , θ t ) } . Here we provide an upper bound of R T with NN-AGP-UCB.

Theorem 1. Suppose δ ∈ (0 , 1) and the following.

The decision variable x ∈ X ⊆ [0 , r ] d and X is convex and compact. The contextual variable θ ∈ Θ ⊆ R d ′ and Θ is convex and compact; g ( θ ) is a known continuous mapping of θ ∈ Θ ; p ( x ) is sampled from a known MGP prior as in (4) and the variance of the noise σ 2 ϵ is known. That is, these parameters do not need learning and updating from data.

For the components of the MGP, there exist constants { a q } Q q =1 , { b q } Q q =1 , { ˜ a l } m l =1 , { ˜ b l } m l =1 satisfying

∀ L q , ˜ L l > 0 and ∀ j = 1 , 2 , . . . , d , q = 1 , 2 , . . . , Q and l = 1 , 2 , . . . , m .

We choose as a hyper-parameter in (3)

where d and r are the dimension and the upper bound of the decision variable, a = ∑ Q q =1 a q + ∑ m l =1 ˜ a l , b = ∑ Q q =1 b q + ∑ m l =1 ˜ b l , ˜ M = sup θ ∈ Θ { {| ∑ m l =1 g l ( θ ) a l,q |} Q q =1 , {| g l ( θ ) |} m l =1 } , where g l denotes the l -th entry of g ( θ ) .

Then the cumulative regret is bounded with high probability as

Here C = (( ∑ Q q =1 ∑ m l =1 a 2 l,q ) +1 ) sup θ ∈ Θ ∥ g ( θ ) ∥ 2 2 . In addition, γ T is the maximum information gain associated with the NN-AGP f ( x ; θ ) , defined by (7).

We postpone the discussion of the maximum information gain γ T to Section 3.3, with some specific kernels employed in the MGP component of the NN-AGP model. We note that GP's with commonlyselected kernel functions, including the Matérn kernel and the radial basis function kernel, satisfy the condition (6) and further discussions can be found in Theorem 5 in [45]. A detailed proof of Theorem 1 is contained in the supplements. Note that Theorem 1 assumes that g ( θ ) is exactly known so does not consider the error of approximating g ( θ ) with the neural networks. In the supplements, we include a detailed discussion of the algorithm that considers the neural network approximation error, as well as the corresponding regret bounds.

At the end of this section, we compare our regret bound with existing work. Specifically, NN-AGPUCB has the same bound of ˜ O ( √ Tγ T ) as CGP-UCB, but is superior when the contextual variable dimension is high; see details in the supplements. In terms of the algorithms which entirely rely on NN, we note that NeuralUCB [99], Neural TS [98], and Neural LinUCB [95] all consider the scenarios when the decision variable x is selected from a finite set. In comparison, we consider that x is selected from a continuous set. When performed on a finite feasible set X , our NN-AGP-UCB also has a regret bound of ˜ O ( √ Tγ T ) , where the maximum information gain γ T further depends on the kernel function of the GP component used in NN-AGP. When the kernel function has an exponential eigendecay (see Definition 1), NN-AGP-UCB has a regret bound of ˜ O ( √ T ) , matching the regret bound of NeuralUCB, Neural TS and Neural LinUCB as well.

3.3 Maximum information gain

In this section, we discuss the information gain γ T of the proposed NN-AGP model. The maximum information gain is defined as

where f T is the reward function evaluated at { ( θ t , x t ) } T t =1 ; y T denotes the observations; I ( y T ; f T ) = H ( y T ) -H ( y T | f T ) is the mutual information between y T and f T ; H ( · ) = E [ -log p ( · )] is the entropy of a random element, where p is the probability density function.

For ease of notation, here we consider the scenario when Q = 1 and there is no { v l ( x ) } in the MGP defined in (4), i.e., p l ( x ) = a l u ( x ) . We also impose more variability so that A = A 1 is a semi-definite positive matrix and not necessarily a rank-one matrix, analogous to [14]. We provide a more general discussion of the maximum information gain of the NN-AGP in the supplements. We first provide a proposition on the kernel function.

Proposition 2. When Q = 1 and there is no { v l ( x ) } in the MGP, the kernel function of the NNAGP is a product of two kernel functions. That is ˜ K (( x , θ ) , ( x ′ , θ ′ )) = ˜ k ( θ , θ ′ ) k ( x , x ′ ) , where ˜ k ( θ , θ ′ ) = g ( θ ) ⊤ A g ( θ ) is a finite rank kernel with respect to θ . Furthermore, suppose that both Θ and X are compact, g ( θ ) is a continuous mapping and k ( x , x ′ ) is a semi-definite kernel function. Then the kernel function ˜ K (( x , θ ) , ( x ′ , θ ′ )) possesses a Mercer decomposition:

Here, { ( µ i , ϕ i ) } m i =1 and { ( λ j , ψ j ) } m j =1 are the Mercer decompositions for ˜ k ( θ , θ ′ ) and k ( x , x ′ ) . That is, ˜ k ( θ , θ ′ ) = ∑ m i =1 µ i ϕ i ( θ ) ϕ i ( θ ′ ) , k ( x , x ′ ) = ∑ ∞ j =1 λ j ψ j ( x ) ψ j ( x ′ ) , where the eigenvalues are µ 1 ⩾ µ 2 ⩾ . . . ⩾ µ m ⩾ 0 and λ 1 ⩾ λ 2 ⩾ . . . ⩾ 0 .

With the Mercer decomposition of the NN-AGP kernel function, we provide the bound of the maximum information gain. Specifically, we consider two scenarios for the employed kernel in the MGP, analogous to [22, 84].

Definition 1. Consider the eigenvalues { λ j } ∞ j =1 of the kernel function k ( x , x ′ ) in decreasing order.

For some C p > 0 , α p > 1 , k is said to have a ( C p , α p ) polynomial eigendecay, if for all j ∈ N , we have λ j ⩽ C p j -α p . An example is the Matérn kernel.

For some C e, 1 , C e, 2 , α e > 0 , k is said to have a ( C e, 1 , C e, 2 , α e ) exponential eigendecay, if for all j ∈ N , we have λ j ⩽ C e, 1 exp( -C e, 2 j α e ) . An example is the radial basis function kernel.

Theorem 2. Suppose that 1) ˜ K (( x , θ ) , ( x ′ , θ ′ )) satisfies the conditions in Proposition 2 ; 2) ∀ x , x ′ ∈ X , | k ( x , x ′ ) | ⩽ ¯ k for some ¯ k > 0 and 3) ∀ j ∈ N , ∀ x ∈ X , | ψ j ( x ) | ⩽ ψ , for some ψ > 0 . If k ( x , x ′ ) has a ( C p , α p ) polynomial eigendecay, then

If k ( x , x ′ ) has a ( C e, 1 , C e, 2 , α e ) exponential eigendecay, then

where

Here, ¯ µ = 1 m ∑ m i =1 µ i denotes the mean of the eigenvalues of the kernel function ˜ k ( θ , θ ′ ) ; ¯ ˜ k = sup θ , θ ′ ∈ Θ ∣ ∣ ∣ ˜ k ( θ , θ ′ ) ∣ ∣ ∣ and ϕ = sup θ ∈ Θ | ϕ ( θ ) | . Moreover, the maximum information gain of the NN-AGP is O ( mγ x ; T ) , where γ x ; T is the maximum information of k ( x , x ′ ) .

A more detailed discussion of the Mercer decomposition of NN-AGP is contained in the supplements, as well as the proofs of Proposition 2 and Theorem 2 . At the end of this section, we compare our results on maximum information gain with [57]. For the composite kernel that is a product of two kernels, the upper bound is ( d + d ′ ) γ x ; T + ( d + d ′ ) log T . Here, γ x ; T is the maximum information gain of the kernel function with the decision variable, and d ′ and d are the dimensions of the contextual variable and the decision variable. That is, the information gain (as well as the cumulative regret) increases as the dimension of the contextual variable increases. In comparison, the maximum information in Theorem 2 does not depend on d ′ . Thus, the NN-AGP model has lower cumulative regret than the classical strategy in [57] when the dimension of the contextual variable is relatively high.

4 Experiments

In this section, we conduct experiments to show the practicality of our neural network-accompanied Gaussian process upper bound confidence (NN-AGP-UCB) approach. We apply different neural networks to different problems, including the fully-connected neural network (FCN) [77] to a synthetic reward function, the long short-term memory (LSTM) [48] neural network to a queuing problem with time sequence contextual variables, and the graph convolutional neural network (GCN) [78] to i.i.d.

a pricing problem with diffusion networks. We add a noise ϵ t ∼ N (0 , 0 . 01) to the ground-truth value of the reward in the first set of experiments. For the latter two sets of experiments, the reward is generated by stochastic simulation (and therefore is 'black-box' with contextual/decision variables) and we postpone the description of the full dynamic to the supplements. We also provide additional experiments in the supplements, including 1) sensitivity on the structure of reward functions; 2) comparison with contextual variables possessing different dimensions; 3) regression tasks on complex functions and 4) finite decision/contextual variables with real data.

The baseline approaches includes CGP-UCB [57], NeuralUCB [99] and NN-UCB [54]. The experiment results provided are mean performances based on repeating the experiments 15 times. Standard deviations (represented by a shadow associated with the mean-value line) are also included. In each iteration, the exogenous contextual variable θ t is randomly selected from Θ with equal probability. In terms of the initialization, we randomly select decision variables independently of observed contextual variables for each approach in the first 20 iterations to attain surrogates. The specific description of the employed surrogate model in each approach is postponed to the supplements.

4.1 Synthetic reward function

In this section, we consider two synthetic reward functions

For NN-AGP-UCB, we consider m = 2 , 3 , 5 to study the effects of model selection on the algorithm performance. For CGP-UCB, we consider both additive kernels and multiplicative kernels. Because both NeuralUCB and NN-UCB are designed for contextual bandits with finite arms, we adapt them to the problem we consider in Section 2 and postpone the details to the supplements. The experimental results of the average regret R T /T are illustrated in Figure 1 and Figure 2 , which provide the following insights. 1. Comparison with baseline approaches. For both reward functions, NN-AGPUCB outperforms both the CGP-UCB and NN-based approaches. The advantage comes from 1) strong approximation power regarding θ due to NN and 2) explicit inference regarding x due to GP. 2. Effects of the dimension m . Generally, when m increases, the model flexibility increases, and the regret might be smaller, although this improvement might not be significant in some scenarios. Furthermore, a relatively large m might result in overparameterization especially when there are not enough iterations. A suggested selection of m is ⌈ d/ 3 + d ′ / 10 ⌉ +3 considering both the algorithm performance and training complexity of models. 3. Breaking the linear assumption. Recall that we assume the reward function is of the form of R ( x ; θ ) = g ( θ ) ⊤ p ( x ) . However, the reward functions here break this linear assumption and yet our NN-AGP-UCB is still applicable and outperforms the baseline approaches. Moreover, additional experiments show that NN-AGP 1) is not sensitive on the structure of the reward functions; 2) has a greater advantage with higher-dimensional contextual variables and 3) achieves better approximation accuracy for complex functions, compared with a joint GP with composite kernels; see the supplements for details. In addition, when the dimension of the input increases, the uncertainty of the regret will increase as well; see Figures 1 & 2 for comparison. We also include the recorded computational time of these bandit algorithms in the supplements.

4.2 Queuing problem with time sequence contextual variables

In this section, we show through experiments that the NN-AGP model is applicable to contextual GP bandits when the objective function depends on the sequence of the contextual variables. That is, the reward function at time t can be approximated by

3.8

NN-AGP-UCB (m=2)

3.6

NN-AGP-UCB (m=3)

NN-AGP-UCB (m=5)

3.4

CGP-UCB(with additive kernel)

AverageRegrets

3.2

CGP-UCB(with multiplicative kernel)

NeuralUCB

NN-UCB

3.0

2.8

2.6

2.4

0

50

100

150

200

250

300

Rounds

1.2

NN-AGP-UCB (m=2)

NN-AGP-UCB (m=3)

NN-AGP-UCB (m=5)

10

CGP-UCB (with additive kernel)

CGP-UCB (with multiplicative kernel)

Average Regrets

8

NeuralUCB

NN-UCB

9

4

2

0

0

200

400

600

800

1000

1200

Rounds

Here, g t ( θ 1 , θ 2 , . . . , θ t ) is modeled by an LSTM neural network. We consider a discrete-time queuing problem. In each time epoch, a contextual variable is first revealed. For example, the contextual variable might includes traffic and weather conditions that affect the arrival process of the queuing system. The number of customers arriving at this epoch depends on the entire sequence of the revealed contextual variables up to now. The agent decides the service rate of the server and the service price for customers (decision variables). The reward function (might be negative) is the expected net income (the income brought by serving customers minus the service cost and penalty for losing customers). We let X = [0 , 5] 2 and sample θ t from multivariate normal distributions. We present the cumulative rewards in Figure 3 and Figure 4 of NN-AGP-UCB with LSTM. The baseline CGP-UCB adopts both additive and multiplicative kernels with the current contextual variable. We also apply kernel functions specifically designed for time series [10] to construct the composite kernel. Experimental results indicate that 1) employing a specific time series kernel enhances the performance of CGP-UCB and 2) NN-AGP-UCB with LSTM outperforms the classical CGP-UCB approaches with different composite kernels.

3000

NN-AGP-UCB (m=2)

NN-AGP-UCB (m=3)

Rewards

2500

2000

NN-AGP-UCB (m=5)

CGP-UCB (with additivekernel)

CGP-UCB (with multiplicative kernel)

CGP-UCB (with additive time-serialkernel)

Accumulative

1500

1000

500

0

0

50

100

150

200

250

300

Rounds

4.3 Pricing with diffusion network

In this section, we show the NN-AGP model is applicable to contextual GP bandits with graphstructured contextual variables. That is, the contextual variable is summarized by a network structure

where V t denotes the set of nodes and E t denotes the set of directed/undirected edges of a network. To approximate g ( θ ) with a graph-structured contextual variable, we apply the GCN model. We consider a pricing problem with a diffusion network, where each node represents a user who decides

12000

NN-AGP-UCB (m=2)

NN-AGP-UCB (m=3)

ewards

10000

8000

NN-AGP-UCB (m=5)

CGP-UCB(withadditivekernel)

CGP-UCB (with multiplicative kernel)

CGP-UCB(with additive time-serialkernel)

ative

6000

4000

2000

0

0

100

200

Rounds

300

400

500

600

to adopt a service or not and the edge between two nodes indicates whether the choices of these two users influence each other. In each iteration, the network structure θ t is first presented, and then the agent decides the price of the service. The reward function is the expected income for the service adoption from the users. The detailed description can be found in [66]. We let X = [0 , 30] and θ t represents an undirected graph with 5 and 10 nodes where each edge exists with probability 1/2. We present the cumulative rewards in Figure 5 and Figure 6 of NN-AGP-UCB with GCN. The baseline CGP-UCB adopts both additive and multiplicative kernels and in terms of the contextual variable, we consider 1) vectorizing the adjacency matrix that summarizes the network structure and 2) applying kernel functions that are specifically designed for graphs [49]. Experimental results indicate that NN-AGP-UCB with GCN outperforms the classical CGP-UCB approach adopting different kernels, and the advantages become greater for networks with more nodes.

NN-AGP-UCB (with GCN)

600

CGP-UCB(graph,multiplicative)

CGP-UCB (graph,additve)

Rewards

500

CGP-UCB (vector,additve)

CGP-UCB(vector,multiplicative)

400

R

lative

300

200

100

0

0

25

50

75

100

125

150

175

200

Rounds

5 Conclusion & impact

We propose a neural network accompanied Gaussian process (NN-AGP) model to address contextual GP bandit problems. The advantages of our approach include 1) flexibility of employing different neural networks appropriate for applications with diverse structures of contextual information; 2) approximation accuracy for the reward function and better performance on cumulative rewards/regrets; 3) tractability of a GP representation regarding the decision variable, thus supporting explicit uncertainty quantification and theoretical analysis. Our approach has potential application to healthcare, where doctors need to develop therapy plans based on patient information to achieve optimal treatment effects. When complex and sparse genetic information is employed, it necessitates the use of neural networks. Another potential application is for Automated Guided Vehicles (AGVs) to enhance workplace safety and reduce carbon emissions, where environmental information is provided to the AGV, and the AGV takes actions accordingly.

In terms of limitations, since NN-AGP retains a GP structure, it suffers from computational complexity with large data sets. To alleviate the computational burden, we consider sparse NN-AGP for future work; see a discussion in the supplements. In addition, incorporating NN into bandit problems generally requires sufficient data to approximate the unknown reward function, thereby bringing the cold-start issue to NN-AGP. To address the challenge, we also include a discussion on employing transfer learning technologies in the supplements. Other potential future work includes 1) adapting NN-AGP to multi-objective/constrained optimization problems and 2) employing NN-AGP in a federated contextual bandit problem with multiple decentralized users.

References

N. Abe, A. W. Biermann, and P. M. Long. Reinforcement learning with immediate rewards and linear hypotheses. Algorithmica , 37(4):263-293, 2003.

S. Agrawal and N. Goyal. Thompson sampling for contextual bandits with linear payoffs. In International conference on machine learning , pages 127-135. PMLR, 2013.

B. Ankenman, B. L. Nelson, and J. Staum. Stochastic kriging for simulation metamodeling. In 2008 Winter simulation conference , pages 362-370. IEEE, 2008.

E. A. Applegate, G. Feldman, S. R. Hunter, and R. Pasupathy. Multi-objective ranking and selection: Optimal sampling laws and tractable approximations via score. Journal of Simulation , 14(1):21-40, 2020.

P. Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research , 3(Nov):397-422, 2002.

Y. Bai, H. Lam, T. Balch, and S. Vyetrenko. Efficient calibration of multi-agent simulation models from output series with bayesian optimization. In Proceedings of the Third ACM International Conference on AI in Finance , pages 437-445, 2022.

R. Bajrachrya and H. Jung. Contextual bandits approach for selecting the best channel in industry 4.0 network. In 2021 International Conference on Information Networking (ICOIN) , pages 13-16. IEEE, 2021.

R. R. Barton, B. L. Nelson, and W. Xie. Quantifying input uncertainty via simulation confidence intervals. INFORMS journal on computing , 26(1):74-87, 2014.

F. Berkenkamp, A. Krause, and A. P. Schoellig. Bayesian optimization with safety constraints: safe and automatic parameter tuning in robotics. Machine Learning , pages 1-35, 2021.

C. Bock, M. Togninalli, E. Ghisu, T. Gumbsch, B. Rieck, and K. Borgwardt. A wasserstein subsequence kernel for time series. In 2019 IEEE International Conference on Data Mining (ICDM) , pages 964-969. IEEE, 2019.

I. Bogunovic and A. Krause. Misspecified gaussian process bandit optimization. Advances in Neural Information Processing Systems , 34:3004-3015, 2021.

I. Bogunovic, A. Krause, and J. Scarlett. Corruption-tolerant gaussian process bandit optimization. In International Conference on Artificial Intelligence and Statistics , pages 1071-1081. PMLR, 2020.

I. Bogunovic, J. Scarlett, and V. Cevher. Time-varying gaussian process bandit optimization. In Artificial Intelligence and Statistics , pages 314-323. PMLR, 2016.

E. V. Bonilla, K. Chai, and C. Williams. Multi-task gaussian process prediction. Advances in neural information processing systems , 20, 2007.

D. Bouneffouf, I. Rish, and C. Aggarwal. Survey on applications of multi-armed and contextual bandits. In 2020 IEEE Congress on Evolutionary Computation (CEC) , pages 1-8. IEEE, 2020.

J. Bowden, J. Song, Y. Chen, Y. Yue, and T. Desautels. Deep kernel bayesian optimization. Technical report, Lawrence Livermore National Lab.(LLNL), Livermore, CA (United States), 2021.

E. Brochu, V. M. Cora, and N. De Freitas. A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv:1012.2599 , 2010.

X. Cai and J. Scarlett. On lower bounds for standard and robust gaussian process bandit optimization. In International Conference on Machine Learning , pages 1216-1226. PMLR, 2021.

S. Cakmak, R. Astudillo Marban, P. Frazier, and E. Zhou. Bayesian optimization of risk measures. Advances in Neural Information Processing Systems , 33:20130-20141, 2020.

S. Cakmak, E. Zhou, and S. Gao. Contextual ranking and selection with gaussian processes. In 2021 Winter Simulation Conference (WSC) , pages 1-12. IEEE, 2021.

I. Char, Y. Chung, W. Neiswanger, K. Kandasamy, A. O. Nelson, M. Boyer, E. Kolemen, and J. Schneider. Offline contextual bayesian optimization. Advances in Neural Information Processing Systems , 32, 2019.

N. Chatterji, A. Pacchiano, and P. Bartlett. Online learning with kernel losses. In International Conference on Machine Learning , pages 971-980. PMLR, 2019.

H. Chen and H. Lam. Pseudo-bayesian optimization. arXiv preprint arXiv:2310.09766 , 2023.

V. Chernozhukov, M. Demirer, G. Lewis, and V. Syrgkanis. Semi-parametric efficient policy learning with continuous actions. Advances in Neural Information Processing Systems , 32, 2019.

S. R. Chowdhury and A. Gopalan. On kernelized multi-armed bandits. In International Conference on Machine Learning , pages 844-853. PMLR, 2017.

K. Cutajar, M. Pullin, A. Damianou, N. Lawrence, and J. González. Deep gaussian processes for multi-fidelity modeling. arXiv preprint arXiv:1903.07320 , 2019.

A. Damianou and N. D. Lawrence. Deep gaussian processes. In Artificial intelligence and statistics , pages 207-215. PMLR, 2013.

S. Daulton, S. Cakmak, M. Balandat, M. A. Osborne, E. Zhou, and E. Bakshy. Robust multi-objective bayesian optimization under input noise. arXiv preprint arXiv:2202.07549 , 2022.

L. De, B. De-Moor, and J. Vandewalle. On the best rank-1 and rank-(r1 r2. . . rn) approximation of higher-order tensors. SIAM journal on Matrix Analysis and Applications , 21(4):1324-1342, 2000.

L. De Lathauwer, B. De Moor, and J. Vandewalle. Computation of the canonical decomposition by means of a simultaneous generalized schur decomposition. SIAM journal on Matrix Analysis and Applications , 26(2):295-327, 2004.

Y. Deng, X. Zhou, B. Kim, A. Tewari, A. Gupta, and N. Shroff. Weighted gaussian process bandits for non-stationary environments. In International Conference on Artificial Intelligence and Statistics , pages 6909-6932. PMLR, 2022.

T. Desautels, A. Krause, and J. W. Burdick. Parallelizing exploration-exploitation tradeoffs in gaussian process bandit optimization. Journal of Machine Learning Research , 15:3873-3923, 2014.

L. Ding, L. J. Hong, H. Shen, and X. Zhang. Knowledge gradient for selection with covariates: Consistency and computation. Naval Research Logistics (NRL) , 69(3):496-507, 2022.

L. Ding, R. Tuo, and X. Zhang. High-dimensional simulation optimization via brownian fields and sparse grids. arXiv preprint arXiv:2107.08595 , 2021.

J. Djolonga, A. Krause, and V. Cevher. High-dimensional gaussian process bandits. Advances in neural information processing systems , 26, 2013.

J. Du, S. Gao, and C.-H. Chen. A contextual ranking and selection method for personalized medicine. Manufacturing & Service Operations Management , 2023.

A. Durand, C. Achilleos, D. Iacovides, K. Strati, G. D. Mitsis, and J. Pineau. Contextual bandits for adapting treatment in a mouse model of de novo carcinogenesis. In Machine learning for healthcare conference , pages 67-82. PMLR, 2018.

D. Eriksson, M. Pearce, J. Gardner, R. D. Turner, and M. Poloczek. Scalable global optimization via local bayesian optimization. Advances in neural information processing systems , 32, 2019.

D. Eriksson and M. Poloczek. Scalable constrained bayesian optimization. In International Conference on Artificial Intelligence and Statistics , pages 730-738. PMLR, 2021.

T. Fauvel and M. Chalk. Contextual bayesian optimization with binary outputs. arXiv preprint arXiv:2111.03447 , 2021.

M. Fiducioso, S. Curi, B. Schumacher, M. Gwerder, and A. Krause. Safe contextual bayesian optimization for sustainable room temperature pid control tuning. arXiv preprint arXiv:1906.12086 , 2019.

D. J. Foster, C. Gentile, M. Mohri, and J. Zimmert. Adapting to misspecification in contextual bandits. Advances in Neural Information Processing Systems , 33:11478-11489, 2020.

P. I. Frazier. A tutorial on bayesian optimization. arXiv preprint arXiv:1807.02811 , 2018.

M. C. Fu, J.-Q. Hu, C.-H. Chen, and X. Xiong. Simulation allocation for determining the best design in the presence of correlated sampling. INFORMS Journal on Computing , 19(1):101111, 2007.

S. Ghosal and A. Roy. Posterior consistency of gaussian process prior for nonparametric binary regression. The Annals of Statistics , 34(5):2413-2429, 2006.

R. Guhaniyogi, C. Li, T. D. Savitsky, and S. Srivastava. Distributed bayesian varying coefficient modeling using a gaussian process prior. The Journal of Machine Learning Research , 23(1):3642-3700, 2022.

W. Hackbusch and B. N. Khoromskij. Tensor-product approximation to operators and functions in high dimensions. Journal of Complexity , 23(4-6):697-714, 2007.

S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation , 9(8):17351780, 1997.

S. Højsgaard and S. L. Lauritzen. Graphical gaussian models with edge and vertex symmetries. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 70(5):1005-1027, 2008.

L. J. Hong and G. Jiang. Offline simulation online application: A new framework of simulationbased decision making. Asia-Pacific Journal of Operational Research , 36(06):1940015, 2019.

L. J. Hong and X. Zhang. Surrogate-based simulation optimization. In Tutorials in Operations Research: Emerging Optimization Methods and Modeling Techniques with Applications , pages 287-311. INFORMS, 2021.

S. Hu, H. Wang, Z. Dai, B. K. H. Low, and S. H. Ng. Adjusted expected improvement for cumulative regret minimization in noisy bayesian optimization. arXiv preprint arXiv:2205.04901 , 2022.

H. Husain, V. Nguyen, and A. van den Hengel. Distributionally robust bayesian optimization with ϕ -divergences. arXiv preprint arXiv:2203.02128 , 2022.

P. Kassraie and A. Krause. Neural contextual bandits without regret. In International Conference on Artificial Intelligence and Statistics , pages 240-278. PMLR, 2022.

P. Kassraie, A. Krause, and I. Bogunovic. Graph neural network bandits. arXiv preprint arXiv:2207.06456 , 2022.

J. Kirschner, I. Bogunovic, S. Jegelka, and A. Krause. Distributionally robust bayesian optimization. In International Conference on Artificial Intelligence and Statistics , pages 2174-2184. PMLR, 2020.

A. Krause and C. Ong. Contextual gaussian process bandit optimization. Advances in neural information processing systems , 24, 2011.

P. L. Salemi, E. Song, B. L. Nelson, and J. Staum. Gaussian markov random fields for discrete optimization via simulation: Framework and algorithms. Operations Research , 67(1):250-266, 2019.

J. Langford and T. Zhang. The epoch-greedy algorithm for contextual multi-armed bandits. Advances in neural information processing systems , 20(1):96-1, 2007.

J. Lee, Y. Bahri, R. Novak, S. S. Schoenholz, J. Pennington, and J. Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint arXiv:1711.00165 , 2017.

C. Li. Bayesian fixed-domain asymptotics for covariance parameters in a gaussian process model. The Annals of Statistics , 50(6):3334-3363, 2022.

C. Li, S. Gao, and J. Du. Convergence analysis of stochastic kriging-assisted simulation with random covariates. INFORMS Journal on Computing , 35(2):386-402, 2023.

L. Li, W. Chu, J. Langford, and R. E. Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web , pages 661-670, 2010.

Y. Li and S. Gao. On the finite-time performance of the knowledge gradient algorithm. In International Conference on Machine Learning , pages 12741-12764. PMLR, 2022.

Z. Li and J. Scarlett. Gaussian process bandit optimization with few batches. In International Conference on Artificial Intelligence and Statistics , pages 92-107. PMLR, 2022.

Y. Lin, H. Zhang, R. Zhang, and Z.-J. M. Shen. Nonprogressive diffusion on social networks: Approximation and applications. Available at SSRN , 2022.

H. Liu, J. Cai, and Y.-S. Ong. Remarks on multi-output gaussian process regression. Knowledge-Based Systems , 144:102-121, 2018.

A. G. d. G. Matthews, M. Rowland, J. Hron, R. E. Turner, and Z. Ghahramani. Gaussian process behaviour in wide deep neural networks. arXiv preprint arXiv:1804.11271 , 2018.

T. V. Nguyen, E. V. Bonilla, et al. Collaborative multi-output gaussian processes. In UAI , pages 643-652. Citeseer, 2014.

E. C. Ni, D. F. Ciocan, S. G. Henderson, and S. R. Hunter. Efficient ranking and selection in parallel computing environments. Operations Research , 65(3):821-836, 2017.

F. Opolka, Y.-C. Zhi, P. Liò, and X. Dong. Adaptive gaussian processes on graphs via spectral graph wavelets. In International Conference on Artificial Intelligence and Statistics , pages 4818-4834. PMLR, 2022.

M. Pearce and J. Branke. Continuous multi-task bayesian optimisation with correlation. European Journal of Operational Research , 270(3):1074-1085, 2018.

Y. Peng, C.-H. Chen, M. C. Fu, and J.-Q. Hu. Efficient simulation resource sharing and allocation for selecting the best. IEEE Transactions on Automatic Control , 58(4):1017-1023, 2012.

D. J. Russo, B. Van Roy, A. Kazerouni, I. Osband, Z. Wen, et al. A tutorial on thompson sampling. Foundations and Trends® in Machine Learning , 11(1):1-96, 2018.

I. O. Ryzhov. On the convergence rates of expected improvement methods. Operations Research , 64(6):1515-1528, 2016.

I. O. Ryzhov, W. B. Powell, and P. I. Frazier. The knowledge gradient algorithm for a general class of online learning problems. Operations Research , 60(1):180-195, 2012.

T. N. Sainath, O. Vinyals, A. Senior, and H. Sak. Convolutional, long short-term memory, fully connected deep neural networks. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP) , pages 4580-4584. IEEE, 2015.

F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini. The graph neural network model. IEEE transactions on neural networks , 20(1):61-80, 2008.

M. Semelhago, B. L. Nelson, E. Song, and A. Wächter. Rapid discrete optimization via simulation with gaussian markov random fields. INFORMS Journal on Computing , 33(3):915930, 2021.

N. Srinivas, A. Krause, S. Kakade, and M. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In Proceedings of the 27th International Conference on Machine Learning . Omnipress, 2010.

N. Srinivas, A. Krause, S. M. Kakade, and M. W. Seeger. Information-theoretic regret bounds for gaussian process optimization in the bandit setting. IEEE transactions on information theory , 58(5):3250-3265, 2012.

W. T. Stephenson, S. Ghosh, T. D. Nguyen, M. Yurochkin, S. Deshpande, and T. Broderick. Measuring the robustness of gaussian processes to kernel choice. In International Conference on Artificial Intelligence and Statistics , pages 3308-3331. PMLR, 2022.

S. S. Tay, C. S. Foo, U. Daisuke, R. Leong, and B. K. H. Low. Efficient distributionally robust bayesian optimization with worst-case sensitivity. In International Conference on Machine Learning , pages 21180-21204. PMLR, 2022.

S. Vakili, K. Khezeli, and V. Picheny. On information gain and regret bounds in gaussian process bandits. In International Conference on Artificial Intelligence and Statistics , pages 82-90. PMLR, 2021.

S. Wang, S. H. Ng, and W. B. Haskell. A multilevel simulation optimization approach for quantile functions. INFORMS Journal on Computing , 34(1):569-585, 2022.

W. Wang and X. Chen. An adaptive two-stage dual metamodeling approach for stochastic simulation experiments. IISE Transactions , 50(9):820-836, 2018.

W. Wang, H. Wan, and X. Chen. Bonferroni-free and indifference-zone-flexible sequential elimination procedures for ranking and selection. Operations Research , 2023.

C. K. Williams and C. E. Rasmussen. Gaussian processes for machine learning , volume 2. MIT press Cambridge, MA, 2006.

J. Wu and P. Frazier. The parallel knowledge gradient method for batch bayesian optimization. Advances in neural information processing systems , 29, 2016.

J. Wu, M. Poloczek, A. G. Wilson, and P. Frazier. Bayesian optimization with gradients. Advances in neural information processing systems , 30, 2017.

G. Wynne and V. Wild. Variational gaussian processes: A functional analysis view. In International Conference on Artificial Intelligence and Statistics , pages 4955-4971. PMLR, 2022.

W. Xie, B. L. Nelson, and R. R. Barton. A bayesian framework for quantifying uncertainty in stochastic simulation. Operations Research , 62(6):1439-1452, 2014.

W. Xie, B. Wang, and P. Zhang. Metamodel-assisted sensitivity analysis for controlling the impact of input uncertainty. In 2019 Winter Simulation Conference (WSC) , pages 3681-3692, 2019.

W. Xie, Y. Yi, and H. Zheng. Global-local metamodel-assisted stochastic programming via simulation. ACM Transactions on Modeling and Computer Simulation (TOMACS) , 31(1):1-34, 2020.

P. Xu, Z. Wen, H. Zhao, and Q. Gu. Neural contextual bandits with deep representation and shallow exploration. arXiv preprint arXiv:2012.01780 , 2020.

Y. Xu and A. Zeevi. Upper counterfactual confidence bounds: a new optimism principle for contextual bandits. arXiv preprint arXiv:2007.07876 , 2020.

H. Zhang, J. He, R. Righter, Z.-J. M. Shen, and Z. Zheng. Machine learning-assisted stochastic kriging for offline simulation online application. Working Paper .

H. Zhang, J. He, D. Zhan, and Z. Zheng. Neural network-assisted simulation optimization with covariates. In 2021 Winter Simulation Conference (WSC) , pages 1-12. IEEE, 2021.

D. Zhou, L. Li, and Q. Gu. Neural contextual bandits with ucb-based exploration. In International Conference on Machine Learning , pages 11492-11502. PMLR, 2020.

Y. Zhu, D. J. Foster, J. Langford, and P. Mineiro. Contextual bandits with large action spaces: Made practical. In International Conference on Machine Learning , pages 27428-27453. PMLR, 2022.

Peer reviews:
comments: SUMMARY:
The paper proposes extension of the Gaussian Process (GP) based contextual bandits to the contextual case, where the reward function is time and context dependent. The proposed method models the context dependency via a multi-output GP and its relationship to the actions via a neural network. The inner product of the outputs of the GP and the neural network determines the reward.

STRENGTHS:
The paper is well-written, its language is clear, experiments thorough, results strong, and the theoretical analysis sound.

Despite the theoretical depth of the work, the paper studies two challenging and interesting use cases as bandit problems: queuing with LSTMs and pricing with graph convolutional neural nets. This gives a paper a solid standing. The graph application comes nicely together with the kernelized nature of the reward definition.

The results reported in Figures 3-6 are intriguing.

WEAKNESSES:
Algorithm 1 has some novel aspects such as the way mu and sigma are calculated but the methodological novelty is rather incremental.

QUESTIONS:
It does not look clear to me what the purpose of Section 3.3. What do we expect from studying the maximum information gain aspect?

Why is the proposed method not compared against NeuralUCB also in queuing and pricing tasks?

LIMITATIONS:
The paper does not discuss the limitations of the proposed approach.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
This work studies the contextual bandit problem with the Gaussian process. Authors tried to use neural networks to learn reward functions and introduce one algorithm NN-AGP. Then conduct empirical evaluation and theoretical analysis for it.

STRENGTHS:
Recently, the bandit community paid more and more attention to the neural network approximation. It is interesting to combine it with GP in bandits. Authors use extensive experiments to demonstrate its practicability and conduct rigorous regret analysis for it.

WEAKNESSES:
(1) This work is an extension of [37] but very similar is very high. The key difference from [37] is changing the reward function to the inner product of $g(\theta)$ and $p(x)$, where $g$ is the neural network function and $p$ is the Gaussian process. The optimization of $p$ and the UCB-based exploration is adapted from [37]. But I don't see very novel aspects from $g$ except that it can be replaced by different neural network functions. Therefore, I think the overall novelty of this paper is limited. 

(2) The required assumption of regret analysis is strong. It requires that the space of $\theta$ is convex and compact, but it is known the parameter space of neural network is non-convex and not compact. Moreover, the analysis is based on [37] and [34,35] (the information gain part). Authors may want to discuss more about [34, 35] from analysis aspect.

QUESTIONS:
(1) In Figure 2, I am wondering why there is a big jump first and then starts dropping.
(2) How about the running time cost (portion) of GP in this algorithm?

LIMITATIONS:
Strong assumption and incremental novelty.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a reward model, called the neural network-accompanied Gaussian process (NN-AGP) for solving contextual bandit problems where the space of contexts and the space of decision variables may be continuous. This model is an inner product of a neural network and a multi-output GP. The neural network captures the dependence of the reward function on the contextual variables while the GP is to model the mapping from the decision variable to the reward.  The authors propose the NN-AGP-UCB algorithm for this problem in the form of the upper confidence bound strategy. They derive the regret for NN-AGP-UCB as well as the maximum information gain in their setting. Finally, they provide the experiments to evaluate their proposed algorithm for complex reward functions, including those with time-varying dependence on sequenced and graph-structured contextual variables.\n\nSTRENGTHS:\n- The paper introduces a reward model for contextual Gaussian process bandits which is more general than the previous work by Krause el al [37] by employing a neural network to capture the dependence of the reward function on the contextual variables. This allows to use different neural networks appropriate for applications with diverse structures of contextual information. This is also demonstrated in their experimental results.\n- The regret analysis and the upper bound of the maximum information gain are provided under this model.  \n- The experimental results on different structures of context information are also a strong point of this paper.\n\nWEAKNESSES:\n- As defined, the reward model is an inner product of a neural network (NN) and a multi-output GP. This reward structure seems not natural compared to the ones which are entirely either GP or NN. In addition, there are also many possible combinations of a NN and a GP to construct a reward model. Therefore, a more general model would be better. \n- In the conclusion section, the authors claimed that the advantage of their approach is the approximation accuracy for the reward function and better performance on cumulative rewards/regrets. However, this is not correct. Using a NN to model reward in overparameterized regime allows the approximation accuracy for the reward function. Moreover, they said their approach has better performance on regrets. However, it is not clear which related works they are comparing with. \n- It lacks the comparison of the regret bounds of the proposed algorithm and related algorithms like NeuralUCB, NeuralTS, and NeuralLinUCB which use entirely a NN, and algorithms which use entirely a GP to model the reward function.\n\nQUESTIONS:\n- Please see the questions in the Weaknesses section. \n- It would be interesting if the authors take into consideration the influence of the proposed NN in their regret analysis, at least in the overparametrized regime.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This work proposes a NN accompanied GP model. It leverages NN to approximate the unknown reward function regarding the context variable and maintains a GP with the decision variable. By introducing NN, the proposed method offers a better approximation accuracy. Theoretical implications, including maximum information gain and regret bounds, are provided for the proposed method. The effectiveness of the proposed method is also supported by empirical evaluation on both a synthetic and real-world dataset.

STRENGTHS:
1. A well-motivated and well-designed algorithm with solid theoretical analysis of the statistical properties of the proposed model. 
2. The proposed model is flexible enough to be used together with different types of NN models. 
3. Empirical evaluation of a diverse set of tasks is included and shows the promising effectiveness of the proposed method.

WEAKNESSES:
1. Potential limitations of the model: Although NN could help enable a better approximation accuracy on the reward function regarding the context variable, the training of NN models may make the model computationally prohibitive. In addition, NN typically works well with a large amount of data, which may cause a cold-start issue when the proposed model is used. 

2. May need to include more metrics in the empirical evaluation, such as computation cost and prediction latency.

QUESTIONS:
Can the authors provide corresponding justifications or empirical evidence on the potential limitations of the proposed method mentioned in the weaknesses section? 

I am willing to adjust my rating based on the authors' response to my question above.

LIMITATIONS:
See weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-13