SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haoting Zhang Jinghai He Rhonda Righter Zuo-Jun Max Shen Zeyu Zheng
Department of Industrial Engineering & Operations Research
Berkeley, CA 94720
haoting_zhang, jinghai_he, rrighter, maxshen, zyzheng@berkeley.edu
Contextual decision-making problems have witnessed extensive applications in various fields such as online content recommendation, personalized healthcare, and autonomous vehicles, where a core practical challenge is to select a suitable surrogate model for capturing unknown complicated reward functions. It is often the case that both high approximation accuracy and explicit uncertainty quantification are desired. In this work, we propose a neural network-accompanyed Gaussian process (NN-AGP) model, which leverages neural networks to approximate the unknown and potentially complicated reward function regarding the contextual variable, and maintains a Gaussian process metamodel with respect to the decision variable. Our model is shown to outperform existing approaches by offering better approximation accuracy thanks to the use of neural networks and possessing explicit uncertainty quantification from the Gaussian process. We also analyze the maximum information gain of the NN-AGP model and prove regret bounds for the corresponding algorithms. Moreover, we conduct experiments on both synthetic and practical problems, illustrating the effectiveness of our approach.
Various applications, including online content recommendation [1], healthcare [37, 15, 36], and autonomous vehicles [7], demand the sequential selection of a decision variable, conditional on the observed contextual variable representing the state of the environment in each round. These applications can generally be framed as contextual bandit problems [5, 59, 63, 2], especially when the reward function associated with each pair of decision and contextual variables is unknown. When both the decision and contextual variables are drawn from continuous sets, a significant challenge is selecting an appropriate surrogate model to approximate the reward function, considering both approximation accuracy and uncertainty quantification. A common approach to alleviate this issue is to employ a Gaussian process (GP) to model the reward function [61, 46], yielding the GP bandit method [80, 81]. Indeed, GP has proven to be an effective surrogate model to address the exploration-exploitation trade-off in estimating the unknown function while optimizing over it; see [88, 75, 90, 43, 51]. On the other hand, most of the existing GP bandit literature does not take the exogenous contextual variable into consideration, despite its critical role in capturing effects beyond the decision variable that influence the reward – effects that are integral to many of the applications previously mentioned [50, 62]. When the contextual variable is included in GP bandit problems, previous work largely adopts a GP to jointly model the reward function with both contextual and decision variables, employing a composite kernel that is either the sum or product of two separate kernels; see [57, 9].
While GP-based bandit methods have proven effective in various applications [3, 8, 92, 93, 86, 6], they may fall short in scenarios where the reward function exhibits intricate dependence on complex
contextual variables, for example, time-varying rewards [13, 31] and graph-structured contextual variables [71]. Specifically, it is a challenge to pre-define an appropriate composite kernel function for the joint GP, which is critical for the performance of the corresponding bandit algorithms, as documented by [21, 82]. Neural networks (NN), on the other hand, have been utilized elsewhere as surrogate models for the reward function in bandit problems [16, 54, 55], thanks to their flexibility and strong approximation power. However, they bring their own set of challenges. The "black-box" nature of the neural network hinders explicit uncertainty quantification and complicates theoretical analysis of the associated algorithms. In particular, the acquisition functions guiding point selection in these algorithms necessitate an approximation of uncertainty [99, 54]. Although this approximation tends to be accurate when the NN's width is large, this can also lead to overparameterization.
Contribution. This paper proposes a neural network-accompanyed Gaussian process (NN-AGP) model for solving contextual bandit problems, especially when the reward functions have intricate dependence on complex contextual variables. The proposed model is an inner product of a neural network and a multi-output GP, where the neural network captures the dependence of the reward function on the contextual variables, and the GP is employed to model the mapping from the decision variable to the reward. Our model generates a joint GP with both contextual and decision variables, which outperforms the existing GP-based bandit methods by specifying the data-driven kernel function through the lens of neural networks, thereby leading to an accurate approximation for the reward function. Moreover, compared with entirely relying on NN's, our model stands out due to the explicit GP expression with respect to the decision variable. This feature enables bandit algorithms with NN-AGP to be implemented efficiently with existing GP-based acquisition functions and provides a theoretical guarantee of the regret bounds. Our main contributions can be summarized as follows:
1. We propose an NN-AGP model and its upper confidence bound (UCB) algorithm for contextual bandit problems, referred to as NN-AGP-UCB. Our algorithm offers a data-driven procedure to specify the kernel functions of the joint GP, thereby achieving superior accuracy and model flexibility. We also prove the upper bound for both the maximum information gain of the NN-AGP model and the regret of the NN-AGP-UCB algorithm.
2. We conduct the experiments to evaluate our approach for complex reward functions, including those with time-varying dependence on sequenced and graph-structured contextual variables. Experimental results demonstrate the superiority of our approach over existing approaches that entirely rely on either GP or NN.
Since the seminal work by [80], the Gaussian process (GP) bandit problem has been extensively studied, where the bandit feedback is modeled as a GP regarding the decision variable (arms to be pulled). Some recent work includes [35, 32, 13, 12, 11, 18, 65]. In addition, GP bandits are also related to Bayesian optimization (BO) problems [38, 19, 94, 39, 34, 28, 85, 52, 23], where both problems consider optimizing black-box functions and therefore require surrogate models (GP in particular). When the number of decision variables is finite, the black-box optimization problem is also known as Ranking & Selection (R&S) [44, 73, 70, 4, 87], where GP models are widely employed as well; see [20, 58, 79, 64]. Our NN-AGP model can also be employed in (contextual) BO or R&S, but the discussion is beyond the scope of this work.
In this work, we specifically take the exogenous contextual variables into consideration. Previous work employs multiplicative and additive kernels to incorporate continuous context spaces into the scalar GP; see [57]. Other work considers safe contextual Bayesian optimization, employing a similar strategy to construct composite kernels; see [41, 9]. Another line of research explores distributionally robust BO [56, 83, 53], where the contextual variable distribution is selected from an ambiguity set. The methodology of representing the objective function using a joint GP has also been widely used in contextual policy search; see [72, 21, 40].
The connection between GP and NN has been explored in [60, 68], documenting that NN's with infinite width approach a GP model when the weight parameters are assigned with Gaussian priors. In addition, deep GP's have been proposed to enhance the model flexibility of neural networks where variational inference is employed; see [27, 26, 91]. GP models in which the parameters are represented by neural networks are studied in [98, 97].
2 Main procedure
We consider a problem of sequentially selecting a system's input variable (decision variable) for  $T$  (not necessarily known a priori) rounds. In each round, we receive a contextual variable  $\pmb{\theta}_t \in \Theta \subset \mathbb{R}^{d'}$  from a set  $\Theta$ , and select a decision variable  $\mathbf{x}_t \in \mathcal{X} \subset \mathbb{R}^d$  from a set  $\mathcal{X}$  of decisions. We then receive an observation
$$
y _ {t} = f (\mathbf {x} _ {t}; \boldsymbol {\theta} _ {t}) + \epsilon_ {t},
$$
where  $f(\mathbf{x};\boldsymbol {\theta})$  is the reward (objective) function and  $\epsilon_t\stackrel {i.i.d.}{\sim}\mathcal{N}(0,\sigma_\epsilon^2)$  denotes the noise that is independent of both the contextual variables and decision variables. We consider the scenarios when both  $\Theta$  and  $\mathcal{X}$  are continuous and  $\theta_{t}$ 's are fully exogenous. That is, the selection of the decision variable in each round does not influence the future contextual variables. Although we focus on unconstrained problems in this work, our proposed model can be employed to approximate the constraints in optimization problems as well; see [9]. We also note that, for the description and discussion of our approach, the contextual variable is represented by a vector. However, we show through experiments in Section 4 that our approach is applicable to contextual variables with other structures.
Since  $f(\mathbf{x},\pmb {\theta})$  is unknown, we will not generally be able to choose the optimal action, and will thus incur regret  $r_t = \sup_{\mathbf{x}'\in \mathcal{X}}f(\mathbf{x}',\pmb {\theta}_t) - f(\mathbf{x}_t,\pmb {\theta}_t)$ , indicating the difference between the optimal reward and the reward we actually receive in each round. After  $T$  rounds, the cumulative regret is  $\mathcal{R}_T = \sum_{t = 1}^T r_t$ . Our goal is to develop an algorithm that achieves sub-linear contextual regret, i.e.,  $\mathcal{R}_T / T\to 0$  for  $T\to \infty$ , which requires a statistical model of the reward function with respect to both context variables and decision variables.
We specifically consider a reward function in the form
$$
f (\mathbf {x}; \boldsymbol {\theta}) = \boldsymbol {g} (\boldsymbol {\theta}) ^ {\top} \mathbf {p} (\mathbf {x}), \tag {1}
$$
where  $g(\theta)$  and  $\mathbf{p}(\mathbf{x})$  are both  $m$ -dimensional vector-valued (unknown) functions, and  $m \in \mathbb{N}$  is a user-selected quantity to indicate the complexity of the function. There are two main reasons for considering this reward function. First, this formulation is a generalization of linear contextual bandits, where the reward function is the inner product of the contextual variables and the unknown parameters. Here, we assume that the inner product is taken with two vector-valued functions with respect to decision variables and the contextual variable, which is similar in spirit to [24, 96, 42, 100]. Second, this reward function is consistent with the tensor-product approximation of a general function; see [29, 30, 47]. Therefore, further analysis on model mis-specification of the reward function can be supported by existing results of tensor-product approximation.
In this work, we specifically assume that  $g(\theta)$  is a vector-valued deterministic mapping from  $\mathbb{R}^{d'}$  to  $\mathbb{R}^m$ , represented by a neural network with a given structure and some weight parameter  $W$ . In addition,  $\mathbf{p}(\mathbf{x})$  is a multi-output Gaussian process (MGP) defined on  $\mathcal{X} \subset \mathbb{R}^d$ . The MGP model is a generalization of the scalar-valued GP, where the output  $\mathbf{p}(\mathbf{x})$  at each  $\mathbf{x}$  is an  $m$ -dimensional vector. The MGP model captures not only the dependence between two outputs but also the dependence between different entries of each output. Thus, the covariance of an MGP  $\mathbf{p}(\mathbf{x})$  is represented by a matrix-valued covariance function, denoted by  $\mathcal{K}(\mathbf{x},\mathbf{x}')$ , and the vector of parameters involved in the MGP is denoted by  $\Phi$ . We postpone the detailed description of the NN-AGP model to Section 3.1 and conclude our brief introduction of NN-AGP with an associated proposition, which follows easily from the fact that the normal distribution is preserved under linear transformations.
Proposition 1. The reward function  $f(\mathbf{x};\pmb {\theta})$  is a scalar-valued mean-zero Gaussian process with respect to  $\mathbf{x}$  and  $\pmb{\theta}$ . The kernel function of this Gaussian process is
$$
\tilde {K} \left((\mathbf {x}, \boldsymbol {\theta}), (\mathbf {x} ^ {\prime}, \boldsymbol {\theta} ^ {\prime})\right) = \boldsymbol {g} (\boldsymbol {\theta}) ^ {\top} \mathcal {K} \left(\mathbf {x}, \mathbf {x} ^ {\prime}\right) \boldsymbol {g} (\boldsymbol {\theta} ^ {\prime}),
$$
where  $\mathcal{K}(\mathbf{x},\mathbf{x}^{\prime})$  is the covariance of the MGP.
Next, we provide a bandit algorithm with the NN-AGP model, during which the surrogate model is sequentially learned from data. We name the algorithm neural network-accompanyed Gaussian process upper confidence bound (NN-AGP-UCB). Suppose we are now in round  $t$  and observe the contextual variable  $\pmb{\theta}_t$ . In addition, we also have the historic data  $\mathcal{D}_{t - 1} = \{(\pmb{\theta}_1,\mathbf{x}_1,y_1),(\pmb{\theta}_2,\mathbf{x}_2,y_2),\dots ,(\pmb{\theta}_{t - 1},\mathbf{x}_{t - 1},y_{t - 1})\}$  in hand. Denote by  $\mathbf{y}_{t - 1} = (y_1,y_2,\dots ,y_{t - 1})$
the vector of observations. The selection of the next decision variable  $\mathbf{x}_t$  depends on the posterior distribution of the reward function, which is  $f(\mathbf{x};\pmb{\theta}_t) \mid \mathcal{D}_{t-1} \sim \mathcal{N}\left(\mu_{t-1}(\mathbf{x};\pmb{\theta}_t), \sigma_{t-1}^2(\mathbf{x};\pmb{\theta}_t)\right)$ . Here
$$
\mu_ {t - 1} (\mathbf {x}; \boldsymbol {\theta} _ {t}) = \tilde {\mathcal {K}} _ {(\mathbf {x}; \boldsymbol {\theta} _ {t})} ^ {\top} \left[ \tilde {\mathcal {K}} _ {\mathcal {D} _ {t - 1}} + \sigma_ {\epsilon} ^ {2} I _ {t - 1} \right] ^ {- 1} \mathbf {y} _ {t - 1}, \tag {2}
$$
$$
\sigma_ {t - 1} ^ {2} (\mathbf {x}; \pmb {\theta} _ {t}) = \pmb {g} (\pmb {\theta} _ {t}) ^ {\top} \mathcal {K} (\mathbf {x}, \mathbf {x}) \pmb {g} (\pmb {\theta} _ {t}) - \tilde {\mathcal {K}} _ {(\mathbf {x}; \pmb {\theta} _ {t})} ^ {\top} \left[ \tilde {\mathcal {K}} _ {\mathcal {D} _ {t - 1}} + \sigma_ {\epsilon} ^ {2} I _ {t - 1} \right] ^ {- 1} \tilde {\mathcal {K}} _ {(\mathbf {x}; \pmb {\theta} _ {t})},
$$
where  $\tilde{\mathcal{K}}_{(\mathbf{x};\pmb{\theta}_t)}$  denotes the covariance vector between  $f(\mathbf{x};\pmb{\theta}_t)$  and  $\{f(\mathbf{x}_{\tau};\pmb{\theta}_{\tau})\}_{\tau = 1}^{t - 1}$ . In addition, for the  $(t - 1)\times (t - 1)$ -dimensional covariance matrix for historical data  $\hat{\mathcal{K}}_{\mathcal{D}_{t - 1}}$ , the  $(i,j)$ -th entry is  $\pmb{g}(\pmb{\theta}_i)^\top \mathcal{K}(\mathbf{x}_i,\mathbf{x}_j)\pmb{g}(\pmb{\theta}_j)$  as in Proposition 1. The required parameters in (2), including the weight parameters  $\mathbf{W}$  of the neural network  $\pmb{g}(\pmb{\theta})$ , the parameters  $\Phi$  involved in the MGP  $\mathbf{p}(\mathbf{x})$  and the variance  $\sigma_{\epsilon}^{2}$  of the noise  $\epsilon_t$ , are all learned and updated with the observations through (5), which we will discuss in Section 3.1.
In terms of the acquisition function (the function that decides the decision variable in the following iteration), we employ the contextual Gaussian process-upper confidence bound (CGP-UCB) introduced in [57]. That is, we decide  $\mathbf{x}_t$  as
$$
\mathbf {x} _ {t} = \arg \max  _ {\mathbf {x} \in \mathcal {X}} \left\{\mu_ {t - 1} (\mathbf {x}; \boldsymbol {\theta} _ {t}) + \beta_ {t} ^ {1 / 2} \sigma_ {t - 1} (\mathbf {x}; \boldsymbol {\theta} _ {t}) \right\}, \tag {3}
$$
where  $\mu_{t-1}(\mathbf{x};\pmb{\theta}_t)$  and  $\sigma_{t-1}(\mathbf{x};\pmb{\theta}_t)$  are the posterior mean and standard deviation of  $f(\mathbf{x};\pmb{\theta}_t)$  as calculated in (2). The optimization problem (3) can be solved efficiently by global search heuristics, as suggested in [17]. In addition,  $\beta_t$  is a user-selected hyper-parameter in each round, addressing the exploration-exploitation trade-off; see discussions in Section 3.2. The procedure for NN-AGP-UCB is summarized in Algorithm 1. We note that other commonly selected acquisition functions for GP bandit problems or Bayesian optimization can be employed with NN-AGP as well, including knowledge gradient [76, 89, 33] and Thompson sampling [25, 74]. We postpone the discussion of these acquisition functions to the supplements.
Algorithm 1 NN-AGP-UCB
Input: Initial values of  $(\mathbf{W},\Phi ,\sigma_{\epsilon}^{2})$    
for  $t = 1,2,\ldots ,T$  do Observe the contextual variable  $\pmb{\theta}_{t}$  . Choose  $\mathbf{x}_t = \arg \max_{\mathbf{x}\in \mathcal{X}}\left\{\mu_{t - 1}(\mathbf{x};\pmb {\theta}_t) + \beta_t^{1 / 2}\sigma_{t - 1}(\mathbf{x};\pmb {\theta}_t)\right\}$  Sample  $y_{t}$  at  $(\pmb {\theta}_t,\mathbf{x}_t)$  Update  $\left(\hat{\mathbf{W}}_t,\hat{\Phi}_t,\sigma_{\epsilon ;t}^2\right)$  as in (5);   
end for
3 Statistical properties
3.1 Specification of NN-AGP
We describe the NN-AGP model here, which is employed as the surrogate for the reward function. Recall that the reward function with the pair of contextual and decision variables  $(\pmb{\theta},\mathbf{x})$  is  $f(\mathbf{x};\pmb{\theta}) = g(\pmb{\theta})^\top \mathbf{p}(\mathbf{x})$ , where  $g(\pmb{\theta})$  is a vector-valued neural network (with weight parameters  $\mathbf{W}$ ) from  $\mathbb{R}^{d'}$  to  $\mathbb{R}^m$  and  $\mathbf{p}(\mathbf{x})$  is an  $m$ -dimensional output Gaussian process defined on  $\mathcal{X} \subset \mathbb{R}^d$ . To be more specific, the  $m$  outputs  $\mathbf{p} = (\mathbf{p}_1,\dots,\mathbf{p}_m)^\top$  are assumed to follow a multi-output Gaussian process (MGP) as
$$
\mathbf {p} (\mathbf {x}) \sim \mathcal {M G P} \left(\mathbf {0}, \mathcal {K} \left(\mathbf {x}, \mathbf {x} ^ {\prime}\right)\right).
$$
Here,  $\mathcal{K}(\mathbf{x},\mathbf{x}^{\prime})$  denotes the covariance matrix of  $\mathbf{p}(\mathbf{x})$  and  $\mathbf{p}(\mathbf{x}^{\prime})$ , defined as  $\mathcal{K}(\mathbf{x},\mathbf{x}^{\prime}) \doteq \left( \begin{array}{ccc} K_{11}(\mathbf{x},\mathbf{x}^{\prime}) & \dots & K_{1m}(\mathbf{x},\mathbf{x}^{\prime}) \\ \vdots & \ddots & \vdots \\ K_{m1}(\mathbf{x},\mathbf{x}^{\prime}) & \dots & K_{mm}(\mathbf{x},\mathbf{x}^{\prime}) \end{array} \right)$  which is positive and semi-definite. The  $(l,l^{\prime})$ -th entry
$\hat{K}_{ll^{\prime}}(\mathbf{x},\mathbf{x}^{\prime})$  represents the covariance (similarity) between outputs  $\mathbf{p}_l(\mathbf{x})$  and  $\mathbf{p}_{l^{\prime}}(\mathbf{x}^{\prime})$ . The NN-AGP
model results in a scalar-valued Gaussian process with both the contextual and decision variables and therefore facilitates explicit acquisition functions and theoretical analysis. This GP representation arises from the linear structure between the MGP and the mapping  $g(\theta)$ .
To specify the covariance matrix, we adopt the collaborative multi-output Gaussian process model [69] as a representative, which generalizes the commonly-used linear model of coregionalization (LMC) and the intrinsic coregionalization model (ICM); see [67]. That is, the MGP is determined by the linear transformation of multiple independent scalar-valued Gaussian processes as
$$
\mathbf {p} _ {l} (\mathbf {x}) = \sum_ {q = 1} ^ {Q} a _ {l, q} u _ {q} (\mathbf {x}) + v _ {l} (\mathbf {x}). \tag {4}
$$
Here,  $\mathbf{p}_l(\mathbf{x})$  is the  $l$ -th element of  $\mathbf{p}(\mathbf{x})$ ,  $Q$  is the number of involved GP's,  $\{u_q(\mathbf{x})\}_{q=1}^Q$  and  $\{v_l(\mathbf{x})\}_{l=1}^m$  are independent scalar-valued GP's, and the  $a_{l,q}$ 's are coefficient parameters. In this way, the correlation between different entries in the MGP  $\mathbf{p}(\mathbf{x})$  is captured by  $\{u_q(\mathbf{x})\}_{q=1}^Q$  through the  $a_{l,q}$ 's. Moreover,  $v_l(\mathbf{x})$  represents specific independent features of  $\mathbf{p}_l(\mathbf{x})$  itself, for  $l = 1,2,\ldots,m$ . Suppose the kernel functions of the  $u_q(\mathbf{x})$ 's and  $v_l(\mathbf{x})$ 's are  $k_q(\mathbf{x},\mathbf{x}')$ 's and  $\tilde{k}_l(\mathbf{x},\mathbf{x}')$ 's and all these kernel functions are less than or equal to one, as is regularly assumed [80, 81, 57]. Then the matrix-valued kernel function of  $\mathbf{p}(\mathbf{x})$  is  $\mathcal{K}(\mathbf{x},\mathbf{x}') = \sum_{q=1}^{Q}\mathbf{A}_qk_q(\mathbf{x},\mathbf{x}') + \mathrm{Diag}\left\{\tilde{k}_1(\mathbf{x},\mathbf{x}')\dots,\tilde{k}_m(\mathbf{x},\mathbf{x}')\right\}$ . Here,  $\mathbf{A}_q$  denotes the semi-definite matrix in which the  $(l,l')$ -th entry is  $a_{l,q}a_{l',q}$ . In some applications, the parameters involved in the kernel functions  $k_q(\mathbf{x},\mathbf{x}')$  and  $\tilde{k}_l(\mathbf{x},\mathbf{x}')$ , and the coefficients  $a_{l,q}$ 's are not known in advance. We denote these unknown parameters and coefficients in the MGP as  $\Phi$ . In addition to the model (4), other types of MGP's can be employed in our methodology as well [14], while the selection of model (4) enables the theoretical analysis of our approach.
In terms of the mapping  $g(\theta)$ , since we have no prior knowledge, we select the neural network as the surrogate model, due to 1) its strong approximation power for intricate dependence on complex variables; 2) its flexibility of adaptation to different application scenarios (e.g. time-series or graph-structured contextual variables) and 3) the availability of fruitful methods and tools for the training procedure. Given the data  $\mathcal{D}_t = \{(\pmb{\theta}_1, \mathbf{x}_1, y_1), (\pmb{\theta}_2, \mathbf{x}_2, y_2), \dots, (\pmb{\theta}_t, \mathbf{x}_t, y_t)\}$ , the learning of unknown parameters in the MGP and the weight parameters in the neural network (as well as the noise variance) is through maximum likelihood estimation (MLE). That is
$$
\left(\hat {\mathbf {W}} _ {t}, \hat {\Phi} _ {t}, \sigma_ {\epsilon ; t} ^ {\hat {2}}\right) = \arg \max  _ {(\mathbf {W}, \Phi , \sigma_ {\epsilon} ^ {2})} L _ {t} \left(\mathbf {W}, \Phi , \sigma_ {\epsilon} ^ {2}\right), \tag {5}
$$
where the (normalized) likelihood function is  $L_{t} = -\ln \left[\left|\tilde{K}_{\mathcal{D}_{t}} + \sigma_{\epsilon}^{2}I_{t}\right|\right] - \mathbf{y}_{t}^{\top}\left[\tilde{K}_{\mathcal{D}_{t}} + \sigma_{\epsilon}^{2}I_{t}\right]^{-1}\mathbf{y}_{t}$ . Here,  $\mathbf{y}_t = (y_1,y_2,\dots ,y_t)$  is the vector of observations and  $\tilde{K}_{\mathcal{D}_t}$  is the covariance matrix of the data  $\mathcal{D}_t$ . The parameters  $\mathbf{W}$  and  $\Phi$  are contained in this covariance matrix. That is, instead of pre-defining a kernel function of the GP, the kernel function of NN-AGP is specified through learning the neural network from the data, yielding better approximation accuracy. We include a discussion of the consistency of training NN-AGP in the supplements.
3.2 Cumulative regret
Recall that the cumulative regret is defined as  $\mathcal{R}_T = \sum_{t=1}^T \left\{ \sup_{\mathbf{x}' \in \mathcal{X}} f(\mathbf{x}', \boldsymbol{\theta}_t) - f(\mathbf{x}_t, \boldsymbol{\theta}_t) \right\}$ . Here we provide an upper bound of  $\mathcal{R}_T$  with NN-AGP-UCB.
Theorem 1. Suppose  $\delta \in (0,1)$  and the following.
1. The decision variable  $x \in \mathcal{X} \subseteq [0, r]^d$  and  $\mathcal{X}$  is convex and compact. The contextual variable  $\pmb{\theta} \in \Theta \subseteq \mathbb{R}^{d'}$  and  $\Theta$  is convex and compact;  $g(\pmb{\theta})$  is a known continuous mapping of  $\pmb{\theta} \in \Theta$ ;  $\mathbf{p}(\mathbf{x})$  is sampled from a known MGP prior as in (4) and the variance of the noise  $\sigma_{\epsilon}^{2}$  is known. That is, these parameters do not need learning and updating from data.
2. For the components of the MGP, there exist constants  $\{a_q\}_{q=1}^Q, \{b_q\}_{q=1}^Q, \{\tilde{a}_l\}_{l=1}^m, \{\tilde{b}_l\}_{l=1}^m$  satisfying