SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {P} \left\{\sup  _ {x \in \mathcal {X}} \left| \frac {\partial u _ {q} (x)}{\partial x _ {j}} \right| > L _ {q} \right\} \leqslant a _ {q} e ^ {- \left(L _ {q} / b _ {q}\right) ^ {2}}; \mathbb {P} \left\{\sup  _ {x \in \mathcal {X}} \left| \frac {\partial v _ {l} (x)}{\partial x _ {j}} \right| > \tilde {L} _ {l} \right\} \leqslant \tilde {a} _ {l} e ^ {- \left(\tilde {L} _ {l} / \tilde {b} _ {l}\right) ^ {2}} \tag {6}
$$
$\forall L_{q}, \tilde{L}_{l} > 0$  and  $\forall j = 1,2,\ldots,d, q = 1,2,\ldots,Q$  and  $l = 1,2,\ldots,m$ .
3. We choose as a hyper-parameter in (3)
$$
\beta_ {t} = 2 \log \left(t ^ {2} 2 \pi^ {2} / (3 \delta)\right) + 2 d \log \left(\tilde {M} t ^ {2} d b r \sqrt {\log (4 d a / \delta)}\right),
$$
where  $d$  and  $r$  are the dimension and the upper bound of the decision variable,  $a = \sum_{q=1}^{Q} a_q + \sum_{l=1}^{m} \tilde{a}_l$ ,  $b = \sum_{q=1}^{Q} b_q + \sum_{l=1}^{m} \tilde{b}_l$ ,  $\tilde{M} = \sup_{\pmb{\theta} \in \Theta} \left\{ \left\{ |\sum_{l=1}^{m} \pmb{g}_l(\pmb{\theta}) a_{l,q}| \right\}_{q=1}^{Q}, \left\{ |\pmb{g}_l(\pmb{\theta})| \right\}_{l=1}^{m} \right\}$ , where  $\pmb{g}_l$  denotes the  $l$ -th entry of  $\pmb{g}(\pmb{\theta})$ .
Then the cumulative regret is bounded with high probability as
$$
\mathbb {P} \left\{\mathcal {R} _ {T} \leqslant \sqrt {\frac {8 C T \beta_ {T} \gamma_ {T}}{\log \left(1 + C \sigma_ {\epsilon} ^ {- 2}\right)}} + \frac {\pi^ {2}}{6}, \forall T \geqslant 1 \right\} \geqslant 1 - \delta .
$$
Here  $C = \left(\left(\sum_{q=1}^{Q}\sum_{l=1}^{m}a_{l,q}^{2}\right) + 1\right)\sup_{\pmb{\theta}\in \Theta}\|\pmb{g}(\pmb{\theta})\|_{2}^{2}$ . In addition,  $\gamma_{T}$  is the maximum information gain associated with the NN-AGP  $f(\mathbf{x};\pmb{\theta})$ , defined by (7).
We postpone the discussion of the maximum information gain  $\gamma_{T}$  to Section 3.3, with some specific kernels employed in the MGP component of the NN-AGP model. We note that GP's with commonly-selected kernel functions, including the Matérn kernel and the radial basis function kernel, satisfy the condition (6) and further discussions can be found in Theorem 5 in [45]. A detailed proof of Theorem 1 is contained in the supplements. Note that Theorem 1 assumes that  $g(\theta)$  is exactly known so does not consider the error of approximating  $g(\theta)$  with the neural networks. In the supplements, we include a detailed discussion of the algorithm that considers the neural network approximation error, as well as the corresponding regret bounds.
At the end of this section, we compare our regret bound with existing work. Specifically, NN-AGP-UCB has the same bound of  $\tilde{\mathcal{O}}\left(\sqrt{T\gamma_T}\right)$  as CGP-UCB, but is superior when the contextual variable dimension is high; see details in the supplements. In terms of the algorithms which entirely rely on NN, we note that NeuralUCB [99], Neural TS [98], and Neural LinUCB [95] all consider the scenarios when the decision variable  $\mathbf{x}$  is selected from a finite set. In comparison, we consider that  $\mathbf{x}$  is selected from a continuous set. When performed on a finite feasible set  $\mathcal{X}$ , our NN-AGP-UCB also has a regret bound of  $\tilde{\mathcal{O}}\left(\sqrt{T\gamma_T}\right)$ , where the maximum information gain  $\gamma_{T}$  further depends on the kernel function of the GP component used in NN-AGP. When the kernel function has an exponential eigendecay (see Definition 1), NN-AGP-UCB has a regret bound of  $\tilde{\mathcal{O}}\left(\sqrt{T}\right)$ , matching the regret bound of NeuralUCB, Neural TS and Neural LinUCB as well.
3.3 Maximum information gain
In this section, we discuss the information gain  $\gamma_{T}$  of the proposed NN-AGP model. The maximum information gain is defined as
$$
\gamma_ {T} = \sup  _ {\left\{\left(\boldsymbol {\theta} _ {t}, \mathbf {x} _ {t}\right) \right\} _ {t = 1} ^ {T} \subseteq \Theta \times \mathcal {X}} I \left(\mathbf {y} _ {T}; f _ {T}\right), \tag {7}
$$
where  $f_{T}$  is the reward function evaluated at  $\{(\pmb{\theta}_t,\mathbf{x}_t)\}_{t=1}^T$ ;  $\mathbf{y}_T$  denotes the observations;  $I(\mathbf{y}_T;f_T) = H(\mathbf{y}_T) - H(\mathbf{y}_T|f_T)$  is the mutual information between  $\mathbf{y}_T$  and  $f_T$ ;  $H(\cdot) = \mathbb{E}[-\log p(\cdot)]$  is the entropy of a random element, where  $p$  is the probability density function.
For ease of notation, here we consider the scenario when  $Q = 1$  and there is no  $\{v_{l}(\mathbf{x})\}$  in the MGP defined in (4), i.e.,  $\mathbf{p}_l(\mathbf{x}) = a_l u(\mathbf{x})$ . We also impose more variability so that  $\mathbf{A} = \mathbf{A}_1$  is a semi-definite positive matrix and not necessarily a rank-one matrix, analogous to [14]. We provide a more general discussion of the maximum information gain of the NN-AGP in the supplements. We first provide a proposition on the kernel function.
Proposition 2. When  $Q = 1$  and there is no  $\{v_{l}(\mathbf{x})\}$  in the MGP, the kernel function of the NN-AGP is a product of two kernel functions. That is  $\tilde{K}((\mathbf{x},\pmb{\theta}),(\mathbf{x}',\pmb{\theta}') = \tilde{k}(\pmb{\theta},\pmb{\theta}')k(\mathbf{x},\mathbf{x}')$ , where  $\tilde{k}(\pmb{\theta},\pmb{\theta}') = \pmb{g}(\pmb{\theta})^\top \mathbf{A}\pmb{g}(\pmb{\theta})$  is a finite rank kernel with respect to  $\pmb{\theta}$ . Furthermore, suppose that both  $\Theta$  and  $\mathcal{X}$  are compact,  $\pmb{g}(\pmb{\theta})$  is a continuous mapping and  $k(\mathbf{x},\mathbf{x}')$  is a semi-definite kernel function. Then the kernel function  $\tilde{K}((\mathbf{x},\pmb{\theta}),(\mathbf{x}',\pmb{\theta}'))$  possesses a Mercer decomposition:
$$
\tilde {K} \left((\mathbf {x}, \pmb {\theta}), (\mathbf {x} ^ {\prime}, \pmb {\theta} ^ {\prime})\right) = \sum_ {j = 1} ^ {\infty} \sum_ {i = 1} ^ {m} \mu_ {i} \lambda_ {j} \phi_ {i} \left(\pmb {\theta}\right) \psi_ {j} (\mathbf {x}) \phi_ {i} \left(\pmb {\theta} ^ {\prime}\right) \psi_ {j} \left(\mathbf {x} ^ {\prime}\right).
$$
Here,  $\{(\mu_i,\phi_i)\}_{i = 1}^m$  and  $\{(\lambda_j,\psi_j)\}_{j = 1}^m$  are the Mercer decompositions for  $\tilde{k} (\pmb {\theta},\pmb{\theta}^{\prime})$  and  $k(\mathbf{x},\mathbf{x}^{\prime})$  That is,  $\tilde{k} (\pmb {\theta},\pmb{\theta}^{\prime}) = \sum_{i = 1}^{m}\mu_{i}\phi_{i}(\pmb {\theta})\phi_{i}(\pmb{\theta}^{\prime}),k(\mathbf{x},\mathbf{x}^{\prime}) = \sum_{j = 1}^{\infty}\lambda_{j}\psi_{j}(\mathbf{x})\psi_{j}(\mathbf{x}^{\prime})$  , where the eigenvalues are  $\mu_1\geqslant \mu_2\geqslant \ldots \geqslant \mu_m\geqslant 0$  and  $\lambda_1\geqslant \lambda_2\geqslant \ldots \geqslant 0$
With the Mercer decomposition of the NN-AGP kernel function, we provide the bound of the maximum information gain. Specifically, we consider two scenarios for the employed kernel in the MGP, analogous to [22, 84].
Definition 1. Consider the eigenvalues  $\{\lambda_j\}_{j=1}^{\infty}$  of the kernel function  $k(\mathbf{x},\mathbf{x}')$  in decreasing order.
- For some  $C_p > 0, \alpha_p > 1, k$  is said to have a  $(C_p, \alpha_p)$  polynomial eigendecay, if for all  $j \in \mathbb{N}$ , we have  $\lambda_j \leqslant C_p j^{-\alpha_p}$ . An example is the Matérn kernel.
- For some  $C_{e,1}, C_{e,2}, \alpha_e > 0, k$  is said to have a  $(C_{e,1}, C_{e,2}, \alpha_e)$  exponential eigendecay, if for all  $j \in \mathbb{N}$ , we have  $\lambda_j \leqslant C_{e,1} \exp(-C_{e,2}j^{\alpha_e})$ . An example is the radial basis function kernel.
Theorem 2. Suppose that 1)  $\tilde{K} ((\mathbf{x},\pmb {\theta}),(\mathbf{x}',\pmb{\theta}')$  ) satisfies the conditions in Proposition 2; 2)  $\forall \mathbf{x},\mathbf{x}'\in$ $\mathcal{X}$ $|k(\mathbf{x},\mathbf{x}^{\prime})|\leqslant \bar{k}$  for some  $\bar{k} >0$  and 3)  $\forall j\in \mathbb{N},\forall \mathbf{x}\in \mathcal{X},|\psi_j(\mathbf{x})|\leqslant \psi ,$  for some  $\psi >0$  . If  $k(\mathbf{x},\mathbf{x}^{\prime})$  has a  $(C_p,\alpha_p)$  polynomial eigendecay, then
$$
\gamma_ {T} \leqslant m \left(\left(\frac {\bar {\mu} \phi^ {2} \psi^ {2} C _ {p} T}{\sigma_ {\epsilon} ^ {2}} \log^ {- 1} \left(1 + \frac {\bar {\tilde {k}} \bar {k} T}{m \sigma_ {\epsilon} ^ {2}}\right)\right) ^ {\frac {1}{\alpha_ {p}}} + 1\right) \log \left(1 + \frac {\bar {\tilde {k}} \bar {k} T}{m \sigma_ {\epsilon} ^ {2}}\right).
$$
If  $k(\mathbf{x},\mathbf{x}^{\prime})$  has a  $(C_{e,1},C_{e,2},\alpha_{e})$  exponential eigendecay, then
$$
\gamma_ {T} \leqslant m \left(\left(\frac {2}{C _ {e , 2}} \left(\log (T) + C _ {\alpha_ {e}}\right)\right) ^ {\frac {1}{\alpha_ {e}}} + 1\right) \log \left(1 + \frac {\bar {\tilde {k}} \bar {k} T}{m \sigma_ {\epsilon} ^ {2}}\right),
$$
$$
C _ {\alpha_ {e}} = \left\{ \begin{array}{l l} \log \left(\frac {C _ {e , 1} m \bar {\mu} \phi^ {2} \psi^ {2}}{\sigma_ {\epsilon} ^ {2} C _ {e , 2}}\right) & i f \quad \alpha_ {e} = 1 \\ \log \left(\frac {2 C _ {e , 1} m \bar {\mu} \phi^ {2} \psi^ {2}}{\sigma_ {\epsilon} ^ {2} \alpha_ {e} C _ {e , 2}}\right) + \left(\frac {1}{\alpha_ {e}} - 1\right) \left(\log \left(\frac {2}{C _ {e , 2}} \left(\frac {1}{\alpha_ {e}} - 1\right)\right) - 1\right) & o t h e r w i s e. \end{array} \right.
$$
Here,  $\bar{\mu} = \frac{1}{m}\sum_{i=1}^{m}\mu_{i}$  denotes the mean of the eigenvalues of the kernel function  $\tilde{k}(\pmb{\theta},\pmb{\theta}')$ ;  $\tilde{k} = \sup_{\pmb{\theta},\pmb{\theta}'\in \Theta}\left|\tilde{k}(\pmb{\theta},\pmb{\theta}')\right|$  and  $\phi = \sup_{\pmb{\theta}\in \Theta}|\phi(\pmb{\theta})|$ . Moreover, the maximum information gain of the NN-AGP is  $\mathcal{O}(m\gamma_{\mathbf{x};T})$ , where  $\gamma_{\mathbf{x};T}$  is the maximum information of  $k(\mathbf{x},\mathbf{x}')$ .
A more detailed discussion of the Mercer decomposition of NN-AGP is contained in the supplements, as well as the proofs of Proposition 2 and Theorem 2. At the end of this section, we compare our results on maximum information gain with [57]. For the composite kernel that is a product of two kernels, the upper bound is  $(d + d')\gamma_{\mathbf{x};T} + (d + d')\log T$ . Here,  $\gamma_{\mathbf{x};T}$  is the maximum information gain of the kernel function with the decision variable, and  $d'$  and  $d$  are the dimensions of the contextual variable and the decision variable. That is, the information gain (as well as the cumulative regret) increases as the dimension of the contextual variable increases. In comparison, the maximum information in Theorem 2 does not depend on  $d'$ . Thus, the NN-AGP model has lower cumulative regret than the classical strategy in [57] when the dimension of the contextual variable is relatively high.
In this section, we conduct experiments to show the practicality of our neural network-accompanyed Gaussian process upper bound confidence (NN-AGP-UCB) approach. We apply different neural networks to different problems, including the fully-connected neural network (FCN) [77] to a synthetic reward function, the long short-term memory (LSTM) [48] neural network to a queuing problem with time sequence contextual variables, and the graph convolutional neural network (GCN) [78] to a pricing problem with diffusion networks. We add a noise  $\epsilon_t^{i,i,d} \sim \mathcal{N}(0,0.01)$  to the ground-truth value of the reward in the first set of experiments. For the latter two sets of experiments, the reward is generated by stochastic simulation (and therefore is "black-box" with contextual/decision variables) and we postpone the description of the full dynamic to the supplements. We also provide additional experiments in the supplements, including 1) sensitivity on the structure of reward functions; 2) comparison with contextual variables possessing different dimensions; 3) regression tasks on complex functions and 4) finite decision/contextual variables with real data.
The baseline approaches include CGP-UCB [57], NeuralUCB [99] and NN-UCB [54]. The experiment results provided are mean performances based on repeating the experiments 15 times. Standard deviations (represented by a shadow associated with the mean-value line) are also included. In each iteration, the exogenous contextual variable  $\theta_{t}$  is randomly selected from  $\Theta$  with equal probability. In terms of the initialization, we randomly select decision variables independently of observed contextual variables for each approach in the first 20 iterations to attain surrogates. The specific description of the employed surrogate model in each approach is postponed to the supplements.
4.1 Synthetic reward function
In this section, we consider two synthetic reward functions
$$
R _ {1} (\mathbf {x}, \boldsymbol {\theta}) = - \sqrt {\left| \sin \left(\left\| \mathbf {x} \right\|\right) \boldsymbol {\theta} ^ {3} \exp \left(\cos \left(\left\| \mathbf {x} \right\| + \left\| \boldsymbol {\theta} \right\|\right)\right) \right|};
$$
$$
R _ {2} (\mathbf {x}, \boldsymbol {\theta}) = - \sqrt {\left| \sin \left(\left\| \mathbf {x} \right\|\right) \mathbf {x} ^ {3} \exp \left(\left\| \mathbf {x} \right\| + \cos \left(\left\| \boldsymbol {\theta} \right\|\right)\right) \right|}.
$$
For NN-AGP-UCB, we consider  $m = 2,3,5$  to study the effects of model selection on the algorithm performance. For CGP-UCB, we consider both additive kernels and multiplicative kernels. Because both NeuralUCB and NN-UCB are designed for contextual bandits with finite arms, we adapt them to the problem we consider in Section 2 and postpone the details to the supplements. The experimental results of the average regret  $\mathcal{R}_T / T$  are illustrated in Figure 1 and Figure 2, which provide the following insights. 1. Comparison with baseline approaches. For both reward functions, NN-AGP-UCB outperforms both the CGP-UCB and NN-based approaches. The advantage comes from 1) strong approximation power regarding  $\theta$  due to NN and 2) explicit inference regarding  $\mathbf{x}$  due to GP. 2. Effects of the dimension  $m$ . Generally, when  $m$  increases, the model flexibility increases, and the regret might be smaller, although this improvement might not be significant in some scenarios. Furthermore, a relatively large  $m$  might result in overparameterization especially when there are not enough iterations. A suggested selection of  $m$  is  $\lceil d/3 + d'/10 \rceil + 3$  considering both the algorithm performance and training complexity of models. 3. Breaking the linear assumption. Recall that we assume the reward function is of the form of  $R(\mathbf{x};\theta) = g(\theta)^{\top}\mathbf{p}(\mathbf{x})$ . However, the reward functions here break this linear assumption and yet our NN-AGP-UCB is still applicable and outperforms the baseline approaches. Moreover, additional experiments show that NN-AGP 1) is not sensitive on the structure of the reward functions; 2) has a greater advantage with higher-dimensional contextual variables and 3) achieves better approximation accuracy for complex functions, compared with a joint GP with composite kernels; see the supplements for details. In addition, when the dimension of the input increases, the uncertainty of the regret will increase as well; see Figures 1 & 2 for comparison. We also include the recorded computational time of these bandit algorithms in the supplements.
4.2 Queuing problem with time sequence contextual variables
In this section, we show through experiments that the NN-AGP model is applicable to contextual GP bandits when the objective function depends on the sequence of the contextual variables. That is, the reward function at time  $t$  can be approximated by
$$
f _ {t} \left(\mathbf {x}; \boldsymbol {\theta} _ {1}, \boldsymbol {\theta} _ {2}, \dots , \boldsymbol {\theta} _ {t}\right) \approx \boldsymbol {g} _ {t} \left(\boldsymbol {\theta} _ {1}, \boldsymbol {\theta} _ {2}, \dots , \boldsymbol {\theta} _ {t}\right) ^ {\top} \mathbf {p} (\mathbf {x}).
$$
Figure 1: Average regret of using  $R_{1}(\mathbf{x};\pmb{\theta})$  with  $\mathcal{X} = [-1,1]^2$  and  $\Theta = [-1,1]^3$ .
Figure 2: Average regret of using  $R_{2}(\mathbf{x};\boldsymbol{\theta})$  with  $\mathcal{X} = [-1,1]^{5}$  and  $\Theta = [-1,1]^{15}$ .
Here,  $g_{t}(\theta_{1},\theta_{2},\dots,\theta_{t})$  is modeled by an LSTM neural network. We consider a discrete-time queuing problem. In each time epoch, a contextual variable is first revealed. For example, the contextual variable might include traffic and weather conditions that affect the arrival process of the queuing system. The number of customers arriving at this epoch depends on the entire sequence of the revealed contextual variables up to now. The agent decides the service rate of the server and the service price for customers (decision variables). The reward function (might be negative) is the expected net income (the income brought by serving customers minus the service cost and penalty for losing customers). We let  $\mathcal{X} = [0,5]^2$  and sample  $\theta_{t}$  from multivariate normal distributions. We present the cumulative rewards in Figure 3 and Figure 4 of NN-AGP-UCB with LSTM. The baseline CGP-UCB adopts both additive and multiplicative kernels with the current contextual variable. We also apply kernel functions specifically designed for time series [10] to construct the composite kernel. Experimental results indicate that 1) employing a specific time series kernel enhances the performance of CGP-UCB and 2) NN-AGP-UCB with LSTM outperforms the classical CGP-UCB approaches with different composite kernels.
Figure 3: Cumulative rewards for a queuing problem with  $\theta_t \stackrel{i.i.d.}{\sim} \mathcal{N}(\mathbf{0}, I_3)$ .
Figure 4: Cumulative rewards for a queuing problem with  $\theta_t \stackrel{i.i.d.}{\sim} \mathcal{N}(\mathbf{0}, I_{10})$ .
4.3 Pricing with diffusion network
In this section, we show the NN-AGP model is applicable to contextual GP bandits with graph-structured contextual variables. That is, the contextual variable is summarized by a network structure
$$
\boldsymbol {\theta} _ {t} = \left(V _ {t}, E _ {t}\right),
$$
where  $V_{t}$  denotes the set of nodes and  $E_{t}$  denotes the set of directed/undirected edges of a network. To approximate  $g(\theta)$  with a graph-structured contextual variable, we apply the GCN model. We consider a pricing problem with a diffusion network, where each node represents a user who decides
to adopt a service or not and the edge between two nodes indicates whether the choices of these two users influence each other. In each iteration, the network structure  $\theta_{t}$  is first presented, and then the agent decides the price of the service. The reward function is the expected income for the service adoption from the users. The detailed description can be found in [66]. We let  $\mathcal{X} = [0,30]$  and  $\theta_{t}$  represents an undirected graph with 5 and 10 nodes where each edge exists with probability  $1 / 2$ . We present the cumulative rewards in Figure 5 and Figure 6 of NN-AGP-UCB with GCN. The baseline CGP-UCB adopts both additive and multiplicative kernels and in terms of the contextual variable, we consider 1) vectorizing the adjacency matrix that summarizes the network structure and 2) applying kernel functions that are specifically designed for graphs [49]. Experimental results indicate that NN-AGP-UCB with GCN outperforms the classical CGP-UCB approach adopting different kernels, and the advantages become greater for networks with more nodes.
Figure 5: Cumulative rewards with a 5-node network diffusion problem.
Figure 6: Cumulative rewards with a 10-node network diffusion problem.
5 Conclusion & impact
We propose a neural network accompanied Gaussian process (NN-AGP) model to address contextual GP bandit problems. The advantages of our approach include 1) flexibility of employing different neural networks appropriate for applications with diverse structures of contextual information; 2) approximation accuracy for the reward function and better performance on cumulative rewards/regrets; 3) tractability of a GP representation regarding the decision variable, thus supporting explicit uncertainty quantification and theoretical analysis. Our approach has potential application to healthcare, where doctors need to develop therapy plans based on patient information to achieve optimal treatment effects. When complex and sparse genetic information is employed, it necessitates the use of neural networks. Another potential application is for Automated Guided Vehicles (AGVs) to enhance workplace safety and reduce carbon emissions, where environmental information is provided to the AGV, and the AGV takes actions accordingly.
In terms of limitations, since NN-AGP retains a GP structure, it suffers from computational complexity with large data sets. To alleviate the computational burden, we consider sparse NN-AGP for future work; see a discussion in the supplements. In addition, incorporating NN into bandit problems generally requires sufficient data to approximate the unknown reward function, thereby bringing the cold-start issue to NN-AGP. To address the challenge, we also include a discussion on employing transfer learning technologies in the supplements. Other potential future work includes 1) adapting NN-AGP to multi-objective/constrained optimization problems and 2) employing NN-AGP in a federated contextual bandit problem with multiple decentralized users.
We thank the anonymous reviewers and chairs for constructive comments that helped improve this work. H. Zhang and J. He were partially supported by the Berkeley graduate fellowship. Z.J.M. Shen was partially supported by NSFC Grant 71991462. R. Righter is supported by the Ron Wolff Chaired Professorship.
[95] P. Xu, Z. Wen, H. Zhao, and Q. Gu. Neural contextual bandits with deep representation and shallow exploration. arXiv preprint arXiv:2012.01780, 2020.
[96] Y. Xu and A. Zeevi. Upper counterfactual confidence bounds: a new optimism principle for contextual bandits. arXiv preprint arXiv:2007.07876, 2020.
[97] H. Zhang, J. He, R. Righter, Z.-J. M. Shen, and Z. Zheng. Machine learning-assisted stochastic kriging for offline simulation online application. Working Paper.
[98] H. Zhang, J. He, D. Zhan, and Z. Zheng. Neural network-assisted simulation optimization with covariates. In 2021 Winter Simulation Conference (WSC), pages 1-12. IEEE, 2021.
[99] D. Zhou, L. Li, and Q. Gu. Neural contextual bandits with ucb-based exploration. In International Conference on Machine Learning, pages 11492-11502. PMLR, 2020.
[100] Y. Zhu, D. J. Foster, J. Langford, and P. Mineiro. Contextual bandits with large action spaces: Made practical. In International Conference on Machine Learning, pages 27428-27453. PMLR, 2022.