EE-Net: Exploitation-Exploration Neural Networks in Contextual Bandits
Authors: Yikun Ban
Yuchen Yan
Arindam Banerjee
Jingrui He
Venue: ICLR 2022 Spotlight (2022)
Keywords: Contextual Bandits
Exploration Strategy
Neural Networks

Abstract: In this paper, we propose a novel neural exploration strategy in contextual bandits, EE-Net, distinct from the standard UCB-based and TS-based approaches. Contextual multi-armed bandits have been studied for decades with various applications. To solve the exploitation-exploration tradeoff in bandits, there are three main techniques: epsilon-greedy, Thompson Sampling (TS), and Upper Confidence Bound (UCB). In recent literature, linear contextual bandits have adopted ridge regression to estimate the reward function and combine it with TS or UCB strategies for exploration. However, this line of works explicitly assumes the reward is based on a linear function of arm vectors, which may not be true in real-world datasets. To overcome this challenge, a series of neural bandit algorithms have been proposed, where a neural network is used to learn the underlying reward function and TS or UCB are adapted for exploration. Instead of calculating a large-deviation based statistical bound for exploration like previous methods,  we propose "EE-Net", a novel neural-based exploration strategy. In addition to using a neural network (Exploitation network) to learn the reward function, EE-Net uses another neural network (Exploration network) to adaptively learn potential gains compared to the currently estimated reward for exploration. Then, a decision-maker is constructed to combine the outputs from the Exploitation and Exploration networks. We prove that EE-Net can achieve $\mathcal{O}(\sqrt{T\log T})$ regret and show that EE-Net outperforms existing linear and neural contextual bandit baselines on real-world datasets. 

Published as a conference paper at ICLR 2022
EE-NET: EXPLOITATIION-XPPOORATOO NEURAL NETWORKS IN CONTEXTUAL BANDITS
Yikun Ban, Yuchen Yan, Arindam Banerjee, Jingrui He University of Illinois at Urbana- Chaampaiinn (y k yucheny5, arindamb, jingrui j @11llnno edu
ABSTRACT
In this paper, we propose novel neural exploration strategy in contextual bandits, EE-Net, distinct from the standard UCB- BCB-based and TS-based approaches. Contex- maa multi-armed bandits have been studied for decades with various applications. To solve the exploitation-exploration tradeoff in bandits, there are three main tech- niques: apsilon-greedd Thompson Sampling (TS), and Upper Confidence Bound (UCB). In recent literature, linear contextual bandits have adopted ridge regression to estimate the reward function and combine it with TS or UCB strategies for exploration. However, this line of works explicitly assumes the reward is based a linear function of arm vectors, which may not be true in real-world datasets. To overcome this challenge, series of neural bandit algorithms have been proposed, where a neural network is used to learn the underlying reward function and TS or UCB are adapted for exploration. Instead of calculating a large-deviation based statistical bound for exploration like previous methods, we propose "EE-Net", novel neural-based exploration strategy. In addition to using a neural network (Exploitation network) to learn the reward function, EE-Net uses another neural network (Exploration network) to adaptively learn potential gains compared to the currently estimated reward for exploration. Then, a decision-maker is constructed to combine the outputs from the Exploitation and Exploration networks. We prove that EE-Net can achieve (VV''og regret and show that EE-Net outperforms existing linear and neural contextual bandit baselines on real-world datasets.
INTRODUCTION
The stochastic contextual multi-armed bandit (MAB) (Lattimore and Szepesvári, 2020) has been studied for decades in machine learning community to solve sequential decision making, with applications in online advertising (Li et al. 2010). personal recommendation (Wu et al.) 2016 Ban and He 2021b), etc. In the standard contextual bandit setting, set of arms are presented to learner in each round, where each arm is represented by a context vector. Then by certain strategy, the learner selects and plays one arm, receiving reward. The goal of this problem is maximize the cumulative rewards of I rounds.
MAB algorithms have principled approaches to address the trade-off between Exploitation and Exploration (EE), as the collected data from past rounds should be exploited to get good rewards but also under-explored arms need be explored with the hope of getting even better rewards. The most widely-used approaches for EE trade-off can be classified into three main techniques: Epsilon-greedy
(Langford and Zhang 2008). Thompson Sampling (TS) [Thompson) 1933), and Upper Confidence
Bound (UCB) (Auer 2002 Ban and He 2020).
Linear bandits Li et al. 2010 Dani et al. 2008 Abbasi-Yadkori et al., 2011 where the reward is
assumed to be linear function with respect to arm vectors, have been well studied and succeeded both empirically and theoretically. Given an arm, ridge regression usually adapted to estimate its reward based on collected data from past rounds. UCB-based algorithms (Li et al. 2010; Chu et al. 2011 Wu et al. 2016 Ban and He 202 calculate an upper bound for the confidence ellipsoid of estimated reward and determine the arm according to the sum of estimated reward and UCB. TS-based algorithms (Agrawal and Goyal 2013) Abeille and Lazaric (017)) formulate each arm as posterior distribution where mean is the estimated reward and choose the one with the maximal

---

Published as a conference paper at ICLR 2022
(z:
Expected Reward
h(x)
Reward
fi (x;
g²
fa(Vi::
f2(V/:0*)
Eatimated Reward
(x(x;
Expected Reward
h(x)
Predict
Predict
fx(; 0+)
Case Upward Exploration
Case 2: Downward Exploration
Potential Gain
Exploitation Network f1
Exploration Network f2
Adaptive Erploitation-Exploraa f1 + f2
Figure 1: Left figure: Structure of EE-Net. In the right figure, Case 1: "Upward" exploration should be made when the learner underestimates the reward; Case 2: "Downward" exploration should be chosen when the learner overestimates the reward. EE-Net has the ability adaptively make exploration according to different cases. In contrast, UCB-bases strategy will always make upward exploration, and
strategy will randomly choose upward downward exploration.
sampled reward. However, the linear assumption regarding the reward may not be true in real-worll applications (Valko al. 2013).
To learn non non- linear reward functions, recent works have utilized deep neural networks to learn the underlying reward functions, thanks to its powerful representation ability. Considering the past selected arms and received rewards as training samples, neural network fi is built for exploitation. Zhou al. (2020) computes a gradient-based upper confidence bound with respect to fi and uses UCB strategy to select arms. ZZhang et al. (2021) formulates each arm as a normal distribution where the mean is and deviation is calculated based on gradient of f1, and then uses the TS strategy to choose arms. Both Zhou et al. (2020) and Zhang et al. (2021) achieve the near-optimal regret bound of D(VTlog?)) 7))
In this paper, we propose a novel neural exploration strategy, named "EE-Net". Similar other neural bandits, EE-Net has another exploitation network fi to estimate rewards for each arm. The crucial difference from existing works that EE-Net has an exploration network f2 predict the potential gain for each arm compared to current reward estimate. The input to the exploration network the gradient of /1 and the ground-truth is residual difference between the true received reward and the estimated reward from f1. The strategy is inspired by recent advances in the neural UCB strategies (Zhou et al. 2020 Ban et al., 202I). Finally, a arms. fo has two modes: linear nonlinear. In linear mode, fs is a linear combination of fi and fz. inspired by the UCB strategy. In the nonlinear mode, fa is formulated as a neural network with input (fi, f2) and the goal is to learn the probability of being an optimal arm for each arm. Figure I depicts the workflow of EE-Net and its advantages for exploration compared to UCB or TS-based methods (see more details in Appendix Dj. To sum up, the contributions of this paper can be summarized as follows:
fa is constructed to select
1. We propose a novel neural exploration strategy, EE-Net, where another neural network is assigned to learn the potential gain compared the current reward estimate.
2. Under standard assumptions of perrpparameterized neural networks, prove that EE-Net can achieve the regret upper bound of O (VVTlogg T), which improves multiplicative factor of vlog T and is independent of either input or effective dimension, compared to existing state-of-the-ar neural bandit algorithms.
3. We conduct extensive experiments on four real-world datasets, showing that EE-Net outper- forms baselines including linear and neural versions of e-greedy, TS, and UCB.
Next, discuss the problem definition in See 3 elaborate the proposed EE-Net in Sec 4l and present our theoretical analysis in Sec In the end, we provide the empirical evaluation (Sec and conclusion.
2

---

Published as a conference paper at ICLR 2022
2 RELATED WORK
Constrained Contextual bandits. The common constrain placed on the reward function is the linear assumption, usually calculated by ridge regression eLi al. 2010 Abbasi-Yadkori al. [01 Valko iet al.
2013 Dani et al. 2008). The linear UCB-based bandit algorithms (Abbasi- Yadkori et al. et al. 2016) and the linear Thompson Sampling (Agrawal and Goyal 2013 Abeille and Lazaric 2017) can achieve successful performance and the near-optimal regret bound of O(VT). To break the linear assumption, Filippi et al. (010) generalizes the reward function to composition of linear and non-linear functions and then adopt UCB-based algorithm to deal with it: Bubeek et al. (201T) imposes the Lipschitz property on reward metric space and constructs hierarchical optimistic optimization to make selections; Valko et al. (2013) embeds the reward function into Reproducing Kernel Hilbert Space and proposes the kernelized TS/UCB bandit algorithms.
2011
Neural Bandits. To learn non-linear reward functions, deep neural networks have been adapted to bandits with various variants. Riquelme al. (2018); Lu and Van Roy (2017) build L-layer DNN to learn the arm embeddings and apply Thompson Sampling on the last layer for exploration. Zhou et al.) (2020) first introduces provable neural-based contextual bandit algorithm with UCB exploration strategy and then Zhang et al. (2021) extends the neural network to Thompson sampling framework. Their regret analysis is built recent advances the convergence theory in ever-parameterized
neural networks Du et al.,
2019 Allen-Zhu et al. 2019) and utilizes Neural Tangent Kernel (Jacot
al. 2018 Arora al. 2019) construct connections with linear contextual bandits (Abbasi-
Yadkori et al. (2011) Ban and He (2021a) further adopts convolutional neural networks with UCB exploration aiming for visual-aware applications. Xu al. (2020) performs UCB-basee exploration on the last layer of neural networks to reduce the computational cost brought by gradient-based UCB. Different from the above existing works, EE-Net keeps the powerful representation ability of neural networks to learn reward function and first assigns another neural network to determine exploration.
3 PROBLEM DEFINITION
We consider the standard contextual aume bandit with the known number of rounds T (Zhou
ett al. 2020: Zhang et al. 2021). In each round t E [T], where the sequence [7] = [1, 2,...... TT. the learner is presented with n arms, X, (x., Nt, ,. in which each arm is represented by feature vector X,, Rdd for each i E [n]. After playing one arm X1,s, its reward rt, is assumed to be generated by the function:
T1,1 h(xx,;) 77..
(3.1)
where the unknown reward function (xxx,;) can be either linear or non-linear and the noise Tt is drawn from certain distribution with expectation [E[n,.]] = O. Following many existing works (Zhou et al. 2020 Ban et al., 2021 Zhang et al. (2021), we consider bounded rewards, Tr, E [a, b). For the brevity, we denote the selected arm in round f by x, and the reward received in by Te. The pseudo regret of T rounds is defined as:
RT E Σ (r) rc)
(3.2)
where E[r? X.] maxie[m] (xx..)) is the maximal expected reward in the round 6. The goal of this problem is to minimize Ry by certain selection strategy.
Notation. We denote by {x, i==1 the sequence ( x:). We use ||ull to denote the Euclidean norm for vector v, and W|2 and ||]| || to denote the spectral and Frobenius norm for matrix W. We use (:,-) to denote the standard inner product between two vectors two matrices. We may oo fi (xx,)) Vo fa represent the gradient Voz fa (xx, 0.)) for brevity. We use fx..rr to represent the collected data up to round l.
4 PROPOSED METHOD: EE-NET
EE-Net is composed of three components. The first component the exploitation network, fi 0¹). which focuses on learning the unknown reward function based on the data collected in past rounds.
3

---

Published as a conference paper at ICLR 2022 Table I Structure of EE-Net (Round t).
Input
Network
Label
{x, 1
fi(: 0') (Exploitation)
{r, }
, (x,
fi
0 ---1))=11
fe(: 0²) (Exploration)
{(r, - fi(s;; O7 0--1))
((1(xx;;==))::2f f1:
_ ())
fs(:;0)) (Decision-
{pr);-1
maker with non-linear function)
The second component is the exploration network, fe(:; 0²), which focuses on characterizing the level of exploration needed for each arm in the present round. The third component is the decision-maker, fa. which focuses on suitably combining the outputs of the exploitation and exploration networks leading to the arm selection.
Exploitation Net. The exploitation net fi neural network which learns the mapping from to rewards. In round denote the network by [i 0.-1). where the superscript of 0 1 the index of network and the subscript represents the round where the parameters of fi finished the last update. Given an arm xz.; i E [m]. fi (x,, f is considered the "exploitation score" " for xt.i- By some criterion, after playing arm Xx, we receive reward Tr. Therefore, we can conduct gradient descent update 0' based the collected training samples (x,; and denote the updated parameters by 0--
2) Exploration Net. Our exploration strategy is inspired by existing UCB-based neural bandits (Zhou et al. 2020 Ban al. 2021). Based on the Lemma 5.2 in (Ban al. 2021), given an arm Xt, with probability at least
o, we have the following UCB form:
1/ (x...) - fi (x 01-1)1 S V (Vo; fi 0 0)-)),
(4.1)
where h is defined in Eq. 33.1 and V is upper confidence bound represented by function with respect to the gradient Vo:_, f1 (see more details and discussions in Appendix D) Then we have the following definition.
Definition 4.1. In round given an arm X1,6, we define h(xx,,) :) - fi (x,, 0)-1) as the "expected potential gain " for x1,; and rt,s (x 0/_)) as the "potential gain for Xt
Let Vt.: = Tt,s (x 1)_1). When Vt, > 0, the arm Xt,s has positive potential gain compared to the estimated reward (x 0),1). A large positive Vt.; makes the more suitable for exploration, whereas a small (or negative) Vr.; makes the arm unsuitable for exploration. Recall that traditional approaches such UCB intend estimate such potential gain yr. using standard tools, e.g., Markov inequality, Hoeffding bounds, etc., from large deviation bounds.
Instead of calculating large-deviation based statistical bound for Dt, use neural network fe(:: 0²) represent V where the input is V (xx,)) and the ground truth is rı.: - ff (x (x 0!_1). 1)
Adopting gradient _o (x+,)) as the input also is due to the fact that it incorporates two aspects of information: the feature of the arm and the discriminative information of f1-
Moreover, in the upper bound of NeuralUCB or the variance of NeuralTS, there is a recursive term A,-1 = I+ E Vol (xx) oo (x,) which is a function of past gradients up to (t - 1) and incorporates relevant historical information. On the contrary, in EE-Net, the recursive term which depends on past gradients is 0²_ in the exploration network f2 because we have conducted gradient descent for 0²_1 based on { Vo:__, fi (x, )==)- Therefore, this form 0²_ is similar to A,-1 in neuralUCB/TSS but EE-net does not (need to) make specific assumption about the functional form of past gradients, and is also more memory-efficient.
To sum up. in round t, we consider fa(Vot_, fx(..))) 0.-1) as the "cxploration score" of X because it indicates the potential gain of xr,, compared to our current exploitation score ( 0 0!_1). Therefore, after receiving the reward we can use gradient descent to update Ø² based on collected training samples
(1 /1 (x+; 0² 0 We also provide other two heuristic forms
for f2 's ground-trutt -oound-truth label: |rr, fi (x,,; 0 0._,) and ReLU(rz, - (x 0/_))). We compare
them in ablation study in Appendix
4

---

Published as a conference paper at ICLR 2022
Algorithm 1 EE-Net
Input: f1. f2, fs. T' (number of rounds), 77 (learning rate for fi). 72 (learning rate for fe). 133 (learning rate for fa), K₁ (number of iterations for fi), K2 (number of iterations for f2). K3 (number of iterations for fs). & (normalization operator)
1: Initialize 0/ 0 0. 0 0 0 0 0 0
2: for t= 1, ...,.... T do
3: Observe n arms {xx, (x Xt,n
4: for each i E [n] do
5: Compute fi f 0 0
6: end for
7:
X arg maxx,, fa (is
8:
Play X₁ and observe reward
9: 0 0r 0:- GRADDENTIDESCENT(O (x+) {x) (r+))=))
10: end for
11:
procedure GRADIENTDESCENTOO, (x-)/=11 (rr)==))
13:
L1 L1 Σx 7- (f(xx=0')) - -r-)" 02.(0) 0
14:
15:
for l E {I...., Ki} do
g².(k) (k-(k=1)
772 11
17:
end for
0. (1((K))
18:
19:
L = 2 E
(ix.)V) - (((x-xx0))
@2.(0) 0¹¹
20:
21:
for E {I...., K2) do
g² (2-(1=-1) 1/2 02
22:
end for
0. 02(K)) (kg)
24:
25:
Determine label pt
26:
L3 1 El=1 1i=1 [pe log fa((fi f2): log - fz((fi,z):00)))
27:
0 0
28:
for h: E (I...,, Ka} do
ga.(k) (3.(k-1)
7/3 V ga.t L3
30:
end for
0 (Ks)
31:
Randomly choose (0). 0¹) uniformly from (0 00) (0) 00,00) ")... 00.....00,000 03) 33 -3 Randomly choose 0% uniformly from (0 0 "........ 0.1 Return 0%, 0², 0²
32:
33:
34:
35:
end procedure
3) Decision-maker. In round f,, given an arm Xt,s; E [n], with the computed exploitation score fi (x 0!-1) and exploration score fa (2(Vo,.. [i; 0/ we use a function fa (fi, f2; 0²) to trade off between exploitation and exploration and compute the final score for X,, The selection criterion defined as
X = arg max fo (fi(x..::)-1)) [(
Note that fa can be either linear or non-linear functions. We provide the following two forms. (1) Linear function. fa can be formulated as linear function with respect to fi and f2
Ja(S1, f2; 0²) wi (xx,1; 0t) waf2(Voi 0²)

---

Published as a conference paper at ICLR 2022
where Wi, W2 two weights preset by the leamne. When w1 UCB-type policy, where the estimated reward fi and potential gain fz are simply added together. In experiments, report its empirical performance in ablation study (Appendix (B).
w2 1, Is can be thought of as
(2) Non-linear function. fs also can be formulated as a neural network to learn the mapping from (fi, f2) to the optimal arm. We transform the bandit problem into binary classification problem. Given an arm X we define pt.; as the probability of being the optimal arm for x in round t. For brevity, we denote by pr the probability of being the optimal arm for the selected arm x, round t. According to different reward distributions, we have different approaches to determine pt.
1. Binary reward. 14 E [T]. suppose ri is a binary variable over a, b(a < b). it is straightfor- ward to set: pt = 1.0 if r2 b; Pt 0.0, otherwise.
2. Continuous reward. Vt [T), suppose ra is continuous variable over the range [a, b), provide two ways to determine pt- (1) Pt can be directly set as r--e_ b-a (2) The learner can set a threshold 0, (a < 0 < b). Then pt = 1.0 if > 0: Pt = 0.0, otherwise.
Therefore, with the collected training samples { (f
S1; ,1:00=1)) ,Pr ) ) in
round 1, can conduct gradient descent update parameters of fa(:; 0%).
Table I details the working structure of EE-Net. Algorithm dedpicts the workflow of EE-Net, where provides version of gradient descent (GD) to update EE-Net, where drawing (0), 0 uniformly from their stored historical parameters for the sake of analysis. One can easily extend EE-Net to stochastic GD to update the parameters incrementally.
the input of f2 is normalized, i.c., fi Algorithm !
Remark 4.1 (Network structure). The networks f2, f3 can be different structures according to different applications. For example, in the vision tasks, fi can be set up convolutional layers (LeCun et al. 1995). For the exploration network fz. the input V fi may have exploding dimensions when the exploitation network f1 becomes wide and deep, which may cause huge computation cost for /2. To address this challenge, we can apply dimensionality reduction techniques to obtain low- dimensional vectors of Vo fi. In the experiments, we use Roweis and Saul (2000) to acquire 10-dimensional vector for Vo fi and achieve the best performance among all baselines.
Remark 4.2 (Exploration direction). EE-Net has the ability to determine exploration direction. Given an arm Xt, when the estimation (xx,,) lower than the expected reward hxx,,)). the learner should make the "upward" exploration, i.e., increase the chance of Xt, being explored; When fi (x,,)) higher than h (xxx,s), the learner should do the "downward" exploration, i.e., decrease the chance of xx,; being explored. EE-Net uses the neural network f2 to lear h(x,,;) fx(x),)) (which has positive and negative scores) and has the ability to determine the exploration direction. In contrast, NeuralUCB will always make "upward" exploration and NeuralTS will randomly choose between "upward" exploration and "downward" exploration (see selection criteria in Table and more details in Appendix D).
Remark 4.3 (Space complexity). NeuralUCB and NeuralTS have to maintain the gradient outer product matrix (e.g., A Vou (o V O+) E RPxP)) and, for 0² E R2, have a space complexity of O(p²) to store the outer product. On the contrary, EE-Net does not have this matrix and only regards V fi as the input of f2. Thus, EE-Net reduces the space complexity from Op²2) to O(p).
REGRET ANALYSIS
In this section, we provide the regret analysis of EE-Net when fa is set as the linear function fa = fi + f2, which can be thought of as the UCB-type trade-off between exploitation and exploration. For the sake of simplicity, we conduct the regret analysis on some unknown but fixed data distribution
D. In each round f, n samples ((x,,1 r(,1). (x,.2, r1,2)...., (Xi... (t,,)) are drawn i.i.d. from D. This is standard distribution assumption in over- paraameeeeeee neural networks (Cao and Gu 2019). Then, for the analysis, we have the following assumption, which a standard input assumption in
neural bandits and peratttreeter neural networks et al. 2020: Allen-Zhu et al. 2019)

---

Published as a conference paper at ICLR 2022
Assumption 5.1 (p-Separability). For any t [T] i E [n].
xxall2 and ft,i E [0, 1]. Then, for [t, 19.5-x0 -xr,-l > p. and suppose there
every pair Xt,;; st, E [T]. E [k], and (t, 1) + (t',
exists an operator such that |
and Ve (xx,)) fi 1)
For example, the operator can be designed as o = /all fa axl
X The
analysis will focus on over-vvraane wer-parameterized neural networks (Jacot et al. 2018 Du et al. 2019 Allen
Zhu et al. 2019). Given an input Rd, without loss of generality, define the full fully-connected network with depth L 2 2 and width mn:
[(x: 0) (. L-2.....000 o....))
(5.1)
where o is the ReLU activation function, Wi E Rmxd W. E R=x"" for 2 2 L WL E 121xm and 0 [[vec[ (W)) T), (ec) ,.... (oc(WL))]
1,
Initialization. For any / E [L 1]. each entry of W. : is drawn from the normal distribution (N - and W, is drawn from the normal distribution ((, ).) Note that EE-Net at most has three networks f1. f2. Ja- We define them following the definition of f for brevity, although they may have different depth or width. Then, we have the following theorem for EB-Net. Recall that m1, 722 are the learning rates for fi, f2; K1 is the number of iterations of gradient descent for fi in each round; and K2 is the number of iterations for fz.
Theorem 1. Let fı, f2 follow the setting of f (Eq. (5.1) with the same width m and depth L. Let L1 L2 be loss functions defined in Algorithm 7 Set J3 as fs fi f2. For any 8 E (0, 1),1),c
(0, O p (0, O( }], suppose
m 2 π poll([T, n, L, nn
1og((10) ev /log(T//8))
m 1/2 min (8)
T
(paty(T P (poly(T, n, L.) 1.)mm/
V282m
(5.2)
(ool(TTn,L)) log (e (E-1)
K1 Kg =
p8²
Then, with probability at least - 8 over the initialization, the pseudo regret of EE-Net in T rounds satisfies
(
Rr S 0011+ (2VT 1)3 (2VT (2VT-1)V 1) 2
(5.3)
Comparison with existing works. Under the similar assumptions in over-parameterized neural networks, the regret bounds complexity of NeuralUCB (Zhou et al. 2020) and NeuralTS ZZang et al. 2021) both are
R1 S SO OVV logT o (VVvlogg) and d d=
log det(I + H/A)
log( 1 Tn/A)
where is the neural tangent kernel matrix (NTK) (Jacot et al. 2018 Arora et al. 2019) and À is regularization parameter. Similarly, in linear contextual bandits, Abbasi- Yadkori et al. (201 achieve
OddTT Tog T' and et al. (2017) achieve O (VVTTlog)))
Remark 5.1. Compared to NCuralUCB/TS. EE-Net roughly improves by a multiplicative factor of Vlog T, because our proof of EE-Net is directly built on recent advances in convergence theory AAlen-ZZhu et al. 2019) and generalization bound (Cao and Gul 2019) of wver-prameterrzee neural networks. Instead, the analysis for NeuralUCB/TSS contains three parts of approximation error by calculating the distances between the expected reward and ridge regression, ridge regression and NTK, and NTK and network function.
Remark 5.2.
The regret bound of EE-Net does not have the effective dimension d or input dimension d. d or d may cause significant error, when the determinant of H extremely large or T.
The proof of Theorem T is in Appendix C and mainly based the following generalization bound. The bound results from an ouline-to-batedd conversion while using convergence guarantees of deep learning optimization.
7

---

Published as a conference paper at ICLR 2022
Lemma 5.1. For any ( (0,1), p. (0,0(H)), suppose m. 11. T2. K1, K2 satisfy the conditions in Eq. (.2 and (X,,i, (r,1) ~ D, [t], [n] Let
fz [V fi ix 01-1)):0 0P-) f fi(x,,;; [e
X₁ =
max
and is the corresponding reward, given (xt,i, rr,)), [n]] Then, with probability at least (1 - 8) over the random of the initialization, holds that
(x) [15o (o(Voo. 0:_ (x (2.a; -(r fi(x;; 0 0 1xx,
(5.4)
2e
3L
2 2log( log(O((n/6))
S SV + +0
+ +1++ 2
V21
t
where the expectation is also taken over (0;-1, 12-1) that are uniformly drawn from (0 0 E [t-1] 1).
Remark 5.3. Lemma 5.1 provides fixed (1 generalization bound for exploitation-
exploration networks fi. in contrast with the relative bound w.r.t. the Neural Tangent Random Feature (NTRF) benchmark (Cao and Gu 2019). We achieve this by working in the regression rather than classification setting and utilizing the convergence guarantees for square loss (Allen-Zhu et al. 2019). Note that the bound in Lemma 5. holds in the setting of bounded (possibly random) rewards [0, 1] instead fixed function in the conventional classification setting.
EXPERIMENTS
In this section, evaluate EE-Net on four real-world datasets comparing with strong state-of-the-art baselines. We first present the setup of experiments, then show regret comparison and report ablation study. Codes are available at
We use four real-wor real-world datasets: Mnist, Yelp, Movielens, and Disin, the details and settings of which are attached Appendix
Movielens
Yelp
Kernalucn
1752
LinüCB
UnACB
1500
Meural
NourolTS
3000
1250
1000
750
500-
1000
250
2000
4000
8000
5000
10000
2002
4000
5000
8000
12020
Rounds
Rounds
Figure 2: Regret comparison on Movielens and Yelp (mean of 10 runs with standard deviation (shadow)). With the same exploitation network f1, EE-Net outperforms all baselines. Baselines. To comprehensively evaluate EE-Net, we choose 3 neural-based bandit algorithms, one linear and one kernelized bandit algorithms.
1. LinUCB (Li al. 2010) explicitly assumes the reward is a linear function of arm vector and unknown user parameter and then applies the ridge regression and un upper confidence bound to determine selected arm.
2. KernelUCB Valko al. 2013) adopts predefined kemel matrix the reward space combined with a UCB-based exploration strategy.
3. PesrrllEpsillon adapts the seailon-greedy exploration strategy on exploitation network f1- Le.. with probability 1 - e, the arm is selected by x; = arg maxie(1] (xx,, 0²) and with probability e, the arm chosen randomly.
8

---

Published as a conference paper at ICLR 2022
Mnist
Disin
Kemalucce
1750
1750
Unuca
1500
1500
Neuralts
1250
1250
Ragret
2000
1000
500
heural-assion
500
heuralts
250
haunalice
250
EC-hie:
2000
4000 6000
6000
20000
2000
4000
6000
6000
10000
Bounds
Rounde
Figure 3: Regret comparison on Mnist and Disin (mean of 10 runs with standard deviation (shadow)).
With the same exploitation network f1. EE-Net outperforms all baselines.
4. NeuralUCB (Zhou et al. 2020) uses the exploitation network fi to learn the reward function coming with an UCB- CCB-based exploration strategy.
5. NeuralT'S (Zhang et al. coming with an Thompson Sampling exploration strategy.
2021) adopts the exploitation network f1 to learn the reward function
Note that do not report results of LinTS and KemnlTTS in experiments, because of the limited space in figures, but LinTT and KernelTSS have been significantly outperformed by NeuralTSS (Zhang et al. 2.
To compare fairly, for all the neural-based methods including EE-Net, the exploitation network fi is built by a 2-layer fully-connected network with 100 width. For the exploration network f2, we use a 2-layer fully-connected network with 100 width as well. For the decision maker fs. by comprehensively evaluate both linear and nonlinear functions, we found that the most effective approach is combining them together, which we call hybrid decision maker". In detail, for rounds < 500, f' is set as fa = fz + fi, and for >> 500. fa set as a neural network with two 20- (iith fully-connected layers. Setting Ja in this way is because the linear decision maker can maintain stable performance in each running (robustness) and the non-linear decision maker can further improve the performance (see details in Appendix B). The hybrid decision maker combine these two advantages together. The configurations of all methods are attached in Appendix A
Setup for EE-Net.
Results. Figure 2 and Figure Show the regret comparison on these four datasets. BE-Net consistently outperforms all baselines across all datasets. For LinUCB and KernelUCN, the simple linear reward function or predefined kernel cannot properly formulate ground-truth reward function existed in real-world datasets. In particular, on Mnist and Disin datasets, the correlations between rewards and arm feature vectors not linear or some simple mappings. Thus, LinUCB and KernelUCB barely exploit the past collected data samples and fail to select correct arms. For neural-based bandit algorithms, the exploration probability of Neural-Epsilon is fixed and difficult be adjustable. Thus it is usually hard to make effective exploration. To make exploration, NcuralUCB statistically calculates gradient- pradee upper confidence bound and NeuralTS draws each arm's predicted reward from normal distribution where the standard deviation is computed by gradient. However, the confidence bound or standard deviation they calculated only consider the worst cases and thus may not be able represent the actual potential of each arm, and they cannot make "upward" and "downward" exploration properly. Instead, EE-Net uses a neural network fa to learn each arm's 'sss potential by neural network's 's powerful representation ability. Therefore, EE-Net can outperform these two state-of-the- bandit algorithms. Note that NeuralUCB/TS does need two parameters to tune UCB/TS according to different scenarios while EE-Net only needs to set up neural network and automatically learns it.
A Ablation Study. In Appendix B conduct ablation study regarding the label function y of fa and the different setting of fa-
7 CONCLUSION
In this paper, we propose novel exploration strategy, EE-Net. In addition to a neural network that exploits collected data in past rounds, EE-Net has another neural network to learn the potential gain compared to current estimation for exploration. Then, a decision maker built to make selections to further trade off between exploitation and exploration. We demonstrate that EE-Net outperforms NeuralUCB and NeuralTS both theoretically and empirically, becoming the new state-offie of-tc-----h-rrr the-art exploration policy.

---

Published as a conference paper at ICLR 2022
Acknowledgements: We grateful Shiliang Zuo and Yunzhe Qi for the valuable discussions in the revisions of EE-Net. This research work is supported by National Science Foundation under Awards No. IIS- 194720370 IIS-2002540, IIS-2137468, IIS- 1SS-1901144, OAC-1934634, and DBI- 2021898, and a grant from C3.ai. The views and conclusions are those of the authors and should not be interpreted representing the official policies of the funding agencies or the government.
REFERENCES
Y. Abbasi-Yadkori, D. Pál, and C. Szepesvári. Improved algorithms for linear stochastic bandits, In Advances in Neural Information Processing Systems, pages 2312-2320, 2011.
M. Abeille and A. Lazaric. Linear thompson sampling revisited. In Artificial Intelligence and Statistics, pages 176-184. PMLR, 2017.
S. Agrawal and N. Goyal. Thompson sampling for contextual bandits with linear payoffs. In International Conference on Machine Learning, pages 127-135. PMLR, 2013.
H. Ahmed, I. Traore, and S. Saad. Detecting opinion spams and fake news using text classification. Security and Privacy, 1(1):e9, 2018.
Z. Allen-Zhu, Y. Li, and Song. convergence theory for deep learning via per-paaamcttiizaiion.. In International Conference on Machine Learning, pages 242-252. PMLR, 2019.
S. Arora, S. S. Du, W. Hu, Z. Li, R. R. Salakhutdinov, and R. Wang. On exact computation with an infinitely wide neural net. In Advances Neural Information Processing Systems, pages 8141-8150, 2019.
Auer. Using confidence bounds for exploitation-exploraaiin trade-offs. Journal of Machine Learning Research, 3(Nov):397-422, 2002.
Y. Ban and J. He. Generic outlier detection in multi-armed bandit. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 913-923, 2020.
Y. Ban and J. He. Convolutional neural bandit: Provable algorithm for visual-aware advertising. arXiv preprint arXiv:2107.07433: 2021a.
Y. Ban and He. Local clustering in contextual multi-armed bandits. In Proceedings of the Web Conference 2021, pages 2335-2346, 2021b.
Y. Ban, J. He, and C. B. Cook. Multi-facet contextual bandits: A neural network perspective. In The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, Singapore, August 14-18, 2021, pages 35-45, 2021.
S. Bubeck, R. Munos, G. Stoltz, and C. Szepesvári. X-armed bandits. Journal of Machine Learning Research, 12(5), 2011.
Y. Cao and Q Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. Advances in Neural Information Processing Systems, 32: 12836-100846, 2019.
N. Cesa-Bianchi, A. Conconi, and C. Gentile. On the generalization ability of on-line learning algorithms. Advances in neural information processing systems, 14, 2001.
E. Chlebus. An approximate formula for a partial sum of the divergent p-series. Applied Mathematics Letters, 22(5):732-737, 2009.
W. Chu, L. Li, Reyzin, and R. Schapire. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 208 224, 2011.
V. Dani, T. P. Hayes, and S. M. Kakade. Stochastic linear optimization under bandit feedback. 2008.
S. Du, J. Lee, H. Li, L. Wang, and X. Zhai. Gradient descent finds global minima of deep neural networks. In International Conference Machine Learning, pages 1675-1685. PMLR, M, 2019.
10

---

Published as a conference paper at ICLR 2022
S. Filippi, O. Cappe, A. Garivier, and C. Szeeesvári. Parametric bandits: The generalized linear case. In Advances in Neural Information Processing Systems, pages 586-594, 2010.
D. Fu and J. He. SDG: A simplified and dynamic graph neural network. In SIGIR '21: The 44th International ACM SIGIR Conference Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 2273-2277, ACM, 2021.
F. M. Harper and J. A. Konstan. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5(4):1-19, 2015.
A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization neural networks. In Advances in neural information processing systems, pages 8571-8580, 2018.
J. Langford and T. Zhang. The epoch-greedy algorithm for multi-armed bandits with side information. In Advances in neural information processing systems, pages 817-824, 2008.
T. Lattimore and C. Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
Y. LeCun, Y. Bengio, et al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10):1995, 1995.
Y. LeCun, I.. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 36(11):2278-2333 1998.
L. Li, W. Chu, J. Langford, and R. E. Schapire. ceatextual-banddee approach personalized news article recommendation. In Proceedings of the 19th international conference on World wide web. pages 661-670, 2010.
Li, Y. Lu, and D. Zhou. Provably optimal algorithms for generalized linear contextual bandits. In International Conference Machine Learning, pages 2071-2080 PMLR, 2017.
S. Li, A. Karatzoglou, and C. Gentile. Collaborative filtering bandits. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 539-548, 2016.
X. Lu and Van Roy. Ensemble sampling. arXiv preprint arXiv: 2017.
C. Riquelme, G. Tucker, and J. Snoek. Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling. arXiv preprint arXiv:1802.09112, 2018.
S. T. Roweis and L. K. Saul. Nonlinear dimensionality reduction by locally linear embedding. science, 29065500);2323-22222 2000.
W. R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285-294 1933.
M. Valko, N. Korda, Munos, Flaounas, and N. Cristianini. Finite-time analysis of kernelised contextual bandits. arXiv preprint 1rXiv:1309.6869 2013.
Q Wu, H. Wang, Q. Gu, and H. Wang. Contextual bandits in collaborative environment. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 529-538, 2016.
Xu, Z. Wen, Zhao, and Q Gu. Neural contextual bandits with deep representation and shallow exploration. arXiv preprint arXiv:2012.01788, 2020.
W. Zhang, D. Zhou, L. Li, and Q Gu. Neural thompson sampling. In International Conference Learning Representations, 2021.
D. Zhou, L. Li, and Q. Gu. Neural contextual bandits with ucb-based exploration. In International Conference Machine Learning, pages 1 11492-1 1500.. PMLR, 2020.
11

---

Published as a conference paper at ICLR 2022
A DATASETS AND SETUP
MNIST dataset. MNIST well-known image dataset (LeCun et al., 1998) for the 10-class classification problem. Following the evaluation setting of existing works (Valko et al. 2013 Zhou et al. 2020; Zhang et al., 202T), we transform this classification problem into bandit problem. Consider image x E R. we aim to classify it from 10 classes. First, in cach round, the image x is transformed into 10 arms and presented to the learner, represented by 10 vectors in sequence X1 = (x ,..,, 0), X2 (0, ....,. 0).
................................ X10 (0, ...... x) R.odd The reward is defined as 1 if the index of selected arm matches the index of x's ground- truth class; Otherwise, the reward is O.
Yelp and and Movielens (Harper and Konstan 2015) datasets. Yelp is dataset released in the Yelp dataset challenge. which consists of 4.7 million rating entries for 1.57 x 10 restaurants by 1.18 million users. MovieLenn is dataset consisting of 25 million ratings between 1.6 x 10 users and 6 x 10s movies. We build the rating matrix by choosing the top 2000 users and top 10000 restaurants and use singular-value decomposition (SVD) to extract 10-dimension feature vector for each user and restau)an((movie). In these two datasets, the bandit algorithm is to choose the restairants(movies) with bad ratings. We generate the reward by using the (oovssss gained stars scored by the users. In each rating record, if the user scores a restaurant (5 stars totally). ). its reward is I; Otherwise, its reward O. In each round, we set 10 arms as follows: we randomly choose one with reward and randomly pick the other 9 intaurants(movves)) with O rewards; then, the representation of each arm is the concatenation of corresponding user feature vector and restaurant(movie) feature vector.
less than 2 stars
Disin (Ahmed et al. 2018) dataset. Disin is a fake news dataset on kaggle inclluding 12600 fake news articles and 12600 truthful news articles, where each article is represented by the text. To transform the text into vectors, we use the approach (Fu and He (202I) to represent each article by a 300-dimension vector. Similarly, form a 10-arm pool in each round, where 9 real news and fake news are randomly selected. If the fake news is selected, the reward is 1; Otherwise, the reward is O.
Configurations. For LinUCB, following (Li et al., 2010), we do a grid search for the exploration constant a over (0.01, 0.1, 1) which is to tune the scale of UCB. For KernelUCB (Valko et al. 2013). we use the radial basis function kernel and stop adding contexts after 1000 rounds, following Valko iet al. 2013 Zhou et al. 2020). For the regularization parameter À and exploration parameter // in KernelUCB, we do the grid search for À over (0.1, 1, 10) and for v over (0.01, 0.1, 1). For NeuralUCB and NeuralTS, following setting of (Zhou et al. 2020: Zhang et al. (021), we use the exploiation network f1 and conduct the grid search for the exploration parameter over (0.001, 0.01, 0.1, 1) and for the regularization parameter À over (0.01, 0.1, 1). For NeuralEppsilon, we use the same neural network S1 and do the grid search for the exploration probability e over (0.01, 0.1, 0.2). For the neural bandits NeuralUCB/TS, following their setting, they have expensive computation cost to store and compute the whole gradient matrix, we use a diagonal matrix to make approximation. For all neural networks, we conduct the grid search for learning rate over (0.01, 0.001, 0.0005, 0.0001). For all grid-searcher parameters, we choose the best of them for the comparison and report the averaged results of 10 runs for all methods.
Create exploration samples for f2- When the selected arm is not optimal in a round, the optimal arm must exist among the remaining arms, and thus the exploration consideration should be added to the remaining arms. Based on this fact, we create additional samples for the exploration network fe in practice. For example, in setting of binary reward, e.g., or reward, if the received reward = 0 while select x,, we add new train samples for fz, (X,,i, )) for each i E [i] (x,, o x,, where C, E (0, 1) usually is a small constant. This measure can further improve the performance of EE-Net in our experiments.
B ABLATION STUDY
In this section, we conduct ablation study regarding the label function for exploration network f2 and seting of decision maker fa on two representative datasets Movielens and Mnist.
hhttps://wwer yelp. www. dataset www.://www...ommmmmmmbboooaaaaaa
12

---

Published as a conference paper at ICLR 2022
2000
2000
EE Net--RLU
EE Net ReUU
1750
EE abs
1750
EE Not abs
28-Net
Et-MMet
nagret
1000
2000
750
750
500
2000
4000
5000
8000
10000
2000
4003
6000
8000
10000
Bounds
Rounds
Figure 4: Ablation study on label function for f2- EE-Net denotes -r---- h. BE-Net-abs denotes y/- |r fil. and EE-Net-ReLL denotes ya ReLU(r fi). EE-Net shows the best
performance on these two datasets.
3000
3000
RE-Ne--NeLnn
FE-Met-NoUn EE Un BE-Mul
Not Un BE-Nut
2500
2500
2000
2000
Rejeet
1500
1500
2000
4000
6000
8000
10000
0
2000
4000
6000
8000
10000
Rounds
Rounds
Figure 5: Ablation study on decision maker f3. EE-Net-Lin denotes fa fi + f2, Ho-NeetNoLin denote the nonlinear one where fa is neural network (2 layer, width 20), EE-Net denotes the hybrid one where fs = fi + f2 if < 500 and fa is the neural network if f > 500. EE-Net has the most stable and best performance.
Label function y. In this paper, we use f1 -- f1 to measure the potential gain of an arm, as the label of f2. Moreover, we provide other two intuitive form Ir- fil and Figure shows the regret with different y, where "EE-Net" denotes our method with default y1- "EE-Net-abs" represents the one with y/2 and "EE-Net-ReLU" is with y/3 On Movielens and Mnist datasets, EE-Net slightly outperforms EE-Net-abs and EE-Net-ReLU, In fact, // can effectively represent the positive potential gain and negative potential gain, such that f2 intends to score the arm with positive gain higher and score the arm with negative gain lower. However, 32 treats the positive/negative potential gain evenly, weakening the discriminative ability. ya recognize the positive gain while neglecting the difference of negative gain. Therefore, usually is the most effective one for empirical performance.
ReLU f1).
Setting of fa. Sa can be set as an either linear function or non-linear function. In the experiment, we test the simple linear function fa = fi + f2, denoted by "BB-Net-Lin", and non-linear function represented by 2-layer 20-width fully-connected neural network, denoted by "EE-Net-NoLin" For the default hybrid setting, denoted by "EE-Net", when rounds < 500, fs = f1 + f2; Otherwise, fs DigureBrreports repoots the regret with these three different modes. EE-Net achieves the best performance with small standard deviation. In contrast, EE-Net-NolLin Ln obtains the worst performance and largest standard deviation. However, notice that EE-Net-NoLin can achieve the best performance in certain running (the green shallow) but is erratic. Because in the begin phase, without enough training samples, EE-Net-NoLin strongly relies on the quality of collected samples. With appropriate training samples, gradient descent can lead fa to global optimum. On the other hand, with misleading training samples, gradient descent can deviate fs from global optimum. Therefore, EE-Net-NoLLig shows very unstable performance. In contrast, EE-Net-Lin is inspired by the UCB strategy, i.e., the exploitation plus the exploration, exhibiting stable performance. To combine their
is the neural network. Figure
13

---

Published as a conference paper at ICLR 2022
advantages together, propose the hybrid approach, EE-Net, achieving the best performance with strong stability.
C PROOF OF THEOREM
In this section, we provide the proof of Theorem Ind related lemmas.
Proof. For brevity, for the selected arm x, in round t, let h(x;) be its expected reward
and x arg diel (xxx,;) be the optimal arm in round Let (x(x00=11
f2 ([Voo__
1 + fx;
Note that (x (r,;)) ~ D, for each E [n]. Then, the expected regret of round is given by Rs = ii [n] [h(x;) h(xx)]
(i [m] (h(x;) -s((x)) + fx() ) M(x+)]
S E (s [h(x;) fa(x;) + fx((x) - fa(x)) +fz(x.) - 1(x+)]
7
ie [h(xt) fa(x;) + fs(x,) h(xx))
Ee [h(x]) fa(x;; + fs( 11-1) h(xx)l
(a) ix in] (x(x7)
fa(x1: 07-1) + (x(x; 12-1) - sx(x7; 02-1) + fa(x;; 0.-1) h(xx))
ix [m] [h(x,) fa(x;; 11-1) + E (e(m) [f(( fa(x;;; 0,-1)
+ Ex i in) Ifs (x:: 01-1) - -(xx)]
fz (o(vo.: fi( 0 01:11) (ri fi 0
E (.E.
[12 (0170 fa (d(vo) (f((x;001)):;
+ Ex..ie(m) [xt( (x00))-((x;;;0000 (1(x;
Te (olvo), fixx 0 (( - (re- ((xxx( (x).))))
Jn
(C.1)
where 1, is because fa(x)) maxielm] fa (x(,)) and fa(x.) - fa(xi) 2 0 and (a) introduces the additional parameters 0-1 (0)- 02²) which will be suitably chosen.
Because, for each E [n]. (Xr,,, Te.i) ~ D, applying Lemma C. and Corollary C.I with probability
least (1 6) over the randomness of initialization, for /2, Is. have
T Is < V
2E
+
+
3L
+ (1 28)
2 loog(O (tn))())))8))
(C.2)
/2t
where
(nnI.lo m
1nn log 11/6 p+/ 33mmmmm m/m/6
E - o
+
(C.3)
Pvm
and apply the union bound over round 7, VT each round T, E [t].
[4] make Lemma C.I and Corollary C.I hold for
For 13, Lt based on Lemma
with probability at least 8, we have
(LL log 5/6 m
)
Lt3
(+t { log 11/6 m
1a, 14 <
1+0
0
log m
+ O
= E1. (C.4)
//1/3m1/6
pym
p4/3
14

---

Published as a conference paper at ICLR 2022
To sum up, with probability at least - 6, we have
(
3L
2 2log( 2log(O(tn)/0) +1 t
R₁ 2
+ +1++25)
(C.5)
v2F
6
Then expected regret of T rounds computed by
T
R
R4
t=1
22
3L
2 llog(O(tn)/(
2
1
+ O
( V2t
+ 1
+
t=1
$ (2VT 1)2v + (2VT 1)3v20(L) + 2(1 + 22) -1) - -1/2oo/001//)) (aap))))))+000 +0(1)
= -2VV - -) 33 (v)) + 2(1 1) O(1) (C.6)
where 12 is because Σ 1
v dr + -1=2
Chlebus 2009 and the bound & is
due to the choice of m, i.e., since & (1 (1/m²//0) and m 2 (ppolyy (T)), can be chosen so that
T& OTT/mm/////m 4/0 < ((1).
Then, when c 1 1/T,, we have
R < 0(1) (2vT 1)3 2(1 - As the choice of m, we have { $ ((1) Therefore, we have
(C.7)
Rr S 0(1) (2vT () (12/T-1)22oo(((TTT)) (220)//0000((((1---////00011)/))
(C.8)
The proof is completed.
Lemma C.1. (Lemma 5.1 restated] For any 5, E (0, 1),11, p E (0,0())), suppose m, m, 12, K1, K₂ satisfy the conditions in Eq. (5.2) and (X,,i, D. Vr E [1], E [n]. Let
f2 (6(Vo (x 0 01))) - (x 01-))
X arg
max [
and rt is the corresponding reward, given (xt,i, ri,)), [n]. Then, with probability least 5) over the random of the initialization, holds that
E
[[22 (12(vVV (VVo [iloo,,,,(x(x (1(xxx))_ 02:
(re fx 0 0111) (x+,
(C.9)
$ 2/2 + 0
2e
3L
2log(O((nn/))
+ + + +2)
22
where the expectation is also taken over (0)_1, P.-1) that are uniformly drawn from (0), 0²), E
1).
Proof. In this proof, we consider the collected data of up round t 1, fx,, rr 1 as the training
dataset and then obtain generalization bound for it, inspired by Cao and Gu (2019).
For convenience, we use Xt,s, r := Ft,s, noting that the same analysis holds for each [n]. Consider the exploration network fz. applying Lemma C.3 E [t). we have
With probability at least - 8, for any
f2 ((x))))) S.
(C.10)
Similarly, applying Lemma C.3 again, with probability at least 6. for any 1 E (T), we have
|]; (xx;0,)1 0 S
(C.11)
15

---

Published as a conference paper at ICLR 2022
Because for any Dr |r| S with Eq. (C.10) and (C.11 applying union bound, with probability at least (1 - 26) over the random initialization, we have
fa (ixx (r -r- -x;(x(x)))) 11+ 2
(C.12)
Noting that C.I22 is for x X,,i for specific E [t]. E [n]. By union bound, (C.12
holds VT E 11,i E [n] with probability at least - (1-nlō). For brevity, let f2(x; 0,) represent 12 ( / (x; 0+)
~2 Recall that, for each T [t 11. 6- and 0 are the parameters training on {x,, Fr' 17-=1 according
Algorithm In round [t], let x, arg (axx....eEjv)) [/1 (x+,;; 0- 0/-1) + f2(xx,;; 0 0 )} given
(Xr,i: D, E [n]. Let r, be the corresponding reward. Let (x+ (T) D, E [n] be shadow
samples from the same distribution and let x arg maxx Jel (x((x 0- -11)) + f2(x 0² 1]] with r being the corresponding reward. Then, we define
E (0=(x);;0-1>> -(rt 1- -x(x(; iii
Vj
(C (aniiii
(C.13)
- -112(x;:00=)) 0²=10 -0--))(r++- (ri-ii ((xx;;0-11) 0--1))
Then, as (Xr.i, (T.s) ~ D. E [m]. based on the definition of (x+, r7); we have
[| (x 0=:: (r+ frx= /xxx [F-1]
B[V,IF,-1]/
E (x ax a.
[Is (x-: 2 04::07-)) ( (x=-((xx:0/11)) | [-
(C.14)
E aa..
where F F--1 denotes the r-algebra generated by the history H--1 (x
Moreover, we have
Σ V
Σ
E
-11- -1 ((x):---)))
1
fz xri 0=;00,))) -
(r, ( -rxx- f((x):0==1))
Since (V+)-= is a martingale difference sequence, inspired by Lemma in 2001), applying the Hoeffding-Azune AAzuaa inequality, with probability least 30, we have
et al.
2 2log(1/0)
P
V
E E
F. (1+2))
S 8
1.
(C.15)
21og(1/5)
P
Vr > (1 2
<8,
where 7 0 according (C.14) and T is because of C.12
16

---

Published as a conference paper at ICLR 2022
(0'_1; 12-1) ) is uniformly drawn from (0)), 0 0/)=0=00 Thus, with proba-
According to Algorithm bility 1 36, we have
E ( Laleel (0)_, 0?_,) [|52 (x/; 02-1) - (xx;
1
(xt a (...Ee ).EE))) f2 (x4:0 (1' f1(x;; 0,-1)
t
(C.16)
f2 X (rri0 - -r- (T=-Si(x;:7-1)) +(1 (1++20)
2 2
t
For I3, according to Lemma C.6. for any o² satisfying 10 -0%12 0 ppmm logm), with probability
1 - 6, we have
fz(x;: 0 0²-1)= ( (11x)::0-1))]
fi(x;
21 f2(x;; (r+ fi fi 0-1))] +0
3L
(C.17)
222
T=1
For I,, according to Lemma C.4 (1), these exists Ø² satisfying 110² - -b8ll2 - 0 ( log m)) with
probability 1 - 8, such that
[2(x=;0') (r- (x (x1;;---1)]
Vt
Σ (Ja(x):0')- - fi(xxi )ا-1))
(C.18)
I6
2e $ Vi
where Is follows by a direct application of Lemma C.4 (1) by defining the loss L(0²)
Et= (fe(xri0) rr, fx 0 )-=))) <
Combining Eq. C..6 Eq. C.17 and Eq. C.18), with probability (1 (1)56) we have
(e.) [[ (x:: 0
(rx
(C.19)
2e
3L
2 log(1/8)
$
0
(1+ 2
2t
where the expectation over (0)_,, 0.-)) that is uniformly drawn from (00 0- 1-1
Then, applying union bound to t, n and rescaling the complete the proof.
Corollary C.1. For any ô,c (0, 1) E (0, (0)())), suppose m. mh 72, K1, K2 satisfy the condi- tions in Eq. 5.2) and (x,))) D, VT E [1] i [n]. For any T E [t], let
x = arg max [h(x+,:)] f
17

---

Published as a conference paper at ICLR 2022
and r is the corresponding reward, given (r.), E [n]. Then, with probability least (1 5) over the random of the initialization, there exist 0.10, 02.0, s.t,, 110 - 0bll2 I O lat pvm log m)
and |10,-1 - 0/112 - O pvm log m) such that
E (1)
f2 ( D:= f1 0111:
(r² - f1;; ((xx;;0221))
{x (x (
2F
3l
2log( (l(n/)))
S V + O
+ (1 + +1++
t
(C.20)
where the expectation is also taken over (0) 1221) that are uniformly drawn from @0, 0²",, 1).
) E
Proof. This direct corollary of Lemma C.1 given the optimal historical pairs {xt, +:]+] For brevity, let fx: 0- represent f2 (0 Vo (x; 0- )):0,
2 are the parameters training on r Suppose that, for each E - 11, 0 and
according to Algorithm I Note that these pairs { are unknown to the algorithm we run, and the parameters (0, 2.a 0²") are not estimated. However, for the analysis, it is sufficient to show that there exist such parameters so that the conditional expectation of the error can be bounded.
In round it), let x = arg maxx, (Ej])) [h(x,,))l given (Xr.i, Tr.i) ~ D, E [n]. Let r be the corresponding reward. Let (x,, rr,)) ~ D, i E [n] be shadow samples from the same distribution and let x"" = arg melm) ((),)) with r being the corresponding reward. Then, we define
E (K [[1((() 0--10 - (1#* ff(xx;xx;xxx 0 0.-1) -112(x; (r= fx;;((;;0
(C.21)
Then, as (Xr,i, ~ D E [n]. we have
fe 0x (1i -1 01-1)) F7-1
EVV F
E
(X, (eE())
[/ (x;:0"1) (r+- - 0 F
(C.22)
E
(x. (e.
0
where F,-1 denotes the o-algebra generated by the history {x),, mf
Fr' 1
15-1
Therefore, {V>)t= is martingale difference sequence. Similarly, applying the Hoeffding-Azum inequality to V-, with probability 1 35, we have
52 x oz.)) - (1t ((x2
E (i. (0l11, 0?2;;
5a (4/;0071 - (1-- - fx(xx 0 0--1)
(xc 2 + |
6 r=1
(C.23)
1
Σ
fz xx; )²:1)) - (r²- fi(;;; 0. 0,-11) +(1+ 22)
2 2log(1/6)
t
t
18

---

Published as a conference paper at ICLR 2022
For 13, according to Lemma C.6 for any 0² satisfying 110 -f812 S O6 O(pim log mm with
probability 1 - 8, we have
f2(x;: 0--1) (1(x;;0,-1)
f2(x;:00) Φ2n (r= fi(x;; 0
0-11)
3L
(C.24)
+0
v2f
For 4. according to Lemma C.4 (1), these exists 0²r satisfying |00
100*~0011
08112 I 0 pym logm). m),
with probability - 6, such that
0=(x;:0²-) -xx; -12;))))))
v
Σ
(1axx;:0"))
fixx;; 0,-1))
(C.25)
1 $ Vi
2c
I.
where Is follows by a direct application of Lemma C.4 (1) by defining the loss C(0²))
fi(;;; 01-))) 6. Combining above incqualities, with proba- E (f((x;;0* 0"") (1; -
bility 55) we have
E iE)
fa xx::0 6 (-1) - (r; - fi (x): (xxx: [[xxx)))+
(C.26)
2c
3L
2log(1/0)
+
+ (1 + 2
/21
Then, applying union bound to I, and rescaling the 8 complete the proof.
Lemma C.2. Given d, E (0, p E (0, (0,)())), suppose 7ı, T2, K1, K2 satisfy the conditions
in Eq. (.2)) Then, with probability least 1 ó, in each round for any ||x||2 1, we have
(1)
lfi 07
fi (x(:0))_
tL log m ))
Lt P/m
L log 116
(C.27)
s 1 1+0
pl/3m2266
logm) m
O
p2/3m
(2)
fz [d(
o1. (VVo,.. ((x;
f2 (o(To)
(i 0 1)-)));;;))))))))))) 02-)))002))))))
11, log 3/6 m ))
Lt3
(
[ log 11/6 m
1og
(C.28)
<
1 +
logm
+ +
1/6
Pmm m
1/13m1/6
(3)
|o fi(x; 0:- 11-i)l2,1Vo0,,
|
VVo fo (o[Vo_,
Ox1); 04-1)
1)) l|2
(LL (LL"1og log 5/6 m
) O(L).
(C.29)
< 1+0
p/00mm1
19

---

Published as a conference paper at ICLR 2022
Proof. According to Lemma C.4 (2), 100 - fbll2 s 0 O(ppm log m), Vr E [t]. Thus, we have
|00-1 0blle I O p./m log m).
First, based on Triangle inequality, for any ||x|| = 1, we have
f1x = || Vog (x: 0 + || fi(x; 0/_1) Vog (xx; 0
t log 5/6 m
(C.30)
< +
m/ O(L)
where the last inequality is because of Lemma
C.4 (3) and Lemma
Applying Lemma C.5 (1), for any x D, | and |0 0/-1 O(
c Dym
log m) w.
have
(x; 0 fi( 1))
<V(V: (x;; 0,-0
0-_1)1 0(1?² log 10 0 01:1//2w1/3
<|| V|l,,__, fi(x;;
0 12211
011112 O(L2 ( vm log (m) 0. 0 11/2
(C.31)
11,3 log 3/6 m
))
Lt
L2 log 11/6 m
< (1+0) 0
O
log m) + +0
p1/3m 1/6
PVm
/433m 1/6
Similarly,
the same way prove the lemmas for f2-
Lemma C.3. Let [:; O,) follow the stochastic gradient descent of J1 f2 in Algorithm Suppose m, 1/2 satisfy the conditions in Eq. (.2). With probability at least - 8, for any x with xx|22 = and [T]. it holds that
( m
tnLL log 11/6 m
log
If(x: 0.) (1x: 501++ +
+0
PVm
p4/3m! 1/6
Proof. Considering an inequality la - bl < c, we have |a| < |6| + c. Let Og be randomly initialized.
Then applying Lemma C.5 (1), for any x D, ||||| 1 and 10, 0ol S we have
(x; 0,)I s (x 0o) (V ; 0o) 0, 0o) + 012 /m log(m)) 0, 0 0ull2w!/6 1/3
s O(1) + Vo.f (x; 0o) 2 Ooll2 / log(m))) 0 0oll2w1/3
1o
t
t
4/3
00(1) + OLL) o
log m
+ (L22 m log(m))
o
log
pvm m
Pvm
Is
( m
L [ Pvm
L2 1o122lo'100 1/6 11/6 m
0(1) +
/4/3m1//
(C.32)
where: To is based on the Lemma C.4 (3); Jn is an application Cauchy-Sehware inequality; T2 is
according to Lemma C.4 (2) and (3) in which 0, can be considered as one step gradient descent; Ig is
due to Lemma C.4 (2).
Then, the proof is completed.
Lemma C.4. Given a constant 0 < 1, suppose satisfies the conditions in Eq. (.. the learning rate r D p.)) the number of iterations K 2 poly(t,n.LL. probability at least 8. starting from random initialization Oo.
-loge=)). Then, with 0²
(1) (Theorem 1 in Ahllen-Zhu et al. 2019) In round t E [TT, given the collected data (r) the loss function is defined as: L(O) Y/ Ei (f(x+;0) rz)² Then,
20

---

Published as a conference paper at ICLR 2022
there exists 0 satisfying ||0 -0ol2 1 0 0 pvmm log m such that ( & in
K (2( (ulylt.m.L) log !) iterations;
P
(2) (Theorem in (Allen-Zhu et al. 2019)) For any l E [K], it holds uniformly that 110
ooll2 S O (p/m log
(3) Following the initialization, given ||x|| 1, it holds that [.eeffxx 0o)112 2 O(L), |f(x: 0o) 0(1)
where ( represents the parameters of f after k E [K] iterations of gradient descent in round t.
Proof. Note that the output dimension d in (Allen-Zhu et al. 2019) is removed because the output of network function in this paper always is scalar. For (1) and (2), the only different setting from (Allen-Zhu et al. 2019) is that the initialization of last layer Wz ~ NOO, 2 in this paper while W, ~ (O, 5) in (Allen-Zhu et al. 2019). Because d = and > d here, the upper bound in (Allen-Zhu et al. 2019) still holds for W r. with probability least - exp (-(mmmL))))))))) W r.1lr S
/ /m/d. Therefore, (1) and (2) still hold for the initialization of this paper.
For (3), based on Lemma 7.1 in Allen-Zhu et al. (2019), we have fx; 0u) 0(1). Denote by D the ReLU function. For any E L].
Iw F (W,,W,,- --I DW,, 1+ || [[W (41 -..1----- x O(VL)
where the inequality is according to Lemma 7.2 in Allen-Zhu et al. (019). Therefore, have (x;; 0o)ll2 & O(L).
Lemma C.5 (Lemma 4.1, (Cao and Gu 2019 For any E (0, if w satisfies
Om 3/2 L 3/2 [lg[ /0) S w S OL m] m)-mm/),
then, with probability at least 1 8 over randonness of 9o, for any E [T], ||x||2 = 1, and 0, e' satisfying @ 0oll2 w and || 0 w, it holds uniformly that
[f(x;; 0) fx;; 0') - o fxx; 0'), 0 0') s O3 L² ( log(m)))) 0'112. (C.33)
Lemma C.6. For any > 0, suppose
m > Ô (poly(T, n L, (og(1/5) evioR1/8)
10 Then, with probability at least 1- 8, setting n22 O 32 for algorithm for any o satisfying 100" -681l2 I O( 6a
pym logm), m), it holds that
Σ Ja (o(Vo) ,(xx;;, - (rr Axxx; ((1::0011))
Σ Sa ( (olVo_ ((x::
(r-- (r+= 011)) +0
3LVV
22
Proof.
This a direct application Lemma 4.3 in Cao and Gu 2019 by setting R = log m,,
LR
and -RR, where v' is some small enough absolute constant. We set L (0,
V2vl'
f2(V 01 f1; -2
-1=--(((x;;0-)) Based on Lemma C.4 (2), for any [t] we have
t log m).
10" 0 0r11l12 < 10- 100l2 +
-100-00--000 - -0- s O
PVm
21

---

Published as a conference paper at ICLR 2022
Table 2: Selection Criterion Comparison (x,: selected arm in round L).
Methods
Selection Criterion
Neural Epsilon-
With probability 1 c, X, arg maxx,, fi 0¹);
Otherwise, select x, randomly.
NeuralTS Zhang al.
2021
For X Vi E [n], draw ir,s from (Nfi( 0*), oe, Then, select x, arg max44]m] Pr,a.-
NeuralUCB Zhou et al.
2020
X₁ = arg max,,, (xi(x 0²) + UCB.,).
EB-Net (Our approach)
Vi [m)] compute fi (x 0²), f2 o S1 (x 0²); 0²) (Ex-
ploration Net). Then x, = arg max si(n) fz f2: 0²)
Then, according to Lemma 4.3 in (Cao and Gu 2019). then, for any o² satisfying 110² 0oll2 s
O ea Pym
log m), there exist small enough absolute constant v', such that
Σ
L. S Σ Li (0 + 31e.
(C.34)
T=1
7=1
Then, replacing completes the proof.
Lemma C.7 (Theorem 5, Allen-Zhu et al.
2019) For any 5 E (0,1), if w satisfies that S O(L 9/2 log -3 m))
Om (/mm-112- 3/2 max max{log 1/0
3/2 m, log (T (T/0)))
(C.35)
then, with probability at least 1 8. for all ||e 00ll S w, we have
[[ 0) Ve.f(x; 00)112 I O O O(Vloomm'' 133 L L 00)1l2.
(C.36)
D MOTIVATION OF EXPLORATION NETWORK
In this section, we list one gradient- baseed UCB from existing works (Ban et al. 2021 Zhou et al. 2020), which motivates our design of exploration network fz. Let g(x;; 0,) = Vo. fx,
Lemma D.1. (Lemma 5.2 in (Ban et al. 2021 Given a set of context vectors {x1]T= and the corresponding rewards [r:)T= B(rr)) h(xx) for any E {x1) Let f'xx;; 0) be the I--lyees fully-connected neural network where the width is m, the learning rate is n, the number of iterations of gradient descent is K. Then, there exist positive constants C1, C2 S, such that if
m 2 Poly(TT n, L, (10)(()))))))) eVlog1/07 = =O[TLL + mA)-1, K Ö(TL/X),
then, with probability at least 1 for any x, E {x1)==1 we have the following upper confidence bound:
[x(x)) f(x;; 0.)l ( 0,)/ /ml | 72 + 7173 Y4.
(D.1)
where
(m, L.) L)= (O+))()))) ((1 - 112 1///+++ +1
(1/08/
dee((A)) det(Al)
72 L, 8) ||,(x;; 0n)/ /mlll
log 8 + 21/25
7m L) = 02m 1/6 log 1 ra(m, L) Cim 1/6 1 x 2/3
A, = AI Σ g(x:: 0x) T.)Tm. A = AI Σ 2x(x; (0)
Note that g(x;; 0o) is the gradient at initialization, which can be initialized as constants. Therefore, the above UCB can be represented as the following form for exploitation network fı: [h(x),)) fi (x,,;; 0!)1 (g((x; 0.)).
22

---

Published as a conference paper at ICLR 2022
Table 3: Exploration Direction Comparison.
Methods
"Upward" Exploration
"Downward" Exploration
NeuralUCB
V
NeuralTS
Randomly
Randomly
EE-Net
EE-Net has smaller approximation error. Given an arm æ, let fi (x) be the estimated reward and h(x)) be the expected reward. The exploration network in EE-Net is to learn h(x)) (ar), i.e., the residual between expected reward and estimated reward, which is the ultimate goal of making exploration. There are advantages of using network f2 to learn h(x) to giving a statistical upper bound for it such as NeuralUCB, (Ban et al. 202T), and NeuralTS (in NeuralTS, the variance can thought of as the upper bound). For EE-Net, the approximation error for h(x) fi (2:) is caused by the genenalization eπror of the neural network (Lemma B.1. in the manuscript). In contrast, for NeuralUCB, (Ban al. 202T), and NeuralTS, the approximation error for f(x)) fi (a:) includes three parts. The first part is caused by ridge regression. The second part of the approximation error is caused by the distance between ridge regression and Neural Tangent Kernel (NTK). The third part of the approximation enror is caused by the distance between NTK and the network function. Because they use the upper bound to make selections, the errors inherently exist in their algorithms. By reducing the three parts of the approximation errors to only the neural network convergence error, EE-Net achieves tighter regret bound compared to them (improving by roughly Vlog T).
(a) EE-Net, compared
h(x.))
Gap
Gap
(2)
h(xy))
Upward Exploration
Case Downward Exploration
Figure 6: Two types of exploration: Upward exploration and Downward exploration. fi is the exploitation network (estimated reward) and h is the expected reward.
EE-Net has the ability to determine exploration direction. The two types of exploration are described by Figure 6 When the estimated reward is larger than the expected reward, i.e., M(x) (x) < 0, we need to do the 'downward exploration', i.e., lowering the exploration score of x to reduce its chance of being explored; when h(x) - (x) >> O, we should do the 'upward exploration', i.e. raising the exploration score of æ to increase its chance of being explored. For EE-Net, fo is to learn h(x) - (z).. When h(x) (x) > 0, fz(x) will also be positive make the upward exploration. When (zx) (z) < 0, fa(r) will be negative to make the downward exploration. In contrast, NeuralUCB will always choose upward exploration, i.e., fi (a) + UCB(x) where UCB(2) is always positive. In particular, when h(x) - (xx < O, NeuralUCB will further amplify the mistake. NeuralTS will randomly choose upward or downward exploration for all cases, because draws a sampled reward from normal distribution where the mean is fi (x) and the variance is the upper bound.
23


Reviews:
[{"review_id": "zKDsGIvBHlk", "reviewer": "Reviewer_SiY7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and outputs a better regret bound with an interesting idea. The reviewer has the one concern about the exploration network: \n     the proposed exploration net is mainly based on the error bound derived from Ban et al. 2021. This paper uses neural networks to learn the complicated function. This step makes the reviewer confused. We already have an analytical equation (even it has some unknown constants) for that error bound (lemma D.1). What’s the point of training a neural network to learn it instead of learning the unknown parts in that equation? Also,  why do we use the gradients as the input for the neural network instead of putting more known items in the input, e.g., A_t in the lemma D.1 (does this give more info to the training process)?  \n", "summary_of_the_paper": "This paper proposes EE-NeT for contextual bandits which contains two neutral networks: one for learning the underlying reward function and another for learning the potential gains of arms if explored. They prove that  EE-NeT achieves better regret bounds than state-of-the-art neural bandits algorithms for both UCB-based and TS-based. This paper also shows that EE-NeT outperforms existing linear and neural bandit approaches.", "summary_of_the_review": "The reviewer doubts the motivation of using neural networks to the confidence found when we have an analytical equation for that. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ry2IO10MGD", "reviewer": "Reviewer_BoiG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "First of all, authors provided a very clear review on the various neural network based contextual bandits and illustrated their proposed approach very clearly through discussing the novelties they claim with their proposed technique in comparison with existing work in the area. The proposed technique has clear advantages over the existing literature and I believe it can be easily applied to existing domains that benefit from contextual bandit approached. The proposed method is well supported by theoretical findings and good empirical evidence. Authors also provide their source code and the datasets they use, which is a very important aspect for interested readers/researchers.\n\nFor the evaluation, I am not an expert on the contextual bandits but from educated guess, the experiment results are extensive, solid and impressive. They compared with existing state of art techniques using benchmark datasets and regret analysis together with ablation experiment results are presented.\n\nI have a few comments/questions/concerns:\n1.\tAuthors provide regret bounds’ complexity analysis for their technique. However, what are some concrete time complexity results for the proposed technique? My concern is, for online learning algorithms “time it takes to respond should be” quick, I am really curious to see how the techniques’ response time compares with the existing techniques response time in literature. Can authors provide this comparison analysis? Is there any overhead that the neural net based exploration brings to the solution? \n2.\tParallel to 1st question, How does the solution scale with larger datasets?\n3.\tFigure 1 results are reported over 10 runs, I am curious why such a low number of runs are performed, not 30 for instance? Is it due to time-constraints or memory constraints for the machines used during experiments? Can you provide details for the machine co\n", "summary_of_the_paper": "This paper presents a neural net based bandit approach with a novel exploration strategy. Specifically, the solution uses an exploitation network to estimate rewards for each arm and an exploration network to predict the potential gain compared to current reward estimate. It then uses a decision maker to select arms using one of the linear or nonlinear modes. The key differentiator/strength of the paper is using neural net based exploration strategy and demonstrating that the proposed solution has a tighter bound than the existing state-of-the-art bandit algorithms. ", "summary_of_the_review": "Overall, I liked the paper, very clearly written, organized. I am convinced with the novelty of the technique being proposed, I had a few concerns around some applicability issues in real life due to time complexity. I have detailed my questions in the previous box. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "3cFODH9cKmx", "reviewer": "Reviewer_vjXZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n\n1. The studied neural contextual bandit is very interesting and important in the area of applying neural networks to online learning.\n2. The proposed algorithm uses a novel and interesting network architecture, EE-Net, which is different from prior UCB-based [Zhou et al. 2020] or Thompson Sampling [Zhang et al. 2020] based neural bandit algorithms.\n3. The authors give rigorous theoretical analysis and tighter regret bounds. While I do not think an improvement of \\sqrt{log T} in the regret bound is significant, this does demonstrate the theoretical soundness of EE-Net.\n4. The experiments are conducted on extensive real-world datasets, and the empirical results validate the superiority of the proposed EE-Net.\n\n\nWeakness:\n\nIt would be better if the authors can highlight the technical challenges in regret analysis and compare their analytical techniques with prior neural contextual bandit works.\n\n\n---After Rebuttal---\n\nI have read the authors' response. They clearly explained the technical challenges of this paper, which well addressed my concerns. I appreciate that their algorithm only stores and updates the $O(p)$-dimensional parameter in neural networks, and does not need to explicitly maintain extra $O(p)$-dimensional vectors or $O(p \\times p)$-dimensional matrix. This point is very good, and I have never seen it before in the neural contextual bandit literature. The theoretical improvement of removing the $d$ or $\\tilde{d}$ is also interesting.\nI am glad to increase my score from 6 to 8.", "summary_of_the_paper": "This paper studies the neural contextual bandit problem, and proposes a neural-based bandit approach with a novel exploration strategy, called EE-Net. Besides utilizing a neural network (Exploitation network) to learn the reward function, EE-Net also uses another neural network (Exploration network) to adaptively learn potential gains compared to currently estimated reward. Then, a decision-maker is\nconstructed to combine the outputs from the Exploitation and Exploration networks. Theoretical guarantees and extensive experiments are provided to demonstrate the performance superiority of EE-Net over existing linear and neural bandit algorithms.\n", "summary_of_the_review": "Overall, I think that the studied neural contextual bandit problem in this paper is well-motivated. The idea behind the proposed algorithm is novel and effective. Tighter theoretical guarantees and extensive experimental results are provided to demonstrate the superiority of the proposed algorithm. Therefore, I recommend “weak accept”.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0CBKIp6qqib", "reviewer": "Reviewer_KcQy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\nThe idea of utilizing neural networks to model potential reward gains for exploration is very interesting; it seems quite different from rule-based UCB and Thompson sampling.\n\n**Weaknesses**\n\nOn the theoretical side, the assumptions on over-parameterized neural networks are utilized to prove the regret bound, which is tighter than NeuralUCB and NeuralTS. However, since the regret bounds of NeuralUCB and NeuralTS are derived from the assumptions of the neural tangent kernel, it is not clear whether the theoretical advantage comes from the method itself, or from the assumptions.\n\nOn the experimental side, it is not sufficiently rigorous to tune the hyper-parameters. It is said that \n\n> For all grid-searched parameters, we choose the best of them for the comparison and report the averaged results of 10 runs for all methods.\n> \n\nIn real applications, it is not possible to tune hyper-parameters in this way primarily due to the bandit feedback. It is necessary to avoid the risk of overfitting in the experimental setting.", "summary_of_the_paper": "This paper studies neural network-based contextual bandits. Different from existing works which utilize UCB or Thompson sampling for exploration, the exploration component of the proposed method EE-Net is also modeled with a neural network.\n\nBased on theoretical tools on over-parameterized neural networks, it is proved that EE-Net achieves the regret bound of $\\mathcal{O}(\\sqrt{T\\log T})$. In contrast, existing neural network-based contextual bandit methods, such as NeuralUCB and NeuralTS, achieves the regret bound of $\\mathcal{O}(\\sqrt{T} \\log T)$ based on the theoretical tool of the neural tangent kernel.\n\nExperiments are conducted on four public datasets to validate effectiveness of EE-Net, compared with a variety of contextual bandit methods including both neural-based methods and linear contextual bandit methods.", "summary_of_the_review": "The idea is interesting and as far as I know it is novel. However, both the theoretical and experimental results should be verified more carefully. See Main Review for details.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary: This paper studies the neural contextual bandit problem, and proposes a neural-based bandit approach with a novel exploration strategy, called EE-Net. Besides utilizing a neural network (Exploitation network) to learn the reward function, EE-Net also uses another neural network (Exploration network) to adaptively learn potential gains compared to currently estimated reward.

Discussions: The reviewers appreciated the novelty and the quality of the ideas and results in this paper. Most questions were about details in algorithm design choices and in the analysis. The authors have addressed these questions and updated their draft. The reviewers have now reached a consensus and recommend accepting this paper.

Recommendation: Accept.
Decision: Accept (Spotlight)