Pure Exploration in Kernel and Neural Bandits

Abstract: We study pure exploration in bandits, where the dimension of the feature representation can be much larger than the number of arms. To overcome the curse of dimensionality, we propose to adaptively embed the feature representation of each arm into a lower-dimensional space and carefully deal with the induced model misspecifications. Our approach is conceptually very different from existing works that can either only handle low-dimensional linear bandits or passively deal with model misspecifications. We showcase the application of our approach to two pure exploration settings that were previously under-studied: (1) the reward function belongs to a possibly infinite-dimensional Reproducing Kernel Hilbert Space, and (2) the reward function is nonlinear and can be approximated by neural networks. Our main results provide sample complexity guarantees that only depend on the effective dimension of the feature spaces in the kernel or neural representations. Extensive experiments conducted on both synthetic and real-world datasets demonstrate the efficacy of our methods.


Reviews and author responses:

 We truly appreciate you providing valuable feedback and taking the time to participate in the committee discussion. We will incorporate your suggestions for improving the paper. Thank you!


 Dear authors, dear reviewing committee, 

Thank you for your response to my questions, I believe that clarifying some of those points will improve the paper and I keep my score to 8 because I believe this is a strong paper. As I became aware that this year we were encouraged to publicly discuss, I apologize for not responding publicly earlier and I notify the authors that I have taken part early in the committee discussion regarding the plagiarism claim. I do not believe this is a plagiarism case and I hope that this was a misunderstanding that will not weigh in the final decision. 

Many thanks.

This paper introduces a new approach to tackle bandit pure exploration in high dimensional setting. Recall that the literature generally deals with a finite number of arms, or scale exponentially with the dimension of the feature space. The idea, developed in Algorithm 1, is as follows. At each iteration of a RAGE-like algorithm, the arms are embedded into a lower dimensional space. The dimension is chosen as the smallest integer such that the induced error does not harm the subsequent computations and guarantees. Guarantees for Algorithm 1 are derived in Theorem 1, and expressed in terms of the effective dimension. An example of sequence of feature maps is given for linear bandits, based on the SVD decomposition of the design matrix. Under boundedness of parameter $\theta_\star$, it is possible to control the induced error by the successive approximations. When the reward function is assumed to be in a RKHS, using the eigendecomposition of the associated integral operator allows to design similarly a series of feature maps with controllable error. The effective dimension can be bounded in terms of the eigendecay of the kernel. The reward function can also be approximated using a neural network. The algorithm (Algorithm 2) then also needs to learn the weights of the network jointly. Guarantees for Algorithm 2 are stated in Theorem 3. Finally, Section 6 presents some experiments on synthetic and real world datasets showing that the proposed Algorithms exhibit smaller sample complexities than Action Elim.  I found the paper globally clear and well written. The motivations are clear, and the idea of using as few dimensions as possible (as long as the error is controllable) is indeed sound. I found the instantiation to linear bandits (Section 3.2) very enlightening. Then, the extension to a kernelized version (Section 4) also appears to be natural, but it is well presented and treated. The instantiation to neural networks is slightly more involved, but of high interest in practice. Overall, I think the authors did a great writing job (right technical level, natural progression, useful remarks) so that the story presented is consistent and sounds very natural.

Could it be possible to extend the proposed approach to other problems than pure exploration? To the contextual case? What would be the difficulties? 

I thought it could be interesting to plot the evolution of $d_k$ with respect to $k$.

Some other minor comments:
- l. 37: address it *in* the
- display eq. 109-110: is it necessary to "assume" here? the equation seems always verified, for any $\psi_d$, as long as we define $\eta_d(x) = h(x) - <\psi_d(x), \theta_d>$
- l. 115: I assume that $||\cdot||_A$ refers to the Mahalanobis norm (w.r.t. matrix $A$) but this should be specified in the notation
- l. 132: one need*s*
- l. 136: citing the RAGE paper here too would be better
- l. 139: $r_d(\zeta)$ is not defined
- Alg. 1/2: putting algorithms at the top of pages help the reading
- display eq. 150-151: an intuition comment on the complexity measure could help
- l. 189: *which* is square..
- l. 230: is *the* use
- l. 240: *to let* --> such that
- alg. 2: line 8 is not very clear

**Post-rebuttal edit**

I thank the authors for their response. I am in favor of acceptance and keep my 7 score. Yes

This paper considers the best arm identification problem for kernel and neural bandits, where the reward function belongs to an RKHS and is approximated by a neural network, respectively. The main contribution is that the authors propose to consider a variant of the Randomized Adaptive Gap Elimination (RAGE) algorithm with adaptive embedding (so that the problem can be viewed as misspecified linear bandits) and then address the corresponding misspecification issue. The sample complexity results are established under known feature mappings and misspecification errors and then adapted to the cases of RKHS and neural bandits. Experimental results on both synthetic datasets and real datasets are provided to validate the proposed algorithms.
  The main idea of this paper is to adaptively choose the embedding and do pure exploration with the corresponding misspecified linear bandits. I like the idea of adaptively selecting the embedding in RAGE as it appears to be a quite generic approach for pure exploration (as long as the feature mapping can be constructed). Based on this design (cf. Alg. 1), the main challenge lies in determining the number of samples ($N_k$ in Line 6 of Alg. 1), which requires careful and non-trivial control of the misspecification error term ($\epsilon_k$). 
As the main theoretical result of this paper, Theorem 1 shows that under known feature embeddings and the proposed careful control, the sample complexity is proportional to $d_{eff}(\epsilon)$, which captures the minimum required dimensionality of the embedding. The proof of Theorem 1 looks sound. Based on Theorem 1, one can accordingly obtain the sample complexity of pure exploration in RKHS and neural bandits (with the corresponding NTK), as described by Theorems 2 and 3. Overall this is a solid theoretical contribution to the best arm identification problem.

I have some questions about the main results:
- How tight are the sample complexity results provided in Theorems 2-3? 

- Regarding Theorem 3 for neural bandits, the hidden layer size $m$ is a function of the number of samples $N_k$, which is at least on the order of $2^{2k}$ and shall be quite large. In practice, how to choose a proper $m$ such that the misspecification error is well controlled? 

On the other hand, I would also suggest the authors to further improve the presentation. Currently, the presentation is quite dense, and several terms, definitions, and designs are introduced without much explanation. This makes some parts of the paper quite difficult to parse. Just to name a few:
- Lines 114-115: The definitions of $g(d,\zeta)$ and $\gamma(d)$ would require more explanation. On the high level, $g(d,\zeta)$ seems related to the transductive linear bandits (e.g., [Fiez et al., 2019]). On the other hand, it is not immediately clear why $\gamma(d)$ captures the suboptimality gap.

- Line 6 in Algorithm 1: While I can understand the intuition behind the choice of $\epsilon_k$, it is not immediately clear to me why $\epsilon_k$ shall take the current form.

- There are several complexity measures that look similar and related. For example, what is the connection between $g(d,\zeta)$ and $\rho^*_d(\epsilon)$?

- By the equation in Line 109, it appears that $\eta_d(x)$ also depends on the choice of $\psi_d(\cdot)$, and hence $\tilde{\gamma}(d)$ shall not only depend on $d$ but also $\psi_d(\cdot)$?

- Line 116: The intuition behind the suboptimality gap $\gamma(d)$ is not clearly explained.

- Is the arm set $\mathcal{X}$ finite?

- Line 551: There might be some typos in these inequalities. What are $x$ and $x’$ here?

- Line 554: Similarly, what is $x’$ in (12)?

Other minor comments:
- Line 550: $\psi(x)\in \hat{S}_k$ -> $x\in \hat{S}_k$

----- Post-rebuttal update -----

Thank the authors for the response. The rebuttal has clearly addressed my questions, and overall this paper is a solid contribution to the pure exploration problems. As a result, I have raised my score accordingly.
 In the checklist, the paper mentions that it has described the limitations in Section 7, but I did not see this mentioned in any section.

 This paper was flagged for plagiarism concerns which I have not assessed as it is outside the scope of ethics review. Plagiarism concern should be separately assessed. Plagiarism concerned should be addressed.

 **Q1:** ''The paper looks incremental ...''

**A1:** We respectfully disagree with your judgment on the contribution of our work. We believe our contribution is significant (as appreciated by other reviewers). 

First, we emphasize the contributions of our work. We introduce the idea of adaptive embedding to avoid the curse of dimensionality. The induced model misspecifications are carefully handled, which is novel in the bandit pure exploration setting. We generalize our algorithmic framework to high-dimensional linear bandits and non-linear settings such as kernel bandits and neural bandits. We theoretically analyze the sample complexity upper bound with respect to well-defined effective dimensions. To the best of our knowledge, this provides the first theoretically founded pure exploration algorithm with a neural network approximation. We conduct extensive experiments on both synthetic and real-world datasets, which confirms the efficacy of our proposed algorithms, especially the neural bandit algorithm. These contributions are recognized by other reviewers.


Regarding your comment that our work is a direct combination of [13, arXiv:2006.11685] (arXiv:2006.11685 is cited as [24] in the submission) and [46], this is definitely not true. We don't see how a direct combination of [13] and [46] would even work. More specifically, due to the high dimensionality of neural tangent feature mapping, we will end up solving a transductive experiment design problem with $K \ll p$, where $K$ is the number of arms, and $p$ is the dimension of the feature map. Simply applying existing results of optimal design would yield an unsatisfying sample complexity result that is linear in $K$.


Regarding the neural bandits work [46], it studies regret minimization in the contextual bandits setting, while our paper studies pure exploration (i.e., best arm identification). So the settings of both papers are fundamentally different. Our work is by no means incremental or trivial given [46]. 

**Q2:** ''Research Integrity??''

**A2:** Algorithm 5 in Appendix F.1 is indeed similar to Algorithm 2 in [46], but this is a standard gradient descent. We agree that adding a reference here is better, but it is by no means plagiarism. Moreover, this algorithm (gradient descent) is not the contribution of our paper at all. 

We cite [17] for Definition 3 because [17] is the first paper that proposes the concept of neural tangent kernel. Note that in [46], the authors also cite [17] in their Definition 4.1. It is a convention to cite the original paper for a certain concept. We are happy to cite both [17] and [46] in Definition 3. But again, there is nothing to do with ''research integrity" here.


The complexity measure after line 150 is a standard measure that has previously been extensively studied in [38, 13, 10] (we emphasize that in Remark 2), and serves as the instance-dependent lower bound of identifying the optimal arm. Since we follow the framework of RAGE [13], we do not believe using such complexity measure constitutes plagiarism. We will add a pointer to RAGE [13] here in the revision. 


We sincerely hope our above explanation can resolve your concerns.

 We thank you for your valuable feedback. We address your specific concerns as follows (reference numbers are the same as the submission). 

**Q1:** ''Could it be possible to extend the proposed approach to other problems than pure exploration? To the contextual case? What would be the difficulties?''


**A1:**  We believe our idea of embedding actions into a lower dimensional space to avoid the curse of dimensionality could be extended to other settings such as regret minimization. Some existing works have explored the cumulative regret minimization problem in the contextual case [42, 46] using the notion of effective dimension (but without using adaptive embedding studied in our submission). The main difficulty for the extension to the linear contextual bandits setting is that the action set is not pre-determined and the action sets can change round by round. So the adaptive embedding needs to be done on the fly and we may need streaming algorithms to handle that.


**Q2:** Question regarding the evolution of $d_k$ with respect to $k$.

**A2:** For adaptive embedding, $d_k$ is generally increasing with respect to $k$ since smaller misspecification levels are tolerated in later iterations. For fixed embedding, $d_k$ is fixed across all iterations. 

For experimental results in Figure 2, our KernelEmbedding and LinearEmbedding algorithms are implemented with fixed embedding (as discussed in Appendix G.1). For NeuralEmbedding, the evolution of $d_k$ with respect to $k$ is summarized as follows (averaged over 50 runs): for MNIST dataset, we have $d_1 = 190.12$ and $d_2 = 190.63$; for Yahoo! dataset, we have $d_1 = 73.44$ and $d_2 = 168.67$. We will include these additional results in the revision.


**Q3:** Question regarding the display in lines 109-110.

**A3:** We only assume the existence of a feature mapping $\boldsymbol{\psi}_d(\cdot)$ here (without providing an explicit construction). Yes, all quantities are well-defined with such feature mapping. Explicit constructions of such feature mappings are provided for high-dimensional linear bandits (Section 3.3), kernel bandits (Section 4), and neural bandits (Section 5).

**Q4:** Question regarding the matrix norm.

**A4:** Mahalanobis norm is used at line 115 and throughout the paper. We will add a formal definition in the revision.

**Q5:** citing [13] after `follows the framework of RAGE' at line 136.

**A5:** We will add citation number [13] after ''follows the framework of RAGE'' at line 136 in the revision.

**Q6:** Question regarding the definition of $r_d(\zeta)$.

**A6:** $r_d(\zeta)$ represents the minimum number of samples needed to solve the optimal design problem in Eq.(1) up to $(1+\zeta)$-approximation. It can be set as $r_d(\zeta) =\frac{d^2 + d + 2}{\zeta}$ [33] or $r_d(\zeta) = \frac{180 d}{\zeta^2}$ [1, 13], depending on which approximate algorithm is being used. We will provide the precise definition in the revision. 


**Q7:** Question regarding the complexity measure after line 150.

**A7:** The complexity measure $\rho_d$ (after line 150) quantifies the hardness of the pure exploration problem, and is previously studied in [38, 13, 10]. It serves as the instance-dependent lower bound of identifying the optimal arm; and we add $\epsilon$ tolerance in the definition for our purpose of identifying an $\epsilon$-optimal arm (as discussed in Remark 2). 


**Q8:** Question regarding line 8 of Algorithm 2.

**A8:** At line 8 of Algorithm 2, we run gradient descent to optimize the loss function $L$. The detailed gradient descent algorithm is in Algorithm 5 in the appendix. 

Thank you for pointing out several typos. We will correct them in the revision.

 We thank you for your valuable feedback. We address your specific concerns as follows (reference numbers are the same as the submission). 

**Q1:** ''How tight are the sample complexity results provided in Theorems 2-3?''

**A1:** We believe the complexity bound in Theorem 2 under exponential eigen-decay is fairly tight, since it essentially matches, up to logarithmic factors, the lower bound of distinguishing a two-armed bandit up to accuracy $\epsilon$ [25]. We believe our effective dimension in Definition 2 captures the ''hardness'' of the corresponding neural bandit problem. Our sample complexity guarantee in Theorem 3 is also  tight for some special cases (for instance, the effective dimension can be bounded by some constants, which degenerates to the linear bandit case). We leave a rigorous analysis regarding the tightness to future work.


**Q2:** ''What is the influence of hidden layer size $m$? How to choose a proper $m$ such that the misspecification error is well controlled?''

**A2:** It is worth noting that $k$ is in the order of $\log(1/\epsilon)$, therefore $m\sim N_k \sim 2^{2k}$ is in the order of $\text{poly}(1/\epsilon)$. 
In practice, as we suggested in Appendix G.1, to obtain a reasonably good result, we choose a three-layer (i.e., two-hidden-layer) neural network with $m = 128$ (128 nodes) for each hidden layer. 

Further explanations about certain quantities are addressed as follows.

**Q3:** Question regarding the definitions of $g(d,\zeta)$ and $\gamma(d)$.


**A3:** $g(d, \zeta)$ (defined at line 114-115) represents the optimal value of the transductive design under feature mapping $\boldsymbol{\psi}_d(\cdot)$, and our algorithm is guaranteed to output an arm with sub-optimality gap smaller than $\gamma(d) : =  ( {16 + 8\sqrt{ g(d , \zeta)}} ) \, \widetilde  \gamma(d) = O(\widetilde \gamma(d) \sqrt{d})$. We believe such guarantee is optimal in the worst case since identifying an arm with sub-optimality gap $o(\widetilde \gamma(d) \sqrt{d})$ requires the number of samples to be exponential in $d$ even with deterministic feedback [28] (as discussed in Remark 1). 



**Q4:** Question regarding the intuition behind the choice of $\epsilon_k$.

**A4:** $\epsilon_k$ essentially captures the additional deviation in the confidence bound due to model misspecifications. This quantity is mainly used in our analysis, e.g., Appendix B.1.


**Q5:** Question regarding the connection between $g(d,\zeta)$ and $\rho^\star_d(\epsilon)$.

**A5:** $g(d,\zeta)$ represents the optimal value of the transductive design without knowing the true reward function (and thus can be calculated). $\rho_d^\star(\epsilon)$ represents the instance-dependent lower bound of the pure exploration problem (with $\epsilon$ slackness), but it can only be analyzed with a known true reward function. We only use $g(d, \zeta)$ (and related quantities) to guide our sampling procedure, and our final instance-dependent sample complexity can be linked to $\rho_d^\star(\epsilon)$. 

**Q6:** Question regarding $\eta_d(\boldsymbol{x})$, $\widetilde \gamma(d)$ and $\psi_d(\cdot)$.

**A6:** Yes, $\eta_d(\boldsymbol{x})$ is defined with respect to the embedding $\boldsymbol{\psi}_d(\cdot)$. Thus, $\widetilde \gamma(d)$ depends on $\psi_d(\cdot)$ as well.

**Q7:** Question regarding the intuition behind $\gamma(d)$.


**A7:** We define $\gamma(d) : =  ( {16 + 8\sqrt{ g(d , \zeta)}} ) \, \widetilde  \gamma(d) = O(\widetilde \gamma(d) \sqrt{d})$ so that our algorithms are guaranteed to output an arm with sub-optimality gap smaller than $\gamma(d)$. At each iteration, our Algorithm 1 uses $\gamma(d)$ to determine the effective dimension $d_{\text{eff}}$ with respect to certain error tolerance $4\cdot 2^{-k}$ (see line 3 in Algorithm 1).


**Q8:** Question regarding the cardinality of action set.

**A8:** The action set is assumed to be finite, which is standard in the setting of the pure exploration problem.

**Q9:** Questions regarding lines 550, 551 and 554.

**A9:** In the display after line 550, we should have $\boldsymbol{x} \in \widehat {\mathcal{S}}_k$ instead of $\boldsymbol{\psi(x)} \in \widehat {\cal S}_k$. In the display after line 551, we should replace $\boldsymbol{x}^\prime$ by $\widehat{\boldsymbol{x}}$ and replace $\boldsymbol{x}$ by $\boldsymbol{x}_\star$. In the display after line 554, we should replace $\boldsymbol{x}^\prime$ by $\boldsymbol{x}_\star$.

**Q10:** ''In the checklist, the paper mentions that it has described the limitations in Section 7, but I did not see this mentioned in any section''

**A10:** We actually have discussed the limitation of our work at line 332-334. The main limitation of the current work is that our analysis with neural representations is in the NTK regime where the neural network parameter $\theta_t$ is close to the initial parameter $\theta_0$, which can only describe a part of the representation of the neural networks. We leave extending our algorithm to more general settings (beyond the NTK regime) as a future direction.

Thank you for pointing out several typos. We will correct them in the revision.

 We thank you for your valuable feedback. We address your specific concerns as follows (reference numbers are the same as the submission).

**Q1:** The RAGE algorithm.

**A1:** At line 72, we try to emphasize that RAGE from [13] achieves near-optimal instance-dependent sample complexity bound for standard linear bandits (while previous algorithms don't). This is the reason why we build our algorithms upon the framework of RAGE, as stated at line 136. We will add a citation at line 136 in the revision. We will also state that our algorithm is related to RAGE [13] in the introduction.

**Q2:** Computational complexity.

**A2:** Thank you for your suggestion. Here we discuss the computational complexity from the following four aspects.

First, feature mappings can be constructed on the fly (thus there is no need to store all the mappings). We provide methods to construct feature mappings in Section 3.3 (at line 179-187, for high-dimensional linear bandits), Section 4 (at line 193-203, for kernel bandits), and Algorithm 3 (line 3 of the Algorithm, for neural bandits). 

Second, the effective dimension (Definition 1 and Definition 2) can be calculated in a binary search manner, which greatly reduces computational complexity. 

Third, the design $\lambda_k$ and value $\tau_k$ of the minimax problem can be approximately solved by the same Frank-Wolfe algorithm used in [13] (and discussed in Appendix G.1., at line 882-883). 

Finally, the ROUND procedure can be efficiently solved with computational complexity $\widetilde O(K N_k^2 d_k^2)$ (at the $k$-th iteration) as discussed in [13, 1].

We will add the above discussion in the revision.

**Q3:** Practical implementation of the algorithm.

**A3:** All these quantities can be efficiently computed as discussed above. We also provided our code for all the implementations in the supplementary materials. 

**Q4:** The precise definition of $r_d(\zeta)$ at line 139 and 141.

**A4:** $r_d(\zeta)$ represents the minimum number of samples needed to solve the optimal design problem in Eq.(1) up to $(1+\zeta)$-approximation. It can be set as $r_d(\zeta) =\frac{d^2 + d + 2}{\zeta}$ [33] or $r_d(\zeta) = \frac{180 d}{\zeta^2}$ [1, 13], depending on which approximate algorithm is being used. We will provide the precise definition in the revision. 

**Q5:** The wordings of the induced approximation error and the misspecification level.

**A5:** $\eta_d(\boldsymbol{x})$ represents the induced approximation error (approximated by the low-dimensional embedding) with respect to individual action $\boldsymbol{x}$ (at line 110). $\widetilde \gamma(d)$ represents the misspecification level: an upper bound of the induced approximation error across all actions, i.e., $\max_{\boldsymbol{x} \in \mathcal{X}} \eta_d(\boldsymbol{x}) \leq \widetilde \gamma(d)$ (at line 114). 
$\gamma(d) : = ({16 + 8\sqrt{ g(d , \zeta)}}) \widetilde  \gamma(d) = O(\widetilde \gamma(d) \sqrt{d})$ quantifies the sub-optimality gap of the identified arm; such quantification is tight in the worst case, and the reason is explained in Remark 1. At each iteration, our Algorithm 1 uses $\gamma(d)$ to determine the effective dimension $d_{\text{eff}}$ with respect to certain error tolerance $4\cdot 2^{-k}$ (see line 3 in Algorithm 1). We will add more explanation in the revision.


**Q6:** The name of 'rounding' and the complexity measure $\rho_d$.

**A6:** The name of ''rounding'' is borrowed from [13], which refers to the procedure of rounding continuous design into the discrete design. The complexity measure $\rho_d$ quantifies the hardness of the pure exploration problem and is previously studied in [38, 13, 10]. It serves as the instance-dependent lower bound of identifying the optimal arm; and we add $\epsilon$ tolerance in the definition for our purpose of identifying an $\epsilon$-optimal arm (as discussed in Remark 2). 

**Q7:** Questions regarding Appendix B.3.

**A7:** $\boldsymbol{x}_1 = - \boldsymbol{1}/\sqrt{D}$ where $\boldsymbol{1}$ represents the all-one vector in $\mathbb{R}^D$. Such a construction satisfies all the requirements at line 600-601.

**Q8:** Questions regarding Appendix D.

**A8:** The correct sentence at line 642 should be ''We then define $\overline \gamma(d) : = 2 \cdot 2^{-n(d)}$, which quantifies the optimality of the identified arm''. Yes, the display line in Theorem 5 should have $N \leq$ in front of the expression. Algorithm 4 proceeds with an embedding dimension $d$ as input, and requires no information of the misspecification level $\widetilde \gamma(d)$ (Algorithm 1 requires such information to adaptively select the embedding dimension at each iteration). 

Thank you for pointing out several typos. We will correct them in the revision.

 Reviewer 7Wf3  flags this paper for research integrity issues on algorithms that are similar to those in other published work. This is out of the scope of ethics review.  Reviewer 7Wf3  flags this paper for research integrity issues on algorithms that are similar to those in other published work. This is out of the scope of ethics review.  This is for committee to review. 

Summary: This paper studies \delta-PAC \epsilon-best arm identification when a kernel or neural representation is used to embed the arms. The paper targets regimes where the number of arms K is really large and the similarity between arms can be exploited to reduce the sample complexity of the tests needed to find the best arm.

General remark: The paper is well written and addresses a hard problem. The algorithms are smart and the empirical results are impressive. Despite a few minor remarks listed below, I recommend accepting this paper.

Disclaimer: Unfortunately, my level of expertise does not allow me to verify Appendix F.
  Positive points:
I think the main contribution of the paper is that Alg2 improves sample complexity of BAI by *orders of magnitude*, while also satisfying high-probability bounds based on the NTK of the network. 
The appendix is remarkably clear and (nearly) flawless, which also translates into verifiable correctness.
The code is provided and just skimming through some of the files, it does look like a high quality implementation. I did not deeply verify it though.

Improvement points: 
1/ The paper is more or less based on Rage from Fiez et al. [13] but that’s not clearly stated. In the Related Work section, the sentence on line 72 that presents this paper is strange: “hat do you mean by “the instance-dependent sample complexity is achieved” ? Here you should already mention RAGE. But more importantly on page 3, line 136, you say “Algo 1 follows the framework of RAGE” but you forgot to cite [13] here such that someone who does not know the reference [13] already, does not follow your point (that was my case). 

2/ *Computational complexity*: I am surprised the computational complexity is not more discussed because the quantities involved in your algorithms do not seem straightforward to compute. First, the feature representation step at the beginning of both algorithms seem like they must be pre-learnt ? So you’d have to store these models. Then the efficient dimension, the \lambda_k, the \tau_k are solutions of minmax problems and do not seem immediate to obtain. For the ROUND function, you mention that “computationally efficient” algorithms exist but I am still curious how that scales with K and d (and N).

3/ Maybe there are smart ways to compute these quantities and it’d be worth discussing this. I actually don’t really doubt there is a way because you have pretty large scale experiments so I believe you found a way around but I think it would be reassuring to see this discussion. I am personally quite worried about having to implement this method myself and not knowing what would cost what, what are the bottlenecks, it does not seem like a simple algorithm. 

4/ You mention the “rounding term” r_d(\xi) at line 139 and 141 but is it defined ? I mean else than in Algo 1 where it is only defined up to a constant.

5/ In Remark 3, you talk about known versus unknown misspecification “levels”, is that \eta_d, or perhaps \gamma_d ? On line 110, you call \eta_d the “induced error with low-dimensional approximation”, but it’s actually the “misspecification level”. 
Also, related wording issue, you say on line 116 that \gamma(d) quantifies the sub-optimality of the identified arm, but \gamma_d is just the max of the \eta_d(x) so it’s more like a misspecification error rather than a gap. I really don’t understand why you mention the gap there. 

6/ The “rounding” procedure is strangely named. Somehow I’d call it the “scheduling” or something of the like.
“Complexity measure” \rho_d is a bit hard to understand intuitively, at least for the non-expert, which makes the RHS of Theorem 1 a bit difficult to sanity-check.
7/ Appendix B.3 : I don’t understand line 603 (page 19) how you define x_1=-1/\sqrt{D}, I guess you are missing the rest of the definition such that x_1 satisfies the definition constraints listed in the paragraph.

8/ Appendix D: “We then define \bar γ(d) as a measure the quantity the optimality of the identified arm” (l 642) is hard to parse. Also, the statement of Theorem 5 is unclear: what is this display line? I guess you are missing “N\leq …”. I am still a bit confused about this section because I don't really understand what you *can* compute and that replaces the lack of knowledge of \tilde \gamma(d) in the algorithm. But I might have missed it.



Minor remarks: 
Line 137 : “itertion”
Line 158, use a “:” for the definition
Line 173 : “details of”->”on”
 NA


This paper studies best arm identification in kernel bandits. This paper provides an algoritihm to solve it, and extends it to neural bandit setting.
  
The paper looks incremental, in that it seems to be a combination of recent progress in pure exploration for linear bandits (cf. [13, arXiv:2006.11685]) and the kernel / neural bandit [46]. Since kernel case is usually similar to the linear case, and neural bandit is just kernel bandit with NTK, I do not see much novelty in this paper.

--- 

I find academic integrity concerns in this paper. I would argue for strong rejection. 
Please refer to Ethical Concerns.

---

At Line 136, RAGE is never introduced before, and one cannot know what RAGE means until Line 282.

Meta-review: This is good paper that should be accepted to the conference. Reviewer 7Wf3 raised a concern about plagiarism and gave a very low score of 2 because of the plagiarism claim. However, the other three reviewers and I agree that there is no plagiarism in this paper; unfortunately the reviewer did not interact with us and did not update his score. Aside form the plagiarism claim there are plenty of comments from the reviewers that the authors should use to improve their paper further. 

Decision: accept