Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Established in deep learning literature that Neural Network (NN) is related to Neural Kernels (NK), especially between the optimization dynamic of infinitely wide NN has shown to be mostly captured by the NTK at initialization. Motivated by the correspondence, this paper proposed to guide bandit policy with NK-GP (or NK-TP to take account for varying noise level in feedback), which gives both reward predictions and uncertainty estimations at the same time. Their main algorithm and its variants are referred to as NK bandits.  

Empirically, this work shows NK bandits outperform other baselines on most of the tested UCI datasets.

As another claimed contribution, this paper designed a framework to empirically measure one bandit algorithm’s ability to learn representation and that to do exploration separately. 


strength_and_weaknesses: **Strengths**:
1. It gives a comprehensive side by side comparison between different bandit policies on nonlinear structure data: methods include NK bandits (multiple variants),  neural-linear (LiM2) bandits, neural bandits, linear TS / UCB and multitask GP.
2. Proposes a way to measure one algorithm’s ability to learn representation and the ability to do exploration, by regulating the bandit environment with the tuple $(\epsilon, \delta)$, whose two components are controlling the learning complexity and the exploration urgency respectively.
 
**Weaknesses**:
1. Algorithmic novelty: it feels to me that the NK bandit policies proposed in this work are not substantially different from existing NK bandits, especially the $NeuralUCB_0$ by [Zhou et al.](http://arxiv.org/abs/1911.04462) in Appendix E, except for using separate kernels for different actions. There is some discussion in section 3.1 addressing this difference, but not sure if the separate kernel is a substantially improved design for general non-linear bandits or only fits for classification-converted bandits.
2. Results interpretation: While the paper gives a thorough description of experiment results, the interpretation is less inconclusive. Also, the wheel dataset designed to separate representation learning and exploration is somehow simple in structure due to the low-dimensionality in context ( which I assume to be $2$ if directly following the setup in [Riquelme et al.](http://arxiv.org/abs/1802.09127)). So it’s questionable whether the results give insights of NK bandits that generally apply to applications.

clarity,_quality,_novelty_and_reproducibility: Evaluation/comments on novelty is referred to **strength/weakness**.

**Clarity, Quality**: Though the rest of this paper is written clearly, the sections related to the experiment design (3 and 4) are somewhat difficult to follow for ones who are not familiar with the couple of papers cited in line: better to pull up some technical/mathematical details from the appendix.

**Reproducibility**: Currently the code link is hidden, but I’d like to trust the authors that their results are reproducible.,

summary_of_the_review: This work makes contribution by empirically comparing different bandit policies on nonlinear data and proposing a novel way to indepedently measure representation learn and exploration as well. But it is limited by not providing substantially improved method from existing work for the purpose of solving nonlinear bandits and not much insights on NK bandits that generally apply to applications were drawn from the experimental results.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper the authors propose an empirical analysis of neural kernel bandits, proposed as a UCB based alternative to Bayesian neural networks. Bayesian NNs have a high computational requirement as often an ensemble must be maintained in order to bootstrap a sample from the posterior, as in the case of Thompson sampling. The paper studies a version of kernel UCB where the kernel is realized by a linearized neural network in the limit where the width of the network goes to $\infty$. The experiments compare against other neural network based bandit algorithms and show that there is a benefit for using NK bandits in terms of generalization ability and computational requirements. th authors also propose a practical setting where the ability of the algorithm to learn representations is studied and show a benefit for using NK bandits here.
strength_and_weaknesses: The paper consider NK bandits, or as studied in [1], termed as NeuralUCB_0 type algorithms.
In the highly cited paper [1], it is shown that when the neural network representation is allowed to change while training, and the changing representation is used to compute uncertainty estimates, the algorithm outperforms the fixed kernel representation based algorithms. This is in contrast to the results presented in this paper, which show the opposite phenomenon.

As the scale of the problem grows, the authors themselves remark in the paper that the complexity of computing the neural kernel is significant. However, the proposed solutions in the paper to remedy this issue are not satisfactory in my opinion.

I also generally find it uncomfortable that the authors study an algorithm which does not use the ability of a neural network to learn a good representation for a problem, and rather use a fixed representation that an infinitely wide network would realize. While there are theorems showing equivalence in a highly overparameterized limit, this is not the typical limit in which practical multi-layer neural network architectures operate. As such the studied algorithm is inherently linear, and so I do not expect it to outperform nonlinear bandit algorithms as the problem scale grows.


[1]: https://arxiv.org/pdf/1911.04462.pdf
clarity,_quality,_novelty_and_reproducibility: I believe the clarity of the paper can be improved, by formally introducing and defining the neural kernel bandit algorithm more clearly and earlier in the paper. Otherwise it is largely free of typos, and reads OK.

In my opinion, the results are not significantly novel, although there is some value in carrying out an empirical study of NK bandits. However, I am not convinced that these results extend to larger scale datasets and problem settings, and I worry about the scaling issues in the considered algorithm. Both of these are not adequately addressed in the current paper, and I believe are the hard problems to study in practice.

I have not checked the reproducibility of the experiments.
summary_of_the_review: - I believe that the studied algorithm is not super significant and misses the point of neural network function approximation.
- I believe that there are scaling issues with the algorithm that cannot be easily resolved in spite of some proposed approaches. I think it requires a larger scale empirical study to assess whether these algorithms actually are statistically and computationally efficient.

Given these two facts, I propose a reject/weak-reject rating for the paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper performs an empirical study of neural kernel bandit algorithms, to investigate the efficacy of these algorithms from different aspects such as representation learning and exploration ability. To achieve this, the paper has proposed a novel benchmark for contextual bandits which allows separately evaluating the efficacy of an algorithm in terms of representation learning and exploration.
strength_and_weaknesses: Strengths:
- The empirical insights drawn from the empirical evaluations of this paper are interesting and can be of interest for the neural bandit community and the bandit community in general.
- The proposed benchmark with the additional ability to evaluate the ability of an algorithm to learn representation is also interesting and can be potentially useful for the broader community.

Weaknesses:
- My biggest concern is regarding the comparison with the recent works on neural contextual bandits which explicitly train a neural network to predict the reward, including NeuralUCB and NeuralTS. **Firstly**, Figure 2 shows that NeuralUCB and NeuralTS are consistently outperformed by LinearUCB and LinearTS. This is surprising to me, because in the previous papers on NeuralUCB (Zhou et al., 2020) and NeuralTS (Zhang et al., 2020), they have both also performed experiments using the UCI datasets and had shown that both NeuralUCB and NeuralTS consistently outperform linear bandit algorithms. What's the reason for this discrepancy between your observations and theirs? **Secondly**, it is mentioned in the Introduction that "...NKs have been shown to lack the full representational power of the corresponding NNs...". The same statement has also been made by a few other papers such as NeuralUCB. As a result of this, the use of a neural network to predict the reward (i.e., using an NN as the term $\mu_{a,t}$ on line 8 of Algorithm 1) is important for achieving good performances in neural bandits. Therefore, it is surprising that NeuralUCB and NeuralTS, which indeed use NNs for reward prediction, are significantly outperformed by the proposed algorithm which does not use NNs for reward prediction. Furthermore, in fact, the NeuralUCB paper also proposed another algorithm NeuralUCB0 which does not use NNs for reward prediction but instead simply treats NTK as a kernel in kernelized bandits. In this sense, NeuralUCB0 is similar to the proposed algorithm when NTKGP is used. But in their paper, they have shown that this NeuralUCB0 is consistently outperformed by NeuralUCB, which they have attributed to the fact that NeuralUCB0 does not use NNs for reward prediction.       
These two concerns here have broader implications, regarding whether it's useful/necessary to explicitly use an NN to predict the reward.
- The technical contribution of the paper may be limited, since the proposed algorithm is a straightforward combination of existing methods.
- Section 3.2: If I understand correctly, for the disjoint model, when estimating the posterior distribution for the reward of an arm, you only use the previous observations collected for this arm? So the observations from all other arms are not used? Isn't this a waste of data? Please clarify whether I misunderstood.
- It's unclear to me which step of Algorithm 1 requires training a neural network?
- (minor) Section 2, first paragraph, third line: "inite-width" should be "infinite-width".
- (minor) Section 2.1, second last paragraph, first line: UCB is in fact not a "stochastic" policy.

Other comments:
- A concurrent work (paper [a] below) has also empirically evaluated neural kernel bandit algorithms in other real-world problems of autoML and reinforcement learning, and hence should be referenced.       
[a] Sample-Then-Optimize batch neural Thompson sampling, NeurIPS 2022.
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well written.

Quality: The empirical evaluations are comprehensive and hence of high quality.

Novelty: The technical novelty may be limited, since the proposed algorithm is a straightforward combination of existing methods.

Reproducibility: The code is submitted for reproducibility.
summary_of_the_review: I think the paper provides some important empirical insights regarding neural kernel bandits. I have an important concern which is the first one listed under "Weaknesses" above, which is regarding some discrepancies with the observations from recent works on neural bandits which explicitly use NNs for reward prediction. If this concern is addressed well, I'll be happy to increase my evaluation.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents am empirical comparison between the performance of various neural kernels used in GPs for contextual bandit optimisation.  Comparisons are done using a modified wheel dataset, and the use of the student t-process to improve exploration in NK bandits is explored.
strength_and_weaknesses: The background is interesting and the various NN kernel do have some potential for use in GP models for sequential model-based optimisation and bandits.

My main difficulty with this paper however is that, in the end, it appears to come down to a simple experimental comparison of various kernels, some of which happen to derive from various models or analysis of neural networks.  I am not convinced that this suffices to have a significant impact.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and the experiments appear to be reproducible.
summary_of_the_review: The paper is well written, but as this boils down to a comparison of kernels in GPs I feel that there is insufficient novelty here to warrant acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers commented that this paper lacks novelty vis-a-vis NeuralUCB and NeuralTS. They also mentioned that this paper boils down to a non-extensive experimental comparison among various kernels. All recommended either a reject or weak reject. The authors' rebuttals did not manage to sway the opinion of the reviewers.
justification_for_why_not_higher_score: Because of the lack of novelty and the insufficiency of experimental comparisons. 
justification_for_why_not_lower_score: Can't go lower.
Decision: Reject