Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes EE-NeT for contextual bandits which contains two neutral networks: one for learning the underlying reward function and another for learning the potential gains of arms if explored. They prove that  EE-NeT achieves better regret bounds than state-of-the-art neural bandits algorithms for both UCB-based and TS-based. This paper also shows that EE-NeT outperforms existing linear and neural bandit approaches.
main_review: The paper is well written and outputs a better regret bound with an interesting idea. The reviewer has the one concern about the exploration network: 
     the proposed exploration net is mainly based on the error bound derived from Ban et al. 2021. This paper uses neural networks to learn the complicated function. This step makes the reviewer confused. We already have an analytical equation (even it has some unknown constants) for that error bound (lemma D.1). What’s the point of training a neural network to learn it instead of learning the unknown parts in that equation? Also,  why do we use the gradients as the input for the neural network instead of putting more known items in the input, e.g., A_t in the lemma D.1 (does this give more info to the training process)?  

summary_of_the_review: The reviewer doubts the motivation of using neural networks to the confidence found when we have an analytical equation for that.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies neural network-based contextual bandits. Different from existing works which utilize UCB or Thompson sampling for exploration, the exploration component of the proposed method EE-Net is also modeled with a neural network.

Based on theoretical tools on over-parameterized neural networks, it is proved that EE-Net achieves the regret bound of $\mathcal{O}(\sqrt{T\log T})$. In contrast, existing neural network-based contextual bandit methods, such as NeuralUCB and NeuralTS, achieves the regret bound of $\mathcal{O}(\sqrt{T} \log T)$ based on the theoretical tool of the neural tangent kernel.

Experiments are conducted on four public datasets to validate effectiveness of EE-Net, compared with a variety of contextual bandit methods including both neural-based methods and linear contextual bandit methods.
main_review: **Strengths**

The idea of utilizing neural networks to model potential reward gains for exploration is very interesting; it seems quite different from rule-based UCB and Thompson sampling.

**Weaknesses**

On the theoretical side, the assumptions on over-parameterized neural networks are utilized to prove the regret bound, which is tighter than NeuralUCB and NeuralTS. However, since the regret bounds of NeuralUCB and NeuralTS are derived from the assumptions of the neural tangent kernel, it is not clear whether the theoretical advantage comes from the method itself, or from the assumptions.

On the experimental side, it is not sufficiently rigorous to tune the hyper-parameters. It is said that 

> For all grid-searched parameters, we choose the best of them for the comparison and report the averaged results of 10 runs for all methods.
> 

In real applications, it is not possible to tune hyper-parameters in this way primarily due to the bandit feedback. It is necessary to avoid the risk of overfitting in the experimental setting.
summary_of_the_review: The idea is interesting and as far as I know it is novel. However, both the theoretical and experimental results should be verified more carefully. See Main Review for details.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies the neural contextual bandit problem, and proposes a neural-based bandit approach with a novel exploration strategy, called EE-Net. Besides utilizing a neural network (Exploitation network) to learn the reward function, EE-Net also uses another neural network (Exploration network) to adaptively learn potential gains compared to currently estimated reward. Then, a decision-maker is
constructed to combine the outputs from the Exploitation and Exploration networks. Theoretical guarantees and extensive experiments are provided to demonstrate the performance superiority of EE-Net over existing linear and neural bandit algorithms.

main_review: Strengths: 

1. The studied neural contextual bandit is very interesting and important in the area of applying neural networks to online learning.
2. The proposed algorithm uses a novel and interesting network architecture, EE-Net, which is different from prior UCB-based [Zhou et al. 2020] or Thompson Sampling [Zhang et al. 2020] based neural bandit algorithms.
3. The authors give rigorous theoretical analysis and tighter regret bounds. While I do not think an improvement of \sqrt{log T} in the regret bound is significant, this does demonstrate the theoretical soundness of EE-Net.
4. The experiments are conducted on extensive real-world datasets, and the empirical results validate the superiority of the proposed EE-Net.


Weakness:

It would be better if the authors can highlight the technical challenges in regret analysis and compare their analytical techniques with prior neural contextual bandit works.


---After Rebuttal---

I have read the authors' response. They clearly explained the technical challenges of this paper, which well addressed my concerns. I appreciate that their algorithm only stores and updates the $O(p)$-dimensional parameter in neural networks, and does not need to explicitly maintain extra $O(p)$-dimensional vectors or $O(p \times p)$-dimensional matrix. This point is very good, and I have never seen it before in the neural contextual bandit literature. The theoretical improvement of removing the $d$ or $\tilde{d}$ is also interesting.
I am glad to increase my score from 6 to 8.
summary_of_the_review: Overall, I think that the studied neural contextual bandit problem in this paper is well-motivated. The idea behind the proposed algorithm is novel and effective. Tighter theoretical guarantees and extensive experimental results are provided to demonstrate the superiority of the proposed algorithm. Therefore, I recommend “weak accept”.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a neural net based bandit approach with a novel exploration strategy. Specifically, the solution uses an exploitation network to estimate rewards for each arm and an exploration network to predict the potential gain compared to current reward estimate. It then uses a decision maker to select arms using one of the linear or nonlinear modes. The key differentiator/strength of the paper is using neural net based exploration strategy and demonstrating that the proposed solution has a tighter bound than the existing state-of-the-art bandit algorithms. 
main_review: First of all, authors provided a very clear review on the various neural network based contextual bandits and illustrated their proposed approach very clearly through discussing the novelties they claim with their proposed technique in comparison with existing work in the area. The proposed technique has clear advantages over the existing literature and I believe it can be easily applied to existing domains that benefit from contextual bandit approached. The proposed method is well supported by theoretical findings and good empirical evidence. Authors also provide their source code and the datasets they use, which is a very important aspect for interested readers/researchers.

For the evaluation, I am not an expert on the contextual bandits but from educated guess, the experiment results are extensive, solid and impressive. They compared with existing state of art techniques using benchmark datasets and regret analysis together with ablation experiment results are presented.

I have a few comments/questions/concerns:
1.	Authors provide regret bounds’ complexity analysis for their technique. However, what are some concrete time complexity results for the proposed technique? My concern is, for online learning algorithms “time it takes to respond should be” quick, I am really curious to see how the techniques’ response time compares with the existing techniques response time in literature. Can authors provide this comparison analysis? Is there any overhead that the neural net based exploration brings to the solution? 
2.	Parallel to 1st question, How does the solution scale with larger datasets?
3.	Figure 1 results are reported over 10 runs, I am curious why such a low number of runs are performed, not 30 for instance? Is it due to time-constraints or memory constraints for the machines used during experiments? Can you provide details for the machine co

summary_of_the_review: Overall, I liked the paper, very clearly written, organized. I am convinced with the novelty of the technique being proposed, I had a few concerns around some applicability issues in real life due to time complexity. I have detailed my questions in the previous box.
review_title: null

Meta-review: Summary: This paper studies the neural contextual bandit problem, and proposes a neural-based bandit approach with a novel exploration strategy, called EE-Net. Besides utilizing a neural network (Exploitation network) to learn the reward function, EE-Net also uses another neural network (Exploration network) to adaptively learn potential gains compared to currently estimated reward.

Discussions: The reviewers appreciated the novelty and the quality of the ideas and results in this paper. Most questions were about details in algorithm design choices and in the analysis. The authors have addressed these questions and updated their draft. The reviewers have now reached a consensus and recommend accepting this paper.

Recommendation: Accept.
Decision: Accept (Spotlight)