Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper adapts Bayesian linear regression to the setting of a limited memory replay buffer. The idea is to calibrate the prior mean and variance when the neural representation of context is updated. Overall the paper is well written and explained clearly. Some experiments are provided to show that the proposed method is able to achieve a performance competitive to Bayesian linear regression with infinite memory.

The result of this paper is interesting. But I am not sure if the current experimental results are convincing enough to justify the significance of the proposed method. 
1. The results in Section 4.2 seem to be following the setting in Riquelme 2018. These datasets are all in a supervised learning setting. It is a bit disappointing that the proposed method is not tested on RL datasets. 
2. No other baseline is provided in the experiments for comparison. 
    a. There are other methods in the literature to overcome catastrophic forgetting of neural networks, e.g regularizing the update of the network. How would that be compared to the proposed method?
    b. What about other methods, like [1]?
3. Most of the experiment details are missing. For example, how is the reward defined in section 4.3? What is the overhead in computation in practice, especially for the SDP?


Other comments:
1. Why would solving a SDP require only O(g^{0.5}) in section 3.1?
2. In the discussion in section 3, even if equation (5) and (6) can be exactly solved, how does the heavy tailed problem mentioned in section 2 been solved?


[1] Elmachtoub, Adam N., et al. "A practical method for solving contextual bandit problems using decision trees." arXiv preprint arXiv:1706.04687 (2017).
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes a neural linear bandits algorithm that is resilient to catastrophic forgetting when using limited memory.

The proposed algorithm Alg. 2 is similar to Thompson sampling for linear contextual bandits, Alg. 1, but using the last layer activation vectors as a linear feature, and also a different way of updating noise parameter prior and posterior is used based on Bayesian linear regression Eq. (2).

Alg. 2 also works with limited memory of history data, therefore after every time, the memory is refreshed, likelihood matching is used to calculate new Phi to make the likelihood (mean and variance) of reward estimation the same as it for the old feature. For mean matching, minimizing MSE Eq. (3) is used and for variance, solving PSD problem Eq. (6) is used.

The complexity of this algorithm is analyzed. And experiments are conducted to show that the proposed method is resilient to catastrophic forgetting and can achieve good cumulative reward results.

The proposed method is reasonable and the results look promising. However, I found several weak points as follows.

1. As mentioned Bayesian linear regression Eq. (2) is used to update noise prior and posterior, but this update has no theoretical guarantees as mentioned. As an algorithm mainly works under bandit settings, this is kind of undesirable.

2. This algorithm works with neural network-based features, but it is in nature not scalable as shown in the complexity analysis (linear dependence on action number). The linear feature is just replaced by the last layer activation of NNs. From this perspective, the experimental results just justify again that the NN feature is somehow powerful, which is as expected.

3. The likelihood matching can deal with catastrophic forgetting with limited history memory, which looks good. But the fact that it actually works for linear feature (last layer activation) together with realization assumption weakens this contribution a lot. The authors find using Eq. (3) is better than the exact mean matching Eq. (5), and there is no explanation for this, which kind of shows the proposed likelihood matching probably is not a good way when using full NNs rather than just linear features (last layer). On the other hand, the SDP seems also can only work under linear feature settings, and is not promising to be generalized to fully update for NNs.

Overall, this is a reasonable paper. However, on the one hand, as an algorithm mainly works under bandit settings, it is a lack of theoretical support. On the other hand, the linear feature setting weakens the contribution of likelihood matching to deal with catastrophic forgetting with limited memory. There are some questions of the proposed mean matching, and the matching is not able to generalize. 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Summary:

This work provides a memory-efficient nonlinear bandit algorithm based on deep neural networks. More specifically, the algorithm in this work only uses part of history information to save the memory usage. To overcome the catastrophic forgetting problem,  the authors provided novel covariance matrix approximation method. Experiment results also suggest that 

Pros:

The writing of this paper is very well. It provides enough introduction of the background of nonlinear bandit problems. The experiment settings and results are convincible. 

Cons:
- The core idea lacks solid theoretical supports. There is no regret bound result in this paper. The reason why I think the authors should add such theoretical proof is that it seems that the idea to construct new prior matrix instead of old one to avoid the catastrophic forgetting is not related to deep neural network at all. Thus, given existing regret analysis for Thompson sampling on linear bandit problems, the authors should also provide a simple analysis on linear case to show that the construction of prior matrix is indeed meaningful. 

- The experiment part does not show the accuracy the SDP solve needs. As the authors mentioned in Discussion part, below equation 6, it is very crucial to decide the accuracy the SDP solver needs. I suggest the authors add more details about the SDP solver in the experiment part.


Minor comments:

- The authors used DNN to minimize equation 3. Have the authors tried  a regularized MSE instead of (3)? I think to add a regularizer can further improve the results. 
- At page 13, below equation 8: why the first equality lacks?
review_title: Official Blind Review #3

Meta-review: Reviewers found the problem statement having merit, but found the solution not completely justifiable. Bandit algorithms often come with theoretical justification because the feedback is such that the algorithm could be performing horribly without giving any indication of performance loss. With neural networks this is obviously challenging given the lack of supervised learning guarantees, but reviewers remain skeptical and prefer not to speculate based on empirical results. 
Decision: Reject