Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper investigates the possibility of using hypermodels in improving the exploration of bandit problems. By using SGD for training the hypermodel parameters, this paper introduces a computationally efficient alternative to ensemble methods. The idea of the paper is novel and interesting; however, I do have several concerns, mainly from numerical experiments that I would like the authors to address those in the rebuttal.

1) My first and the most important concern is that the numerical experiments do not evaluate different aspects of the method. There are numerous ways to check the sensitivity of your method for the choice of hyperparameters that I think could be added to the appendix. In addition to testing various values for $\sigma_p$, $\sigma_w$, and $\nu_0$, I think that multiple experiments are missing:
    i) larger neural network
    ii) I was expecting to see what would happen without additive prior. It could be one of the baselines in Figure 3. Even though (Osband et al., 2018) discuss the effect of this extension, but the use of this model is not numerically justified. 
    iii) How the experiments are sensitive to the noise of the output variable? What will happen if you do not add noise?

There are also other experiments possible such as testing on a real scenario that would significantly improve the presentation of the work. This is not a requirement though.

2) I didn't get what is the purpose of the last term in the loss function defined in Section 2.1. Why you are looking preferring $\nu$ to be close to $\nu_0$? 

3) P4, "it is natural to consider linear hypermodels in which parameters a and B are linearly constrained." This sentence needs to be clarified. I didn't comprehend how you are dealing with large neural network issues.

4) In Section 6, I was expecting to see a simulation showing a comparison of linear hypermodel with hypernetworks. 


Minor:
* On P2, "informatino-directed" -> "informatino-directed"
* In the second paragraph of Section 2.1, it is mentioned that a hypermodel involves perturbing data. My understanding is that what is meant here by perturbing data is to add some noise to X. However, in the later formulae, there is no such thing as perturbing data. You could say that since our numerical experiments didn't show any improvement using data perturbation, we didn't include it in our notations. Please remove the confusion.
* very minor, but I would suggest using a different notation for $a$ in Sections 2.1 and 2.3 to remove any possible confusion. 
* I think that the summation in computing the variance of IDS should be over $\tilde{Z}_{x^*}$.


review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: The paper builds on a classical idea of sampling model parameters apart from learning them. Specifically, it combines hierarchical sampling with neural networks and proposes models that can help explore the parameter space efficiently. The proposal is evaluated appropriately.

What exactly do we mean by intelligent exploration? Is this quantified via the #samples needed or variance of sampled parameters? Or is it via regret? 

The paper is clearly written and the idea makes sense. However the experiments are essentially based on simulated data. It is not entirely clear as to how this would translate to real setups. 

Is it possible that the linear hypermodel is performing well because the data was generated according to a linear model in section 5?

If the baseline is a classical ensembling setup, then why not use classical performance measures to evaluate the benefit of hypermodeling? like accuracy etc. Why are we specifically talking about bandits? In other words, does the proposed hyper sampling allow for better weak learners in general as well? 
 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The authors demonstrate advantages of a linear hypermodel over an ensemble method in exploration guided by epistemic uncertainty. They perform an empirical study in the bandit setting and claim that their approach both outperforms the ensemble method and offers a significant increase in computational efficiency. The theoretical contribution is that they prove universality in the sense that an arbitrary distribution over functions can be represented by a linear hypermodel. The experiments support their claims. Some of the explanations, however, are confusing, and relations to prior work should be clarified. 

Figure 3 shows a surprisingly large performance gap between the hypermodel and the ensemble method as the number of actions increases. But how about comparing linear hypermodels with different index sizes? Do we also expect asymptotic improvement as we increase the index size?

Imprecise or confusing explanations in the paper:

1) Page 2, first Q: In theory, the effectiveness of the ensemble method should converge to that of the hypermodel as the ensemble size increases. They only tried ensemble size [10, 30, 100, 300] and then concluded that linear hypermodel can be effective regardless of the size of ensembles. Why?

3) Page 3, Section 2.1, second paragraph, first sentence: Please clarify a bit more what do you mean by perturbing data? Random shuffling of the dataset in each training epoch? What does ‘response variables’ mean?

4) Page 3, Section 2.1, second paragraph, last 2 sentences about $A_t$: we guess it should be $A_t ~ N(0, I)$ if $p_z$ is unit Gaussian according to the description in this paragraph. The current text claims it is the other way around, perhaps a typo?

5) Page 3, Section 2.1, first equation: Why take the inner product between $a$ and $z$ ? How does this reflect the randomized computation (the motivation for augmented random vector $A_t$)? The objective is to maximize the log-likelihood of the prediction under the Gaussian assumption. Please clarify the assumptions about random variables $Y_t$ at the beginning of this paragraph. 

6) Same place as in 5): Why regularize hypermodel parameters such that they are not too far from the initial vector? Is $\nu_0$ actually the additive prior model described in Section 2.5?

7) Page 3, Section 2.1, second equation: why multiply $|D|$ in the first term within the parentheses? Why not just $1/|D_tilde|$ to average the prediction error over the mini-batch?

8) Page 3, Section 2.1, second equation: is the cardinality of the index set $|Z_tilde|$ independent of mini-batch size? I.e. for each training data point there could be multiple models realized by multiple indices $z$

9) Page 4, Section 2.5: Why use this decomposition for training the hypermodel? If the intuition is to keep the initial weight small, what if we just simply initialize small values for $f_\theta(x)$ without decomposition?

10) Page 5, last second sentence: The notation of partition (the set notation after ‘Here,….’) is supposed to be $\hat{\mathcal{Z}}_{x^*} = \{ z\in \hat{\mathcal{Z}} | x^* in \argmax_{x} f_{g_{\nu}(z)}(x) \}$

Minor typos:

- Page 2, third paragraph: ‘…we compare their [efficacy] when used...’ ->  [efficiency] ?
- Page 2, the last paragraph before Section 2, first sentence: ‘Approaches to approximating TS and [informatino]-directed sampling...’ -> [information]

Relations to prior work:

1. Page 2: Hypernetworks (where one neural net learns to generate the weights of another net) are much older than this recent reference of 2016. One should relate this work to the original references since 1991 [FAST0-3a][FAST5][FASTMETA1-3][CO2] in section 8 of the overview http://people.idsia.ch/~juergen/deep-learning-miraculous-year-1990-1991.html 

2. Intro 2nd par: dropout was first published much earlier in 1990 as the stochastic delta rule: 
Hanson, S. J.(1990). A Stochastic Version of the Delta Rule, PHYSICA D,42, 265-272. See also arXiv:1808.03578, 2018. 

We might improve our rating provided the comments above were addressed in a satisfactory way in the rebuttal.

Edit after rebuttal: The authors replied: "Thanks for pointing out typos and citations that we will add." But apparently in the revised PDF this did not happen. 
review_title: Official Blind Review #2

Meta-review: This paper considers ensemble of deep learning models in order to quantify their epistemic uncertainty and use this for exploration in RL. The authors first show that limiting the ensemble to a small number of models, which is typically done for computational reasons, can severely limit the approximation of the posterior, which can translate into poor learning behaviours (e.g. over-exploitation). Instead, they propose a general approach based on hypermodels which can achieve the benefits of a large ensemble of models without the computational issues. They perform experiments in the bandit setting supporting their claim. They also provide a theoretical contribution, proving that an arbitrary distribution over functions can be represented by a linear hypermodel.

The decision boundary for this paper is unclear given the confidence of reviewers and their scores. However, the tackled problem is important, and the proposed approach is sound and backed up by experiments. Most of reviewers concerns seemed to be addressed by the rebuttal, with the exception of few missing references which the authors should really consider adding. I would therefore recommend acceptance.
Decision: Accept (Poster)