Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: * Edit: I increased my rating to 6. The authors fixed the first error I pointed out below. Regarding the second point: I still think it is possible to take a limit of sigma -> 0 in MNF, which makes the methods very similar.

The authors propose a new method of defining approximate posteriors for use in Bayesian neural networks. The idea of using hypernetworks for Bayesian inference is compelling, and the authors show some promising first results. I see two issues, and would be willing to increase my rating if these were sufficiently addressed.

- The paper says it uses an "isotropic standard normal prior on the weights of the network". However, the stochastic part of the generated weights (i.e. the scales) is of a lower dimension than the weights. It seems to me this means that the KL divergence between prior and posterior is undefined, or infinite, as the posterior is only defined on a sub-manifold. What exactly is the loss term that is added to the training objective? And how is this justified?

- The instantiation of Bayesian hypernetworks that is used in experiments seems to be a special case of the method of multiplicative normalizing flows as proposed by Louizos and Welling and discussed in this paper. If the variances / sigmas are zero in the latter method, their approximation seems functionally equivalent to Bayesian hypernetworks (though with different parameterization). Is my understanding correct? If so, the novelty of the proposed method is limited.
review_title: interesting idea. not rigorous. limited novelty

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes Bayesian hypernetworks to carry out Bayesian learning of deep networks. The idea is to construct a generative model capable of approximating the posterior distribution over the parameters of deep networks. I think that the paper is well written and easy to follow. 

I like the idea of constructing general approximation strategies for complex posterior distribution and the proposed approach inherits all the scalability properties of modern deep learning techniques. In this respect, I think that the paper tackles a timely topic and is interesting to read. 

It is not entirely clear to me why the Authors name their proposal Bayesian hypernetworks. This seems to suggest that also the hypernetwork is infered using Bayesian inference, but if I understand correctly this is not the case. 

I have some comments on novelty and realization of the experiments. In the positioning of the work in the literature, the Authors point out that hypernetworks have been proposed before, so it is not clear what is the actual novelty in the proposal. Is it the use of Real NVPs and IAFs as hypernetworks? These methods have been already proposed and extensively studied in the literature, and even if they have been adapted to be hypernetworks here, I believe that the novelty is fairly limited. 

The experimental part is interesting as it explores a number of learning scenarios. However, I think that it would have been useful to add comparisons with standard variational inference (e.g., Graves, 2011) for deep networks to substantiate the claims that this approach underestimates uncertainty. I believe that this would strengthen the comparative evaluation. 

I think the paper would have made a stronger case by including other approaches to approximate posteriors using generative models. For example, the variational Gaussian process paper sounds like an ideal method to include here. 

[1] D. Tran, R. Ranganath, and D. M. Blei. Variational Gaussian process. arXiv preprint arXiv:1511.06499, 2015.
review_title: Interesting paper - novel enough?

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents Bayesian Hypernetworks; variational Bayesian neural networks where the variational posterior over the weights is governed by a hyper network that implements a normalizing flow (NF) such as RealNVP and IAF. As directly outputting the weight matrix with a hyper network is computationally expensive the authors instead propose to utilize weight normalisation on the weights and then use the hyper network to output scalar scaling variables for each hidden unit, similarly to what was done at [1]. The main difference with this prior work is that [1] consider these NF scaling variables as auxiliary random variables to a mean field Gaussian distribution over the weights whereas this paper attempts to posit a distribution directly on the weights via the NF. This avoids the nested variational approximation and auxiliary models of [1], which can potentially yield a tighter bound. The proposed method is evaluated on extensive experiments.

This paper seems like a plausible idea with extensive experiments but the similarity with [1] make it an incremental contribution and, furthermore, it seems that it has a technical issue with what is explained at Section 3.3. More specifically, if you generate the parameters \theta according to Eq. 7 and posit a prior over \theta then you will have a problematic variational bound as there will be a KL divergence, KL(q(\theta) || p(\theta)), with distributions of different support (since q(\theta) is defined only along the directions spanned by u), which is infinite. For the KL to be valid you will need to posit a prior distribution over `g`, p(g), and then consider KL(q(g) || p(g)), with q(g) being given by the NF. From the experiment paragraph at page 5 though I deduct that you instead employ “an isotropic standard normal prior over the weights”, i.e. \theta, thus I believe that you indeed have a problematic bound. How do you actually compute logq(\theta) when you employ the parametrisation discussed at 3.3? Did you use that parametrisation in every experiment?

Other than that, I believe that it would be interesting to experiment with a `full` hyper network, i.e. generating directly the entire parameter vector \theta, e.g. at the toy regression experiment where the dimensionality is small. This would then better illustrate the tradeoffs you make when you reduce the flexibility of the hyper-network to just outputting the row scaling variables and the effect this has at the posterior approximation.
 
Typos:
(1) Page 3, 3.1.1 log(\theta) -> logp(\theta).
(2) Eq. 6, it needs to be |det \frac{\partial h(\epsilon)}{\partial \epsilon}|^{-1} or |det \frac{\partial h^{-1}(\theta)}{\partial \theta}| for a valid change of variables formula.

[1] Louizos & Welling, Multiplicative Normalizing Flows for Variational Bayesian Neural Networks.
review_title: Incremental idea with a potential technical issue.

final_score: rating: null
review_content: * You use the same noise sample for all examples in a minibatch. Is this because the computation in the hypernet otherwise becomes too expensive? An advantage of your proposed approach seems to be that by outputting only the scales of the parameters you could easily use different scales for different examples as far as the primary network is concerned.

* You claim to show "that BHNs act as a regularizer, outperforming dropout and traditional mean field". However the results shown in e.g. table 1 for CIFAR-10 seem to be quite a bit worse than previous SOTA results obtained with dropout. Why the difference?

* Please expand the caption in Figure 3. Is this MNIST? With the full training set or a restricted set?

* How do your anomaly detection results compare against methods that use ensembles? (e.g. http://papers.nips.cc/paper/7219-simple-and-scalable-predictive-uncertainty-estimation-using-deep-ensembles)
review_title: some questions

Meta-review: This paper presents a new method for approximate Bayesian inference in neural networks.  The reviewers all found the proposed idea interesting but originally had questions about its novelty (with regard to normalizing flows) and questioned the technical rigor of the approach.  The authors did a good job of addressing the technical concerns, causing two of the reviewers to raise their scores.  However, the paper remains just borderline and none of the reviewers are willing to champion the paper as their questions about novelty and empirical evaluation remain.  The reviewers all questioned fundamental technical aspects of the paper (which were clarified in the discussion), indicating that the paper requires more careful exposition of the technical contributions.  Taking the reviewers feedback and discussion into account, running some more compelling experiments and rewriting the paper to make the technical aspects more clear would make this a much stronger submission.

Pros:
- Provides an interesting idea for approximate Bayesian inference in deep networks
- The paper appears correct
- The approach is scalable and tractable

Cons:
- The technical writing is not rigorous
- The reviewers don't seem convinced by the empirical analysis
- Incremental over existing (but recent) work (Luizos and Welling)
Decision: Reject