Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a posterior approximation for BNN that models correlations between the layers weights. The paper begins by pointing out that, for any posterior distribution approximation, the optimal conditional posterior distribution over the top-layer weights given the weights of the previous layers has a closed-form in the case of Gaussian likelihood, which, due to Bayes rule, turns out to be product of the likelihood and the top-layer prior. Based on this insight the paper proposes to model each conditional posterior distribution over intermediate layers weights given previous layers weights following the same structure, that is, as a product of a 'pseudo-likelihood' over unobserved noisy activations and the prior for that layer. In order to make inference tractable, the paper proposes the use of global inducing points as well as noisy pseudo-observations of the activations of intermediate layers which are treated as variational parameters.  The paper also describes how such procedure applies to convolutional neural networks and how it can be applied for DGPs as well.

Even though the paper covers several topics, its presentation is clear and straightforward. The main issue I see with the paper is that the validation of the proposed model is done mostly in terms of accuracy and log-likelihood. As pointed out by [1], one of the main interests on posterior approximations that allow correlations between the weights is for capturing the uncertainty of the compositional structure. Based on this, I think the paper could increase its strength by describing the advantages on interpretability of the learned approximate posterior provides over a fully independent posterior approximation such as [2]. In particular, it would be useful to see how the conditional posterior distributions evolve given the previous layers weights.

[1] Ivan Ustyuzhaninov, Ieva Kazlauskaite, Markus Kaiser, Erik Bodin, Neill DF Campbell, and Carl Hen-
rik Ek. Compositional uncertainty in deep Gaussian processes. UAI, 2020.

[2] Hugh Salimbeni and Marc Deisenroth. Doubly stochastic variational inference for deep Gaussian
processes. In Advances in Neural Information Processing Systems, pp. 4588–4599, 2017.

review_title: see review

final_score: rating: 7: Good paper, accept
review_content: **Summary**: 
The paper proposes a posterior estimation for Bayesian neural networks (BNNs) and deep Gaussian processes (DGPs). The difference from the previous approaches is to use global inducing points which help to take into account correlations across layers. 

I find a paper well written and the results worth publishing. 

**Comments**:
“We found that the ELBO for methods that factorise across layers — factorised and local inducing — drops rapidly as networks get deeper and wider (Fig. 2). This is undesirable behaviour, as we know that wide, deep networks are necessary for good performance on difficult machine learning tasks. In contrast, we found that methods with global inducing points at the last layer decay much more slowly with depth, and perform better as networks get wider.” I wonder if it can be due to the increasing tails with depth in BNNs with Gaussian prior as stated by Vladimirova et al. (2019). According to their paper, the distributions on the units across the layers become heavier-tailed but knowing that the limit is Gaussian Process (Matthews et al., 2018; Lee et al., 2018), the distributions on the units for wider neural networks are closer to Gaussians (I think the center of the distribution reminds the Gaussian more and more). And as the methods that factorise across the layers use the assumptions of Gaussian units, the error is higher for shallow NNs. So maybe it is possible to keep in mind the “heaviness” for shallow NNs induces due to the correlations between the units and to improve the results. 

Lee et al. (2018) Deep neural networks as Gaussian processes, ICLR

Matthews et al. (2018) Gaussian Process Behaviour in Wide Deep Neural Networks, ICLR

Vladimirova et al. (2019) Understanding priors in Bayesian neural networks at the unit level, ICML



**Minor**:
- “initalization”
- I think you should stick either to American, either to British version of writing: “factorises”, “parameterise”, “initialization”, “initialize”
- “work using using infinite-width neural networks”
- “The next block show”
- “it is difficult design” -> to
- “downweighted” 
- “by coupling inducing inputs to inducing outputs” -> to induce 
- “the number of locations within an patch”
- “we need to be able sample”
- “these issues do not arises”
review_title: Approximate posterior taking into account correlations across the layers

final_score: rating: 7: Good paper, accept
review_content: The paper proposed a new way of doing Bayesian deep learning in which the optimal conditional posterior for the last layer weights could be reached if the inducing input $Z_0$ is chosen to be the input data $X$ and the pseudo-observation for the last layer $V^L$ is the observation $Y$. Instead of factorizing the inducing points The global inducing input $Z_0$ is propagated through the network to ensure posterior dependencies across layers. The authors also extend this idea to deep Gaussian processes so that the latent functions across layers are correlated. Experiments show better performance than previous methods on both synthetic and real datasets, without the need to anneal the weights for the KL term in ELBO.

Pros:
The idea of capturing the posterior dependencies between layers in BNN and DGP is great, and may have a good amount of significance in both areas.

Cons:
The description of the method itself is not very clear. For the BNN part, while Alg. 1 is easy to follow, how does it relates to the training process of BNN? The DGP part is even harder to understand -- what is the complete form of the variational distribution and the ELBO? How is the ELBO optimized? The paper didn't say much about these important details, and by just staring at Eq. (16) I really couldn't figure them out. Comparing to methods the paper compares with such as (Salimbeni and Deisenroth, 2017), Section 3 of this paper is not clearly written.

Other questions:

How does $N_{l-1}$ in Eq. (2) disappear in Eq. (14)?

Does this method apply to GP classification?

In Sec. 4.1, 100 inducing points per layer was used on a dataset with only 40 points. What if the number of inducing points is smaller? Does the proposed method still work as expected?


-----------------------

I thank the authors for their detailed reply. The revision after the initial review solves the concerns and questions I had. Very good work. I vote for accept.
review_title: Clever idea but not well explained about how it actually works

Meta-review: The paper proposes a variational inference method for Bayesian neural networks where the approximate posterior models the correlations between the weights at all layers, using the concept of “global” inducing points. 

Some concerns raised by the reviewers regarding how global inducing points allow us to capture uncertainty across the compositional structure and clarity of the manuscript have been addressed by the authors. However, the more general issue of interpretability has been left for future work. 

One of the main deficiencies of this paper is that it seems to ignore other scalable approaches that also provide more complex posteriors, for example, those based on stochastic gradient Hamiltonian Monte Carlo (see, e.g., https://arxiv.org/abs/1806.05490, and references therein). Overall, although there was support for this paper, it is unclear if approaches such as those presented here are really necessary. A comparison between the two methodologies maybe not only illustrative but required. 

Decision: Reject