Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a variant of BNN with stochasticity added to the activation layer (auNN). It's proposed to solve the underestimation of uncertainty for instances located in-between two clusters of training points. It also shows comparison to DGP and indicates that auNN is better than DGP by requiring less inducing points and better suited for deep architectures. There is a strong connection between auNN with DGP which is discussed in the later part of the paper which I appreciate. 

However, I think there are still a few things needed to be more explicitly stated: In page 6, "Whereas auNN considers combinations of the same Dl 1D functions to define the (l + 1)-th layer from the l-th one, DGP-add describes Dl different 1D functions for each unit in the (l + 1)-th layer. " This is not quite clear, why auNN combines the same functions while DGP combines different. From figure 1, it seems that the only difference between auNN and DGP is whether f1,f2,f3 are summed up into a scalar a or not. Which part is different or same? I think it might need a bit math to demonstrate that and it would be helpful to draw figures for multiple nodes in the lth layer rather than just one. Cuz from the description under eq1 and figure 1, my understanding is f1, f2 and f3 in the l-1th layer are combined together with w to get a scalar a. Then a 1D GP is constructed from this scalar a to generate a scalar f for the lth layer. When we want another f in the lth layer, we just use another w to linearly combine f1,f2 and f3 (l-1th layer) and generate f using another GP from a. While in an ordinary DGP, fd in the lth layer is from a kernel taking f1,f2,f3 (l-1th layer) all together. So presumably, fd in auNN is from different GP functions with different kernels while fd in DGP is from the same GP with the same kernel. I just think this connection/distinction is a bit confusing to me. A better way to clarify it is using mathematical equations to express the unit in auNN and DGP respectively, including what the kernel looks like, what the expression for fd is, etc. There are such definitions for auNN but not DGP, and conceptual-level description is not clear enough. 

Moreover, although the paper emphasized the difference between auNN and DGP, I still think auNN is a specific/simplified version of DGP. Here is the reason: in auNN, f1 is only dependent on a1 and f2 is only dependent on a2. This is saying, the joint distribution of (f1,f2) has a block diagonal covariance and the blocks are different for each d. In DGP, f1 and f2 are usually considered to be independent to each other, thus the joint distribution of (f1,f2) also has a block diagonal covariance but the blocks are the same. But people can make DGP more general by allowing different blocks (kernel functions) for f1 and f2. Another difference is the input to the kernel (It would be appreciated too if the authors can explicitly write out the kernel definitions for auNN and DGP for easy understanding). For DGP, the input to fd (lth layer) is a D-dimensional vector (f1,f2,...,fD) in the l-1th layer; different fd share the same inputs. For auNN, the input to fd (lth layer) is a scalar ad=wd*(f1,f2,...,fD), and different fd have different inputs since wd is different. This part is more like different designs of the kernel. If so, then the inference method for DGP would be just applicable to auNN (adding the optimization for w).

In sum, this design of kernel and independent fd is interesting but the overall novelty is not strong enough from both the methodology perspective and inference perspective. 

A few other things:
1. In the intro, it says "First, it has been recently shown that BNNs with well-established inference methods ... underestimate the predictive uncertainty for instances located in-between two clusters of training points. Second, BNNs do not extrapolate sensibly to out-of-distribution (OOD) data. Both issues are illustrated graphically in Figure 3." I don't see both issues in Figure 3. Or more precisely speaking, what's the difference between these two points?

2. Define d when it first shows up in the paper. 

3. Define the dimensions for mud, Kd.

4. It shows that auNN requires less inducing points but the number of parameters is actually more than DGP due to w. It would be helpful to have such a discussion too. Since people know that the optimization for DGP could be tricky even with two parameters (e.g., length scale and marginal variance in RBF) in the kernel. Now the kernel has more parameters, how it would behave during the optimization and at the optimum? I think the larger variance of auNN across layers might be due to w, so comparing auNN with DGP-add might a bit unfair. 

----------------------------------------------------
I raise my score to 6 after the rebuttal given the clarification the author added to answer my confusion.

review_title: the technique is interesting but not novel enough

final_score: rating: 6: Marginally above acceptance threshold
review_content: Either putting the uncertainty on the weights (e.g., Bayes by BP) or on the activation (e.g., fast dropout or variants of natural-parameter networks [2,3] or Bayesian dark knowledge [4]) or both [1] have been investigated before. The idea of moving the uncertainty from the weight to the activation function is not new. One could argue that VAE-style parameterization or local reparameterization trick is also a kind of methods that put uncertainty in the activation function. In fact the proposed method does involve the reprarameterization trick in each layer as shown in Eq. 7.

The hypothesis that introducing stochasticity in the activation functions is effective in handling uncertainty estimation has also been investigated before [1,2,3].

In terms of uncertainty estimation OOD data, note that BDK [4] and natural-parameter networks (NPN) [1] also claimed satisfactory performance in uncertainty estimation OOD data (see figures in both papers on the toy regression dataset). Therefore it would be interesting to make proper comparison. 

This work is heavily built on the DGP work with help from the reparameterization trick. The author mentioned their differences at the end of Sec. 2, emphasizing that DGP define the function on D^{l-1} dimensions while the proposed method is on 1D. Looking from another perspective, the proposed method combined with the previous linear layer can also be seen as ‘defining the function on D^{l-1} dimensions’. In this sense, I would guess that the different is really on the factorization, i.e., how they factorize the linear plus nonlinear operations.

Given that this is built on GP, a complexity analysis on running time (both for training and inference) is provided. I am wondering what is the scale for M, i.e., the number of inducing points, for the method to work properly. Also, it would be good to provide and compare the actual running time.

Experiments are either on toy datasets or two very small data UCI datasets. It would be more convincing to evaluate on larger/higher-dimentional data. 

In Figure 3, it is mentioned that BNN underestimate uncertainty between two clusters. I was wondering whether this is true for all types of BNNs. As an example, see Figure 1 of [1], where areas with fewer points do result in higher uncertainty from the PBP and NPN models (which also handles uncertainty in the activation). This comes back to my concern on the experiments. It would also have been stronger if more baselines are included to make a more convincing case.

Another important experiment that is missing is to include DGP in Figure 3. Would DGP perform similarly or even better than auNN?

Another problem with the toy dataset in Figure 3 is that: it is hard to say which way of extrapolation is ‘more correct’. Should the model extrapolate like BNN does or like fBNN does. Given that the input is in a continuous space and that inside each cluster the output is indeed increasing with the input, I would say extrapolating as BNN does also makes sense.

The presentation may be improved. For example, it would be helpful to make it clear what U^l denotes from the beginning. It would also be good to better structure Sec. 2.

[1] Natural-Parameter Networks: A Class of Probabilistic Neural Networks, NIPS 2016
[2] Feed-forward Propagation in Probabilistic Neural Networks with Categorical and Max Layers, ICLR 2018
[3] Sampling-free Epistemic Uncertainty Estimation Using Approximated Variance Propagation, CVPR 2019
[4] Bayesian Dark Knowledge, NIPS 2015

---------
After rebuttal

After the discussion phase, I agree that with R3 that in cases such as with periodic patterns, auNN would be a better way to incorporate such knowledge. In this case, the paper would be stronger if related experiments on such patterns are included, to highlight what the key difference between auNN and typical BNN.

Gap uncertainty is indeed a real problem in BNN, and I am glad that auNN performs well on this, this is also why I would like to see more comparison to other baselines (e.g., NPN) with good performance on gap uncertainty, to better gauge its improvement/capability (e.g., see https://arxiv.org/pdf/1611.00448.pdf Figure 1(right) where there is also a gap in the middle, although not as large as the example from the auNN paper).

If this paper is accepted, it would be great if the author could include R3’s summary during the discussion into the final version (see below) as well as proper BNN baselines, which I think will be quite helpful in positioning the current work:

‘Typical BNNs use stochastic weights and deterministic activations, while the authors’ model (auNN) uses deterministic weights and stochastic activations. You are definitely correct in asserting that uncertainty in weights manifests as uncertainty in activation functions. Directly modeling activation function uncertainty however provides distinct advantages, has not been studied carefully before.’

Experiments mentioned in the first paragraph would also be a good addition to make the paper’s point.

review_title: Official Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors capture uncertainty in deep networks by employing stochastic activation functions but deterministic network weights.  They place a Gaussian process (GP) prior over activation functions, propose a customized GP kernel, and demonstrate that activation uncertainty leads to both more sensible extrapolations and better-calibrated interpolation uncertainties when compared to standard Bayesian neural networks (BNNs)  with weight uncertainty and deterministic activations. 

The observation that BNNs have too many parameters for meaningful posterior inference is widely acknowledged; modeling activation uncertainty instead of weight uncertainty is an interesting approach for alleviating this issue.  While GP based activations have been considered before, the paper does a particularly good job of executing on the idea and clearly demonstrating its advantages. The experiments are well thought out and nicely illustrate the benefits afforded by the paper's contributions.  Overall, I  enjoyed reading this paper and only have minor quibbles.

1. As with any GP based approach computational scalability is a concern. While it is reassuring that only a modest number of inducing points are needed, the complexity of one epoch: $O(NM^2(D_1 …+ D_L))$ suggests that it would be challenging to scale auNN to deeper / wider networks. Section 3.4 demonstrates scalability to large data but it does not demonstrate scalability to large architectures. It would be good to include a discussion of this issue and how the authors envision scaling the approach to more standard deep learning architectures. Table 3 should include timing numbers for both standard variational BNNs and functional BNNs (even if they are not competitive in terms of performance). Having these numbers would help guide follow up work interested in replacing GPs with (potentially more scalable) parametric function approximations.
2. The choice of the GP kernel seems to have a large effect on the learned activations (Figure 8). It would be interesting to consider uncertainty over GP kernels as opposed to a-priori fixing all activations to be drawn from GPs with the same kernel and potentially allow different layers or different units in a layer to prefer activations drawn from GPs with different kernels.
3. I found the sentences (section 3.1, page 6, last paragraph) highlighting differences between a deepGP with an additive kernel and auNN confusing. I think the point being made is that in auNN different nodes in layer $l+1$ use a *shared* set of distinct $D_l$ functions, with each node weighting these functions differently. While in DGP there are no shared functions. From the text it almost sounds like that in auNN functions in layer $l$  (all $D_l$ of them) are the same, which doesn’t seem to be the case. 
4. The observation that many BNN inference techniques struggle with appropriate “gap” uncertainty was concurrently made by Foong et al., and Yao et al., https://arxiv.org/pdf/1906.09686.pdf 
review_title: Clearly written and well motivated

final_score: rating: 7: Good paper, accept
review_content: 
In this paper, the authors contribute to the recent trend of replacing uncertainty in the weight space in favor of uncertainty in the function space. Their main contribution, of implementing activation-level uncertainty, is a valuable one and will inspire future research. 

Overall, this paper is well-motivated, clear-to-read, novel, and interesting. The mathematical reasoning is sound and the Figures are of excellent quality: in particular, Figure 2 is a very good illustration of the paper's core concepts. Figure 6 is also clear, thought-provoking, and informative. 

To improve the work, the authors could devote more time to discussing the limitations of the proposed method. In particular, the authors make a point of the scalability: however, the datasets used in the network, while having many training samples, are quite small in their dimensionality (D=28 for HIGGS and 18 for SUSY). While the work compares extensively against deep GPs, a broader set of comparisons would improve the paper. Work such as 'Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning' shows that Bayesian inference can be scaled to work on ImageNet, using resnet-50 — although this work is not cited in the paper, it does show that existing approaches can scale to large networks and large datasets. Finally, the computational complexity of the proposed method should be outlined more explicitly — although there is some content in a footnote, a table comparing different models and their requirements would be of interest. 

In terms of the modeling, I would also appreciate some discussion of the 'independence between units' assumption in the probabilistic model. The assumption that the uncertainty of a given unit depends only on the value of its activation seems to me to be a fairly strong one, so some more explicit discussions of the possible consequences of this choice would improve the work. 

One structural change I would recommend is to move the 'related work' section forward: it currently precedes the conclusion, whereas I feel it would be better earlier in the manuscript to set the scene more thoroughly. 

There are a couple of more minor points to address: 

* I find the 'Introduction' section of the work to be poorly-researched. All of the underlying motivations of modeling uncertainty have references to deep-learning papers from the last 5 years or so. It has the unfortunate effect of implying that people in these fields had never thought about these issues prior to the current deep learning boom. Using the cited papers would be fine if they were phrased more like "deep learning has applied uncertainty estimation to the following critical applications", but if the authors want to support the statements they've actually written, they should do more research. A possible starting point would be Bishop, §1.5.3. 
* The authors refer often to 'reversing' when they mean 'reverting' (to revert is to return to a previous condition, which is exactly the meaning the authors want to convey). 
* The paper https://openreview.net/pdf?id=B1l08oAct7 could also be referenced and discussed. 

review_title: Activation-level uncertainty is an interesting method to model uncertainty in the function space. 

Meta-review: This paper presents  a new approach to model uncertainty in DNNs, based on deterministic weights and simple stochastic non-linearities, where the stochasticity is encoded via a GP prior with a triangular kernel inspired by ReLu. The empirical results are promising. The comments were properly addressed. Overall, a good paper.
Decision: Accept (Poster)