Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: The paper proposes a differentiable Bayesian neural network. Traditional BNN can model the model uncertainty and data uncertainty via adding a prior to the weight and assuming a Gaussian likelihood for the output y. However, it's slow in practice since evaluating the loss function of BNN involves multiple runs over the entire network. Also when the input data is non-stationary, the output function can not be differentiated with respect to the input data. The paper proposes to use an online code vector histogram (OCH) method attached to the input and output of a classical DNN. Input OCH captures the distribution of both input data and network weights, and the output OCH captures the distribution of the predictions. Since these OCHs are differentiable, the proposed DBNN model can be used for streaming input data with time-variant distributions. 

I think the idea is interesting and novel. It explores a different way of modeling distributions with DNN. Instead of adding priors, DBNN relies on histograms, which is usually used to describe distributions for discrete observed input data. So the paper is well-motivated. 

1. The paper needs more literature review in the area of data streaming. Papers, such as [1], have proposed to use a vector quantization process that can be applied online to a stream of inputs. This paper introduces the vector quantization but doesn't mention the use of it in streaming data in related work, which kind of blurs the contribution a bit. Moreover, it would be helpful for readers to learn about useful techniques for streaming data from this paper. 

[1] Hervé Frezza-Buet. Online computing of non-stationary distributions velocity fields by an accuracy controlled growing neural gas

2. I think the paper might need a bit more explanation about codevector, since it's not a very well-acknowledged concept in this field. The main issue for me to understand it is how to get these codevectors. When DBNN deals with streaming data and starts from no input, is the set of codevector empty at the beginning? The input data points are accumulated as codevectors? I hope the authors could clarify this process a bit more. 

3. Given the insufficient understanding of codevector, figure 2 is a bit hard to read. 1) (a)-(d) are figures for x0 at t=0, which is not time-variant. 2) what are these codevectors picked. 3) It seems that the codevectors are out of the regime of the distribution of y. But according to algorithm 2, y_*<-T(c_*), would that be a problem? I think (a)-(d) are informative but not straightforward to read. The authors need to put more text to explain these figures, since this simulated example can help readers to understand what is codevector and how it helps for uncertainty estimation. 

Overall I think the paper is well-written. The idea is novel and practical in the scenario of DNN. I would vote for accept. 


review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposed differentiable Bayesian neural networks (DBNN) to speed up inference in an online setting. The idea is to combine online vector quantization with Bayesian neural networks (BNN), so that only incremental computation is needed for a new data point.

It seems the idea of online codevector histogram, or online vector histogram, is not new, which is not surprising since the original codevector histogram work is from decades ago (1982). A quick search shows several related work in this domain, e.g., ‘An Online Incremental Learning Vector Quantization’. It would be better if the authors could clarify the differences if they want to claim the contribution.

One of my major concerns is the fairness in terms of comparison to BNN approaches. For example, in Table 2, BNN (shown as MU) is significantly slower than DU/DBNN and DNN. The authors mentioned that this is because MU predicts results for 10 batches of size 3, and therefore 30 times slower. Since DU and DBNN also uses MC-dropout, why is this not an issue for DU/DBNN. Such large overhead is also inconsistent with the description in Section 5.3, saying that the ‘overhead of sampling weights is negligible’. Could the authors elaborate on this?

A related problem for clarification: it is mentioned before Section 5.1 that 30 samples are drawn from the BNN’s posterior. Do you mean 30 feedforward passes of MC-dropout, as done in the MC-dropout paper? Also, the authors should have made it clear that they are using MC-dropout as a BNN.

In the introduction, the authors motivate the proposed DBNN by saying that BNN needs dozens of samples from weight distributions and therefore is rather inefficient. However, there are a lot of modern BNN that are both sampling-free and differentiable. For example, natural-parameter networks (NPN) parameterize both the weight and neuron distributions using natural parameters. Even the earlier work, probabilistic BP (PBP), as cited in the current paper, also counts as sampling-free. The claim of ‘being differentiable’ without acknowledging prior work is rather misleading.

The point above is also related to the MU baseline in Table 2. The issue of needing 30 passes can be readily resolved if modern BNN such as NPN (or PBP), which takes only one pass, is used.

The organization could be improved to make the paper more readable. It would be better if the problem setting of online inference is introduced at the beginning, followed by the overview of DBNN and then the OCH details. Otherwise, it is not clear what the focus of DBNN is, until Section 4.

Minor:

It might be better to denote the weight as ‘w’ rather than ‘x’, to avoid confusion.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Summary of the Paper:
  
This paper describes a method for training Bayesian neural networks in the context of stream data. The method proposed is based on using a quantization approach with some techniques to estimate the change in probability distributions. The proposed approach is compared on some tasks, including real and synthetic datasets.

Detailed comments:

The paper needs to improve the writing. For example, the sentence "it has been unable to estimate the uncertainty of the predictions until recently" sounds awkward.

The authors have to better explain the features and challenges of stream data.

The paper is unclear. There are several steps of the proposed method that are not well described.

How is the posterior distribution of the weights computed?

The notation x_0 and x_1 for the test point and the NN weights is confusing.

Figure 2 is not clear.

The description of the baselines the authors compare with is not clear.

What do you mean by degenerated in section 5.1.?

The paper is missing a related work section describing state of the art methods to address stream data.

It seems the real experiments of section 5.1. only consider one baseline MU. I believe this is insufficient.

In table 2 the benefits of the proposed approach are not very significant.

The experiments are missing error bars. It is not possible to extract conclusions of significance without them.

My overall impression is that the paper needs to better explain the approach followed and improve the notation to facilitate the reading. I believe that this paper needs for work and is not yet suitable for acceptance.

review_title: Official Blind Review #1

Meta-review: The main contribution is a Bayesian neural net algorithm which saves computation at test time using a vector quantization approximation. The reviewers are on the fence about the paper. I find the exposition somewhat hard to follow. In terms of evaluation, they demonstrate similar performance to various BNN architectures which require Monte Carlo sampling. But there have been lots of BNN algorithms that don't require sampling (e.g. PBP, Bayesian dark knowledge, MacKay's delta approximation), so it seems important to compare to these. I think there may be promising ideas here, but the paper needs a bit more work before it is to be published at a venue such as ICLR.

Decision: Reject