Differentiable Bayesian Neural Network Inference for Data Streams

1 INTRODUCTION . While deterministic neural networks ( DNNs ) surpass human capability in some area in terms of prediction accuracy ( He et al. , 2015 ; Silver et al. , 2016 ; Ardila et al. , 2019 ) , it could not estimate a trustworthy uncertainty of prediction . Since the prediction can not be perfect and the misprediction might result in fatal consequences in areas such as medical analysis and autonomous vehicles control , estimating uncertainty as well as predictions will be crucial for the safer application of machine learning based systems . Bayesian neural network ( BNN ) , a neural network ( NN ) that uses probability distributions as weights , estimates not only predictive results but also uncertainties . This allows computer systems to make better decisions by combining uncertainty with prediction . Moreover , BNN can achieve high performance in a variety of fields , e.g . image recognition ( Kendall et al. , 2015 ; Kendall & Gal , 2017 ) , language modeling ( Fortunato et al. , 2017 ) , reinforcement learning ( Kahn et al. , 2017 ; Osband et al. , 2018 ) , meta-learning ( Yoon et al. , 2018 ; Finn et al. , 2018 ) , and multi-task learning ( Kendall et al. , 2018 ) , by exploiting uncertainty . Although BNNs have these theoretical advantages , they have not been used as a practical tool . The predictive inference speed has detained BNNs from wide applications . BNN executes NN inference for dozens of samples from weight distributions . Since sampling and multiple NN executions are difficult to be parallelized , the inference execution takes an order of magnitude more time . Particularly , this is a significant barrier for processing data streams with low-latency . Most time-varying data streams change continuously , and so do the predictions of BNNs . Thus , we estimate prediction by calculating the increments between two consecutive results , instead of calculating the separate prediction for each input data . This is equivalent to calculating the differentiation of the BNN ’ s prediction for arbitrary data , because the difference of prediction for the new data is the line integration of the gradient of prediction over the new data . In this work , we propose a differentiable BNN ( DBNN ) inference with respect to an input data . The prediction of the DBNN is given by a Monte Carlo ( MC ) estimator for distributions of data streams and weights . We speed up the inference by approximating the distribution using histogram and calculating the gradient for this MC estimator . We show that the time complexity of this model is nearly the same as that of deep DNNs . We evaluate DBNN with semantic segmentation using road scene video sequences and the results show that DBNN has almost no degradation in computational performance compared to deep DNNs . The uncertainty predicted by DBNN is comparable to that of BNN in various situations . The main contributions of this work are as follows . • We propose online codevector histogram that estimates the probability of a high-dimensional data stream . Then , we show that this histogram can be used to obtain the MC gradient estimation . • We propose differentiable Bayesian neural network inference with respect to input data as an approximation of Bayesian neural network inference for data streams . DBNN uses the online codevector histogram to approximate probabilities . This model is nonparametric and applied to trained BNN without significant modifications . • We theoretically and empirically show that the computational performance of DBNN is almost the same as that of deep DNNs . 2 BACKGROUND AND RELATED WORK . BNNs are a state-of-the-art method to estimate predictive uncertainty while DNNs based approaches have been developed recently ( Lakshminarayanan et al. , 2017 ; Guo et al. , 2017 ) . BNNs with probability distributions as weights produce probabilistic results . To make prediction , BNNs sample from the weight probabilities and performs DNN for each sample . In this section , we describe the details and challenges of BNNs inference process . 2.1 BAYESIAN NEURAL NETWORK INFERENCE . Suppose that p ( w ) is posterior probability of NN weights w and p ( y|x , w ) is the BNN model for input data x . Then , the inference result of BNN is a predictive distribution : p ( y|x ? ) = ∫ p ( y|x ? , w ) p ( w ) dw ( 1 ) where x ? is observed input data vector and y is output vector . For simplicity , BNNs are usually modeled to have a probability distribution with the mean of the prediction of DNN , e.g . in HernándezLobato & Adams ( 2015 ) ; Gal & Ghahramani ( 2016 ) : p ( y|x ? , w ) = N ( y|NN ( x ? , w ) , τ−1 ) ( 2 ) where NN ( · ) is prediction of DNN and τ is a given parameter . ( 1 ) can be approximated using MC estimator : p ( y|x ? ) ' 1 Nw ? ∑ w ? p ( y|x ? , w ? ) ( 3 ) where w ? ∼ p ( w ) and Nw ? is the number of samples . The expected value of the obtained predictive distribution is the predictive result of BNN and the variance is the predictive uncertainty . The process of calculating the equation consists of two steps : sampling weights from p ( w ) , e.g . using Markov chain Monte Carlo ( MCMC ) ( Neal et al. , 2011 ; Hoffman & Gelman , 2014 ) , and executing the DNN for each weight sample . Since real-world data is large and practical NNs are deep , the repetitive computation of NNs is difficult to be fully parallelized due to various problems , such as memory limitations of the GPU ( Kendall & Gal , 2017 ) , and the iterative computation of DNNs results in the decrease of computation speed . MCMCs are slow and , despite recent achievements ( Tran et al. , 2018 ) , it is challenging to achieve linearly scaling performance for multi-GPU MCMC . In addition , it is difficult to check for the convergence of MCMCs . To mitigate these problems , we approximate the difference in the prediction for one new data as one NN calculation , and sample the weights from the posterior before testing . Several sampling-free BNNs , e.g . Hernández-Lobato & Adams ( 2015 ) ; Wang et al . ( 2016 ) ; Hwang et al . ( 2018 ) ; Wu et al . ( 2018 ) , have been proposed recently . The sampling-free BNNs predict results with only one forward pass . These methods , however , have limitations . First , sampling-free BNNs necessitate specific types of weight-distribution with parameters such as Gaussian distribution or exponential family instead of using MC estimator to approximate ( 1 ) . When a posterior has multimodality or skew , they can give inaccurate approximations . Second , in many cases , sampling-free BNN uses a method other than variational inference to obtain a posterior because evidence lower bound ( ELBO ) is not amenable . These constraints make it difficult to apply sampling-free BNNs to deep NNs . Therefore , we mainly consider sampling-based BNN for comparison in this paper . 2.2 BAYESIAN NEURAL NETWORK GRADIENT ESTIMATION . It is standard to use a gradient of MC estimator to optimize the loss function of BNN , e.g . ELBO . Several direct and indirect methods are used to obtain the MC gradient estimation : reparametrization trick ( Kingma & Welling , 2013 ; Blundell et al. , 2015 ; Fortunato et al. , 2017 ) , score function ( Kleijnen & Rubinstein , 1996 ; Williams , 1992 ; Glynn , 1990 ) , dropout ( Gal & Ghahramani , 2016 ; McClure & Kriegeskorte , 2016 ) , batch normalization ( Teye et al. , 2018 ) , expectation propagation ( HernándezLobato & Adams , 2015 ) , and smooth transformation for samples ( Liu & Wang , 2016 ; Burda et al. , 2015 ; Maddison et al. , 2017 ; Naesseth et al. , 2017 ; Le et al. , 2017 ; van den Oord et al. , 2017 ) . However , it is not appropriate to use these techniques to calculate a gradient of MC estimator L = Ep ( x ) [ f ( x ) ] for arbitrary functions f ( · ) when a data stream is non-stationary and p ( x ) , the distribution of the data stream , is time-variant . For example , the reparameterization trick , one of the most widely used methods for MC gradient estimation , separates the probability p ( x ) into a deterministic function h ( · ) and an invariant probability p ( ) , i.e. , x = h ( ) and ∂xp ( ) = 0 . Then , the L is expectation over the invariant distribution , i.e. , L = Ep ( ) [ f ( x ) ] , and the approximated L becomes differentiable , i.e. , ∂xL = Ep ( ) [ ∂xf ( x ) ] . To use this technique , we need to adapt h ( · ) to the non-stationary data stream . If h ( · ) is modeled in an NN , we may need to continuously train this NN on the data stream ; this leads to a decrease in computational performance . To alleviate this problem , we use histogram to estimate probability with nearest neighbor search , instead of using an NN-based generative model . In addition , the histogram is a linear sum of disjoint bins that is analytically useful while simplifying calculations . 3 ONLINE CODEVECTOR HISTOGRAM . Vector quantization was introduced in Gersho ( 1982 ) ; Gray ( 1984 ) in order to compress a probability distribution to a dozens of samples called codevectors . Kotani et al . ( 2002 ) showed that the histogram of codevectors—codevectors augmented with counts—can effectively represent the features of face image dataset . Xu et al . ( 2012 ) ; Frezza-Buet ( 2014 ) ; Ghesmoune et al . ( 2016 ) have proposed algorithms that changes codevectors depending on the data stream . We introduce online codevector histogram ( OCH ) to estimate the probability distribution of non-stationary data stream as well as dataset . OCH is a probabilistic algorithm that changes not only codevectors but their counts and calculates the difference in distribution as the data stream changes , at low computational cost . OCH can add a new codevector and delete old ones , while counting the matches of each codevector for the past data stream . OCH maps the input vector to the codevector using nearest neighbor search . It is a high-dimensional histogram where the Voronoi diagram is the boundary and a codevector represents the corresponding bin . DBNN uses OCH to approximate the probability distributions of input and output vector data streams . Algorithm 1 shows how OCH operates in three steps . ( a ) Given a new input vector , OCH finds the nearest codevector and increases its count . Then , it divides the corresponding bin by inserting the input vector as a new codevector with probability proportional to the count . ( b ) OCH decrease all counts by the same rate to reduce the contribution of old data . ( c ) To keep the number of codevectors small , OCH deletes codevectors with probability in inverse proportion to its counts . OCH takes three hyperparameters : K , λ , and φ. K is the number of codevectors kept in OCH on average . λ determines how fast OCH depreciates old counts with the rate of γ. φ with counts of each bin regulates the probability of adding and deleting codevectors . Nearest neighbor search is the most computationally intensive step in alg . 1 . To lower the computational complexity , we use locality-sensitive hashing with stable distribution , h = b ( a · x ? + b ) /wc for a vector a and scalars b and w. It requires up to logK hashes for precise search . The upper bound of the computational complexity is O ( dim ( x ) logK ) . This is faster than continuous training of NN-based generative models for the data stream changing over time . Algorithm 1 : Update OCH input : input vector x ? , count of input vector n ? , OCH = { ( ci , ni ) } where ci is codevector and ni is its count , hyperparameters K , λ , and φ output : updated OCH 1 ci ← Search the nearest codevector to x ? in OCH 2 ni ← ni + n ? where ni is count of ci 3 p ∼ Bernoulli ( σ ( πi − π̄ + φ ) ) where σ ( · ) is sigmoid , N = ∑ i ni , πi = ni/N , and π̄ = 1/K 4 if p = 1 then 5 OCH← OCH ∪ { ( x ? , ( 1− γ ) · ni ) } where γ = e−λ/N 6 ni ← γ · ni 7 forall ( cj , nj ) ∈ OCH do 8 nj ← γ · nj 9 forall ( ck , nk ) ∈ OCH do 10 q ∼ Bernoulli ( σ ( π̄ − πk + φ ) · π̄ ) 11 if q = 1 then 12 OCH← OCH \ { ( ck , nk ) } Meanwhile , OCH approximates probability distribution p ( x ) as : p ( x ) ' ∑ i πiV ( x|ci ) ( 4 ) where N = ∑ i ni , πi = ni/N , and V ( x|c ) is the bin or neighborhood of c with ∫ V ( x|c ) dx = 1 . The right-hand-side of ( 4 ) is clearly differentiable with respect to πj , while a set of random samples from p ( x ) is not differentiable . Given a new input data , OCH changes only one count of the nearest codevector . Therefore , if n ? N , the difference of OCH for a new input vector is approximately : δp ( x ) ' ∑ i δπiV ( x|ci ) ( 5 ) ' αV ( x|c ? ) ( 6 ) where c ? is the nearest codevector to the input vector , α = δn/N , and δn is the difference of the count of c ? . When OCH creates a new codevector , δn is n ? . Here , we used the fact that codevectors are invariant for input data , so V ( x|ci ) is also invariant . This property is useful when the linear operator g ( · ) is applied to p ( x ) . By definition of linear operator , g ( p ( x ) ) = ∑ i πig ( V ( x|ci ) ) , then its difference is δg ( p ( x ) ) = ∑ i δπig ( V ( x|ci ) ) = αg ( V ( x|c ? ) ) .

Summary: The paper proposes a differentiable Bayesian neural network. Traditional BNN can model the model uncertainty and data uncertainty via adding a prior to the weight and assuming a Gaussian likelihood for the output y. However, it's slow in practice since evaluating the loss function of BNN involves multiple runs over the entire network. Also when the input data is non-stationary, the output function can not be differentiated with respect to the input data. The paper proposes to use an online code vector histogram (OCH) method attached to the input and output of a classical DNN. Input OCH captures the distribution of both input data and network weights, and the output OCH captures the distribution of the predictions. Since these OCHs are differentiable, the proposed DBNN model can be used for streaming input data with time-variant distributions.