Bayesian Prediction of Future Street Scenes using Synthetic Likelihoods

1 INTRODUCTION The ability to anticipate future scene states which involves mapping one scene state to likely future states under uncertainty is key for autonomous agents to successfully operate in the real world e.g. , to anticipate the movements of pedestrians and vehicles for autonomous vehicles . The future states of street scenes are inherently uncertain and the distribution of outcomes is often multi-modal . This is especially true for important classes like pedestrians . Recent works on anticipating street scenes ( Luc et al. , 2017 ; Jin et al. , 2017 ; Seyed et al. , 2018 ) do not systematically consider uncertainty . Bayesian inference provides a theoretically well founded approach to capture both model and observation uncertainty but with considerable computational overhead . A recently proposed approach ( Gal & Ghahramani , 2016b ; Kendall & Gal , 2017 ) uses dropout to represent the posterior distribution of models and capture model uncertainty . This approach has enabled Bayesian inference with deep neural networks without additional computational overhead . Moreover , it allows the use of any existing deep neural network architecture with minor changes . However , when the underlying data distribution is multimodal and the model set under consideration do not have explicit latent state/variables ( as most popular deep deep neural network architectures ) , the approach of Gal & Ghahramani ( 2016b ) ; Kendall & Gal ( 2017 ) is unable to recover the true model uncertainty ( see Figure 1 and Osband ( 2016 ) ) . This is because this approach is known to conflate risk and uncertainty ( Osband , 2016 ) . This limits the accuracy of the models over a plain deterministic ( non-Bayesian ) approach . The main cause is the data log-likelihood maximization step during optimization – for every data point the average likelihood assigned by all models is maximized . This forces every model to explain every data point well , pushing every model in the distribution to the mean . We address this problem through an objective leveraging synthetic likelihoods ( Wood , 2010 ; Rosca et al. , 2017 ) which relaxes the constraint on every model to explain every data point , thus encouraging diversity in the learned models to deal with multi-modality . In this work : 1 . We develop the first Bayesian approach to anticipate the multi-modal future of street scenes and demonstrate state-of-the-art accuracy on the diverse Cityscapes dataset without compromising on calibrated probabilities , 2 . We propose a novel optimization scheme for dropout based Bayesian inference using synthetic likelihoods to encourage diversity and accurately capture model uncertainty , 3 . Finally , we show that our approach is not limited to street scenes and generalizes across diverse tasks such as digit generation and precipitation forecasting . 2 RELATED WORK . Bayesian deep learning . Most popular deep learning models do not model uncertainty , only a mean model is learned . Bayesian methods ( MacKay , 1992 ; Neal , 2012 ) on the other hand learn the posterior distribution of likely models . However , inference of the model posterior is computationally expensive . In ( Gal & Ghahramani , 2016b ) this problem is tackled using variational inference with an approximate Bernoulli distribution on the weights and the equivalence to dropout training is shown . This method is further extended to convolutional neural networks in ( Gal & Ghahramani , 2016a ) . In ( Kendall & Gal , 2017 ) this method is extended to tackle both model and observation uncertainty through heteroscedastic regression . The proposed method achieves state of the art results on segmentation estimation and depth regression tasks . This framework is used in Bhattacharyya et al . ( 2018a ) to estimate future pedestrian trajectories . In contrast , Saatci & Wilson ( 2017 ) propose a ( unconditional ) Bayesian GAN framework for image generation using Hamiltonian Monte-Carlo based optimization with limited success . Moreover , conditional variants of GANs ( Mirza & Osindero , 2014 ) are known to be especially prone to mode collapse . Therefore , we choose a dropout based Bayesian scheme and improve upon it through the use of synthetic likelihoods to tackle the issues with model uncertainty mentioned in the introduction . Structured output prediction . Stochastic feedforward neural networks ( SFNN ) and conditional variational autoencoders ( CVAE ) have also shown success in modeling multimodal conditional distributions . SFNNs are difficult to optimize on large datasets ( Tang & Salakhutdinov , 2013 ) due to the binary stochastic variables . Although there has been significant effort in improving training efficiency ( Rezende et al. , 2014 ; Gu et al. , 2016 ) , success has been partial . In contrast , CVAEs ( Sohn et al. , 2015 ) assume Gaussian stochastic variables , which are easier to optimize on large datasets using the re-parameterization trick . CVAEs have been successfully applied on a large variety of tasks , include conditional image generation ( Bao et al. , 2017 ) , next frame synthesis ( Xue et al. , 2016 ) , video generation ( Babaeizadeh et al. , 2018 ; Denton & Fergus , 2018 ) , trajectory prediction ( Lee et al. , 2017 ) among others . The basic CVAE framework is improved upon in ( Bhattacharyya et al. , 2018b ) through the use of a multiple-sample objective . However , in comparison to Bayesian methods , careful architecture selection is required and experimental evidence of uncertainty calibration is missing . Calibrated uncertainties are important for autonomous/assisted driving , as users need to be able to express trust in the predictions for effective decision making . Therefore , we also adopt a Bayesian approach over SFNN or CVAE approaches . Anticipation future scene scenes . In ( Luc et al. , 2017 ) the first method for predicting future scene segmentations has been proposed . Their model is fully convolutional with prediction at multiple scales and is trained auto-regressively . Jin et al . ( 2017 ) improves upon this through the joint prediction of future scene segmentation and optical flow . Similar to Luc et al . ( 2017 ) a fully convolutional model is proposed , but the proposed model is based on the Resnet-101 ( He et al. , 2016 ) and has a single prediction scale . More recently , Luc et al . ( 2018 ) has extended the model of Luc et al . ( 2017 ) to the related task of future instance segmentation prediction . These methods achieve promising results and establish the competence of fully convolutional models . In ( Seyed et al. , 2018 ) a Convolutional LSTM based model is proposed , further improving short-term results over Jin et al . ( 2017 ) . However , fully convolutional architectures have performed well at a variety of related tasks , including segmentation estimation ( Yu & Koltun , 2016 ; Zhao et al. , 2017 ) , RGB frame prediction ( Mathieu et al. , 2016 ; Babaeizadeh et al. , 2018 ) among others . Therefore , we adopt a standard ResNet based fully-convolutional architecture , while providing a full Bayesian treatment . 3 BAYESIAN MODELS FOR PREDICTION UNDER UNCERTAINTY . We phrase our models in a Bayesian framework , to jointly capture model ( epistemic ) and observation ( aleatoric ) uncertainty ( Kendall & Gal , 2017 ) . We begin with model uncertainty . 3.1 MODEL UNCERTAINTY . Let x ∈ X be the input ( past ) and y ∈ Y be the corresponding outcomes . Consider f : x 7→ y , we capture model uncertainty by learning the distribution p ( f |X , Y ) of generative models f , likely to have generated our data { X , Y } . The complete predictive distribution of outcomes y is obtained by marginalizing over the posterior distribution , p ( y|x , X , Y ) = ∫ p ( y|x , f ) p ( f |X , Y ) df . ( 1 ) However , the integral in ( 1 ) is intractable . But , we can approximate it in two steps ( Gal & Ghahramani , 2016b ) . First , we assume that our models can be described by a finite set of variables ω . Thus , we constrain the set of possible models to ones that can be described with ω . Now , ( 1 ) is equivalently , p ( y|x , X , Y ) = ∫ p ( y|x , ω ) p ( ω|X , Y ) dω . ( 2 ) Second , we assume an approximating variational distribution q ( ω ) of models which allows for efficient sampling . This results in the approximate distribution , p ( y|x , X , Y ) ≈ p ( y|x ) = ∫ p ( y|x , ω ) q ( ω ) dω . ( 3 ) For convolutional models , Gal & Ghahramani ( 2016a ) proposed a Bernoulli variational distribution defined over each convolutional patch . The number of possible models is exponential in the number of patches . This number could be very large , making it difficult optimize over this very large set of models . In contrast , in our approach ( 4 ) , the number possible models is exponential in the number of weight parameters , a much smaller number . In detail , we choose the set of convolutional kernels and the biases { ( W1 , b1 ) , . . . , ( WL , bL ) } ∈ W of our model as the set of variables ω . Then , we define the following novel approximating Bernoulli variational distribution q ( ω ) independently over each element wi , jk′ , k ( correspondingly bk ) of the kernels and the biases at spatial locations { i , j } , q ( WK ) =MK ZK zi , jk′ , k = Bernoulli ( pK ) , k ′ = 1 , . . . , |K ′| , k = 1 , . . . , |K| . ( 4 ) Note , denotes the hadamard product , Mk are tuneable variational parameters , zi , jk′ , k ∈ ZK are the independent Bernoulli variables , pK is a probability tensor equal to the size of the ( bias ) layer , |K| ( |K ′| ) is the number of kernels in the current ( previous ) layer . Here , pK is chosen manually . Moreover , in contrast to Gal & Ghahramani ( 2016a ) , the same ( sampled ) kernel is applied at each spatial location leading to the detection of the same features at varying spatial locations . Next , we describe how we capture observation uncertainty . 3.2 OBSERVATION UNCERTAINTY . Observation uncertainty can be captured by assuming an appropriate distribution of observation noise and predicting the sufficient statistics of the distribution ( Kendall & Gal , 2017 ) . Here , we assume a Gaussian distribution with diagonal covariance matrix at each pixel and predict the mean vector µi , j and co-variance matrix σi , j of the distribution . In detail , the predictive distribution of a generative model draw from ω̂ ∼ q ( ω ) at a pixel position { i , j } is , pi , j ( y|x , ω̂ ) = N ( ( µi , j |x , ω̂ ) , ( σi , j |x , ω̂ ) ) . ( 5 ) We can sample from the predictive distribution p ( y|x ) ( 3 ) by first sampling the weight matrices ω from ( 4 ) and then sampling from the Gaussian distribution in ( 5 ) . We perform the last step by the linear transformation of a zero mean unit diagonal variance Gaussian , ensuring differentiability , ŷi , j ∼ µi , j ( x|ω̂ ) + z × σi , j ( x|ω̂ ) , where p ( z ) is N ( 0 , I ) and ω̂ ∼ q ( ω ) . ( 6 ) where , ŷi , j is the sample drawn at a pixel position { i , j } through the liner transformation of z ( a vector ) with the predicted mean µi , j and variance σi , j . In case of street scenes , yi , j is a class-confidence vector and sample of final class probabilities is obtained by pushing ŷi , j through a softmax .

Summary: The submission considers a disadvantage of a standard dropout-based Bayesian inference approach, namely the pessimization of model uncertainty by means of maximizing the average likelihood for every data sample. The formulation by Gal & Ghahramani is improved upon two-fold: via simplified modeling of the approximating variational distribution (on kernel/bias instead of on patch level), and by using a discriminator (i.e. classifier) for providing a "synthetic" likelihood estimate. The latter relaxes the assumptions such that not every data sample needs to be explained equally well by the models.