Mohammad Emtiyaz Khan

Goals and Challenges
Goal: To design AI that can continually learn using Bayesian principles.

Scene

Ground truth(a) Input Image(b) Ground Truth

Examples: Uncertainty: Knowing how much we don’t know, is useful to design• Robots that can understand and reason about their environments.
• Methods that improve performance of deep-learning methods.
Challenge: Computation of the posterior distribution is difficult

Taken from Kendall et al. 2017m

Approximate Bayesian Inference Team(c) Semant
Segmentati

Figure 1: Illustrating the difference between aleatorion the CamVid dataset [8]. Aleatoric uncertainty captu
Main Idea: Approximate integration by using optimization, and design simple
exhibits increased aleatoric uncertainty on object bou
uncertainty accounts for our ignorance about whichalgorithms that can be implemented within existing
deep
learning
frameworks
Segmentation (e)Uncertainty (e) Epistem(a)Ground
Input Image(b) Ground Truth different(c) Semantic(d)Aleatoric(a) Input Image
(b)
Truth
(c) Semantic(d)Aleatoric
Epistemic
measure of uncertainty and in (e) our model e
Segmentation
Uncertainty
Segmentation
Uncertainty
Uncertainty
and visually challenging pixels. The bottom row sho
1: Illustrating
thealeatoricdifference
betweenaleatoric
epistemic
uncertainty
semanticep
Figure 1: IllustratingFigure
the difference
betweenandepistemic
uncertainty
for thesemantic
segmentation
model
fails
toand
segment
footpath
due tofor
increased
theAleatoric
CamVid dataset[8]. Aleatoric
uncertainty
capturesnoise inherentInin(d)theourobservations.
In (on the CamVid dataseton[8].
uncertainty
captures noise
inherent
in the observations.
model
exhibitsuncertainty
increased onaleatoricboundaries
and thefor camera.
objects far
from the camer
exhibits increased aleatoric
object uncertainty
boundaries on
andobjectfor objects
far from
Epistemic
accounts
for whichour ignorance
about which
generated
our collected
data. This
uncertainty accounts uncertainty
for our ignorance
about
model generated
our modelcollected
data. Thisis a notably
which
capturesourignorance
about
which
mod
different
measureofuncertainty
and
in
(e)
our
model
exhibits
increased
epistemic
uncertainty
fordifferent measure of uncertainty and in (e) our model exhibits increased epistemic uncertainty for semanticallycanbeexplained
away
given
enough
data,
and
is
and
visually
challenging
pixels.
Thebottomrow
showsafailurecaseofthesegmentationmoand visually challenging pixels. The bottom row shows a failure case of the segmentation model when theuncertainty
beuncertainty.
categorized
into homos
fails to due
segment
the footpath
due touncertainty,
increased
epistemic
uncertainty,
but not aleatoric
uncer
model fails to segmentmodel
the footpath
to increased
epistemic
butcannotfurther
aleatoric

Fast and Simple Algorithms for Variational Inference

Variational Inference

Parameters
Intractable
integral

Data
Variational Approximation

Naturalparameters

Maximize the Evidence Lower Bound (ELBO):

Gradient descent (GD) :Example: Linear Regression
likelihood>(yX✓) (yprior>

approxX✓) + ✓ ✓<latexit sha1_base64="8pEf7sZY+C83gDA6D/6vh96XwME=">AAACIXicbVDLSgMxFM34rPVVdekmWISKWGbc2JUU3LisYB/QqeVOmrahycyQ3BFK6a+48VfcuFCkO/FnTDtd1NYDgcM553JzTxBLYdB1v5219Y3Nre3MTnZ3b//gMHd0XDNRohmvskhGuhGA4VKEvIoCJW/EmoMKJK8Hg7upX3/m2ogofMRhzFsKeqHoCgZopXauVBheNXzsc4SLJx+jmC4I9JL6PVAKaCqkgZS3c3m36M5AV4k3J3kyR6Wdm/idiCWKh8gkGNP03BhbI9AomOTjrJ8YHgMbQI83LQ1BcdMazS4c03OrdGg30vaFSGfq4sQIlDFDFdikAuybZW8q/uc1E+yWWiMRxgnykKWLuomkGNFpXbQjNGcoh5YA08L+lbI+aGBoS83aErzlk1dJ7brouUXvwc2Xb+d1ZMgpOSMF4pEbUib3pEKqhJEX8kY+yKfz6rw7X84kja4585kT8gfOzy+AvKJ9</latexit>
sha1_base64="rX8IyNjEbJi9KltELKA+f3dm0QQ=">AAACIXicbVDLSgNBEOz1bXxFPQoyKIIihl0vepKAF48KxgSyMfROJsngzO4y0yuE4IfkIoK/4sWDIt7En3GS9RAfBQNFVTU9XVGqpCXf//AmJqemZ2bn5gsLi0vLK8XVtSubZIaLCk9UYmoRWqFkLCokSYlaagTqSIlqdHM69Ku3wliZxJfUS0VDYyeWbcmRnNQsHu/2DmohdQXh3nVIScrGBLbPwg5qjSwX8kDOm8Vtv+SPwP6S4JtslzcHgwcAOG8W38NWwjMtYuIKra0HfkqNPhqSXIm7QphZkSK/wY6oOxqjFrbRH114x3ac0mLtxLgXExup4xN91Nb2dOSSGqlrf3tD8T+vnlH7uNGXcZqRiHm+qJ0pRgkb1sVa0ghOqucIciPdXxnvokFOrtSCKyH4ffJfcnVYCvxScOHaOIEcc7ABW7ALARxBGc7gHCrA4R6e4AVevUfv2Xvz3vPohPc9sw4/4H1+AXSOpK0=</latexit>
sha1_base64="NZi+y9DK5eA4/AsYeLzD4AbBZ9M=">AAACIXicbVDLSgMxFM3Ud31VXQoSLEJFLDNu7EoENy4V7AM6tdxJ0zY0mRmSO0Ip/RBduPFX3LhQxJ34M6adLmrrgcDhnHO5uSeIpTDout9OZmFxaXlldS27vrG5tZ3b2a2YKNGMl1kkI10LwHApQl5GgZLXYs1BBZJXg97VyK8+cG1EFN5hP+YNBZ1QtAUDtFIzVyr0T2s+djnC8b2PUUynBHpC/Q4oBTQV0kDKm7m8W3THoPPEm5D85cHjCE83zdyX34pYoniITIIxdc+NsTEAjYJJPsz6ieExsB50eN3SEBQ3jcH4wiE9skqLtiNtX4h0rE5PDEAZ01eBTSrArpn1RuJ/Xj3BdqkxEGGcIA9ZuqidSIoRHdVFW0JzhrJvCTAt7F8p64IGhrbUrC3Bmz15nlTOip5b9G5tGxckxSrZJ4ekQDxyTi7JNbkhZcLIM3kl7+TDeXHenE/nK41mnMnMHvkD5+cX0cumcg==</latexit>

2

Empirical Evaluation(a) MNIST

2(b) CIFAR10

Empirical Evaluation(a) MNIST

stant
for different inputs,
and heteroscedastic un
Expectation
Parameters
the inputs to the model, with some inputs potent
Sato 2001, Honkela et al. 2010, Hoffman et.al. 2013
Expectation/moment/
eroscedastic
uncertainty
isespecially
importantfmeanparameterswhich
capturesourignorance
about
which
model
generated
ourcollected
data.
This
which captures our ignorance Natural
aboutGradient
which model generated our collected data. This Sufficient
uncertainty
statistics
NGD:
depth
regression,
highlytextured
input
imagesw
canbeexplained
away
given
enough
data,
and
is
often
referred
to
as
model
uncertainty
can be explained away given enough data, and is often referred to as model uncertainty. Aleatoricconfident
predictions,
an
input
image
o
For
Gaussians,
it’smeanuncertainty,
andwhereas
correlation
matrix
uncertainty
canfurtherbecategorized
intohomoscedastic
uncertainty
whic
uncertainty
canfurtherbecategorized
intohomoscedastic
uncertainty,
uncertainty
which
stays
conFisher Information Matrix (FIM)
uncertainty.
stant
for
different
inputs,andheteroscedastic
uncertainty.
Heteroscedastic
stant for different inputs, and heteroscedastic uncertainty. Heteroscedastic
uncertainty
dependsuncertainty
on
the inputs
to the inputs
model,potentiallywith somehaving
inputspotentiallynoisyothe inputs to the model,
with some
more
noisy
outputs
thanobservation
others.outputs
Het-thatthan
In thispaper
wehaving
make morethein meroscedastic
uncertainty
is especially
importantforcomputervisionapplications.
Foreffe
eroscedastic uncertainty
is especially
importantfor computer
visionapplications.
For
example,
for
to
deep
learning
withimagedata),
itismost
A key relationship:
depth
regression,
textured
inputwhich
imageswithstrongvanishinglines
areexpected• depthFast convergenceto optimizationin highly
regression, duehighlytextured
input
imageswith strong
vanishinglines
areexpected
toresultincannotbeexplained
away.
This
is
Natural Gradient wrt
Gradient wrt expectation in com
predictions,whereasinput
image
ofawaya expectedfeaturelesswall
isexpected
to hav
Riemannian
manifoldconfident(not
Euclideannaturalparameter
confident
predictions,whereasanspace).
input imageof aanfeaturelesswallishaveparameter
very
high
explainedwith thetolarge
amountsof datao• uncertainty.
But requires additional
computations.
uncertainty.
that modeling aleatoric uncertainty alone comes• Can we simplify/reduce this computation?
NGD:
identified
withepistemic
uncertainty,
cannot
be
id
In
this
paper
we
make
the
observation
that
in
many
big
data
regimes
(such
as
the
on
In this paper we make the observation that in many big data regimes (such as the ones common
to deep
imagedata), ittois
most
to
model
aleatoric
uncertainty,
to deep learning withimagelearningdata), itwith
is mosteffectivemodel
aleatoric
uncertainty,
uncertaintyForthis effective
we present
a unifiedBayesiandeep lea
which cannot
explained
away. Thispingsin fromcomparison
to epistemic
uncertainty
whic
which cannot be explained
away. beThis
is in comparison
toisepistemic
uncertainty
ismostlyinput
data
to whichaleatoric
uncertaintyaexplained
withofthedatalarge
amounts
of
data
often
availableinmachinevision.
Wefexplained away withthe largeaway
amounts
often
availableinmachinevision.
Wefurthershow
tainty approximations. We derive our framework
MLE
vs
NGD-VI
Bayesian
NeuralNetworkthatmodeling
aleatoric
uncertainty
alone
comes
at
a
cost.
Out-of-data
examples,whthat modeling aleatoric uncertainty alone comes at a cost.
examples,
which
can
be
and Out-of-data
present resultsfor per-pixel
depth
regression
identified
withepistemic
uncertainty,
cannotbeidentified
withaleatoric
uncertainty
alon
identified likelihood
with epistemic
uncertainty,
cannotbeidentified
withaleatoric
uncertainty
alone.
thesupplementary
video
for
examples).
We
show
prior
approxRMSprop for MLE
NGD for mean-field VI
can
be
used
to
learn
loss
attenuation,anddeveloFor
thiswe
presentaunifiedBayesian
deep
learning
frameworkwhich
allowsFor this we present a unified Bayesian deep learning framework which allows us to learn map- us to
case.
This
demonstrates
theefficacy
of
our
appro
pings
from
input
data
to
aleatoric
uncertainty
andcomposethese
togetherwithneuralnetworkpings from input data to aleatoric uncertainty and compose these together with epistemic uncer- episte
taintyWe derive
our frameworkforbothregressionandclassificationatainty approximations.
Weapproximations.
derive our framework
for bothregressionandclassification
applications
The main contributions of this work are;andper-pixel
present resultsfor per-pixel
regression
and semantic
tasks (see F
and present results for
depth regression
anddepth
semantic
segmentation
tasks segmentation(see Figure 1 andWehow
capturean accurate
understanding
the supplementaryfor examples).
We 1.
show
modeling
aleatoric
uncertaintyof
inthe supplementary video
for examples).video
We show
how modelingaleatoric
uncertainty
in regression
Back-propagated
with
a
novel
approachforclassification,
can
be
used
to
learn
loss
attenuation,anddevelopacomplimentary
approachfortheccan be used to learn loss attenuation,anddevelopacomplimentary
approachfortheclassification
gradient & Hessiancase. theThisdemonstrates
the efficacyof
our approach
onscaledifficultand large scale tasks.
Variational
OnlineGauss-Newton(VOGN)
case. This demonstrates
efficacyof our approachondifficultand large
tasks.
2. We improve model performance by 1
The ofmain
effect of noisy data with the implied
The main contributions
thiscontributions
work are; of this work are;Variational RMSprop(Vprop)aleatoric
uncertainty,
1.
We
captureanaccurate
understandingofaleatoricand epistemic
uncertainties, i
1. We capture an accurate understanding of aleatoric and epistemic uncertainties,
in particularwithfora novel
approach for classification,with a novel approachclassification,
3. We study the trade-offs between modeli
izing
the
propertiesof eachuncertainty
2.
We
improvemodel
performance
by
1
3%
over
non-Bayesianbaselines
by r
2.
We
improvemodel
performance
by
1
3%
over
non-Bayesianbaselinesbyreducingthe
Stochastic, Low-Rank, effectApproximate,
time.
of
noisy
data
with
the
impliedattenuation
obtained
from
SLANG
is
Fasterthan
GD explicitly reffect
of
noisy
data
with
the
impliedattenuation
obtainedfromexplicitly
representing
Natural-Gradient (SLANG)
aleatoric
uncertainty,
aleatoric uncertainty,
Classification on USPS with BNNs
NeurIPS 2018

VI with Natural-Gradient Descent(b) CIFAR10

LeNet-5 on CIFAR10(b) CIFAR10• Low-rank + diagonal covariancematrix.
3.
Westudy
the modelingtrade-offs aleatoric
between modelingaleatoric
or epistemic
uncertainty b
3.
We
study
the
trade-offs
betweenor
epistemic
uncertainty
bycharacter• SLANG is linear in D!
izing the properties of each uncertainty and comparing model performance an

Log Loss
Error

Test
VOGN Adam
Log Loss 0.065
0.108
Error
2.109
1.079
Test
Train
VOGN VOGN
Adam
Adam
Log
Loss
0.058
0.065
0.1080.001
Error
1.718
0.026

Test
VOGN Adam
Log Loss 1.130
8.341
Error
37.01
Test40.47
Train
VOGN
VOGN
Adam
Log Loss
0.077
Log
Loss 0.8151.130
Error
27.18
2.248

Adam
8.341
40.47

D⇥L

2.109
1.079
Error
37.01
Figure 2: Evaluation
metrics on Train and Test sets for both optimizers.Train
Adam overfits while VOGN
Traindoes a good job of keeping test and train errors close. VOGN outperforms Adam on CIFAR10 but
Adam
Adam
underperforms onVOGN
MNIST for test
accuracy. For test log loss, VOGN isVOGN
better than Adamin bothcases.
in Table 1.
Log Model
Lossarchitectures
0.058 given0.001
Log Loss
0.815
0.077
Error
1.718
0.026
Error
27.18
2.248

L⇥D

+

Anirudh
Jain)
Figure 2: Evaluation metrics on Train and Test sets for both(Byoptimizers.
Adam
overfits while VOGN
does a good job of keeping test and train errors close. VOGN outperforms Adam on CIFAR10 but
underperforms on MNIST for test accuracy. For test log loss, VOGN is better than Adam in bothcases. Model architectures given in Table 1.

(1

D⇥M

M ⇥D

gradient

izing the propertiesof each uncertainty and comparing model performance and inference
Low-Rank + diagonal
time.
time.

gradientD⇥L
fast_eig=)Ut U>
t

L⇥D

2

2

⇡

Ut+1 U>t+1

G(✓ t )

Figure 2: This figure illustrates Equations (6) and (7) which are used to derive SLANG.

References
The new covariance approximation can now be used to update µaccording to (2) as shown below:hi
1. Fast and Scalable Bayesian Deep Learning by Weight-Perturbation
inµ Adam,2018),
Nielsen, Tangkaratt, Lin, Gal, and Srivastava.
SLANG: µ=↵ U U (ICML+D[ĝ(✓ ) + µ Khan,
],
(11)
above
update uses a stochastic,
low-rank
covariance estimate
to approximate
natural-gradient
2. SLANG: Fast Structured Covariance Approximations The
for
Bayesian
Deep
Learning
with
NaturalGradient, (NeurIPS 2018), Mishkin, Kunstner, Nielsen,
updates, which is why we use the name SLANG.
Schmidt, Khan.
When L = D, U U
is full rank and SLANG is identical to the approximate natural-gradientˆ
update (2). When
L < D, SLANG in
produces
matrices ⌃withdiagonals matching(2) at2018),
every
3. Fast and Simple Natural-Gradient Descent for Variational
Inference
Complex
Models(ISITA
Khan and Nielsen.
iteration. The diagonal correction ensures that no diagonal information is lost during the low-rank
t+1

3

t+1

t+1

t

t

t+1

1

>
t+1

t+1

t

>
t+1

t

Test
VOGN

t

1

approximation of the covariance. A formal statement and proof is given in Appendix D.

Adam

3

Figure 4: This figure compares the convergence behavior on two datasets: USPS
Breast Cancer (bottom); and two models: Bayesian logistic regression (left) and B

We also tried an alternative method where Ut+1 is learned using an exponential moving-average ofthe eigendecompositions of Ĝ(✓). This previous iteration of SLANG is discussed in Appendix B,