Approximate Bayesian Inference Team
Mohammad Emtiyaz Khan

Goals and Challenges
Scene

Uncertainty estimate

Goal: To design AI that can continually learn using Bayesian uncertainty.
Examples: Uncertainty, i.e., knowing how much we don’t know, is useful in designing:• Robots that can understand their environments (e.g., for elderly care)• AI to design high-performance buildings• Methods that improve performance of deep-learning methods
Challenge: Computation of Bayesian uncertainty is difficult
Main Idea: Approximate integration by optimizationp(D|✓)p(✓)
p(✓|D) = Rp(D|✓)p(✓)d✓Intractable integral

For depth estimation, the algorithm is uncertain
about the edges in the scene (see right image).
This knowledge can improve safety for selfdriving cars (taken from Kendall et. al. 2017)

Simplifying Approximate Bayesian Inference for Complex Models(a) Input Image(c) Semantic(a) Input Image
Segmentation(d) (b)Aleatoric
Ground Truth
Uncertainty(e) SemEp(c)
VadUncer
Segme

Figure 1: Illustrating the difference between
aleatoricand epistemic
uncertaintyfor semaFigure1: Illustrating
the difference
betweenalea330
onIntegral
the CamVid
dataset [8]. AleatoriccapturesQuality
noise inherent
in theobservations.
the 331
CamViddatasetAleatoric
uncertainty cBayesianis difficultof[8].
Uncertainty
Estimates
Avoidinguncertainty
LocalonMinimaexhibits
increasedaleatoric uncertainty onexhibits
object332
boundaries
andfor
objects
farfromtheca
Priorp(theta)
increased aleatoric
uncertaintyonobjectp(D|theta)
Uncertainty Approximations
10
uncertainty accounts for our ignorance about
which model
generatedourcollecteddata.
uncertainty
accounts
forourignorance
about
whi
333
4
5x
different
measure of uncertainty and in (e) different
our modelmeasureexhibitsofincreased
epistemic
uncertainty
uncertainty
and in (e)our mod334
3
and visually challenging pixels. The bottom
showsa failure caseof theand row
visuallychallenging
pixels.
Thesegmentation
bottomrow
335
model fails to segment the footpath due to increased
uncertainty,
butduenot toaleatoricumodel failsepistemic
to segment
the footpath
2 increase
MAP-32For non-conjugate models, we propose
Conjugate-Computation Variational
Inference (CVI) to approximate theBayesian integration• CVI is a general method that is alsoeasy to implement for many models• It expresses inference in a complex
model as an inference in a simpler
model, e.g., logistic Regression
expressed as linear Regression.
• For a general graphical model, this canbe implemented using message passing(b) Ground Truth

Logistic Regression(N = 581K, D = 54)

Test Log Loss

336
VI-Hessian
VI-Reparam1337
VI-GGN
1Vadam
338
000.950510 collected
15about
20 whichwhichSGD
captures our ignorance about whichmodelgenerated
ourdata. mT
whichcapturesourignorance
339
0.9
1canbeexplained
away
given
enoughdata,
andisoften
referredtoas
model
uncerta
CVI
canbe
explained
away
given
enough
data,
an
340
0.85
For 2-D logistic, variances for
uncertainty can further be categorized uncertainty
into homoscedastic
uncertainty,uncertaintyw
further
becategorizedintoho
341 can0.8
ourmethod(red line)ofare
Figure 2.
(Left)
Illustration
Vadam for 2D
stant for different inputs, and heteroscedastic
uncertainty.
Heteroscedastic
stant fordifferent
inputs,
and
0.75
342
shrunkcompared
toheteroscedastic
thetruthconverges
thefastest
while
BBVI
isuncertai
the slowe
0.7
theinputstothemodel,
withsome
inputspotentiallyhavingmore
noisy
outputs
th343(dashedblacklines)
theinputstothemodel,withsome
inputspotandVadam)
work
betterthanmethods
withou
10
10
10
10
10
Seconds
eroscedastic
uncertainty is especially importantcomputer visionapplications.
F
344 foruncertainty
eroscedastic
is especially
importa345
depth regression, highly textured
input
imageswithstrong
vanishinglines
areexpedepth
regression,
highlytextured
input
image
Our method (thick lines)
Bayesian Regression with DNN
346 ofconfident predictions, whereas
an
input
image
a featurelesswall
isexpectedtoconfident
whereas
an
input
imag
reaches flat minima, while predictions,(N=1030, D = 8) DNN [50] with tanh347
uncertainty.
Naïve VI
uncertainty.
gradient descent (black lines)(with 200348
gets stuck.
MC samples)
In this paper we make the observationInthat
inpapermanywebig
datatheregimes(such asthis349
makeobservationthatthei
To enable Gaussian variational inference in deep models,
we learning with image data), it is
to deep
to modelto mostdeep
learning withimagealeatoricdata), ituncertai
is most350effective
Vprop
slightly modify existing deep learning methods, such whichascannot be explained away. Thiswhichis in351
comparison
toepistemic
uncertainty
w
cannotbeexplainedaway.
Thisisin
VAdam
Vprop for Approximate
RMSprop for Maximum
RMSprop and Adam, to perform variational inferenceexplained away
withthelarge
amounts
ofdataoften
availableinmachine
vision.
W352
explainedawaywiththelarge
amounts
ofdat
Vprop10
Bayesian Inference
Vprop10
Likelihood Estimation+momentum• Perturb the network weights during gradient evaluations
that modeling aleatoric uncertainty alone
comescost.
Out-of-data
353 at aaleatoric
thatpmodelinguncertainty examples
alone co✓µ✓µcannot+ ✏/ s +354
be identified
with aleatoric
uncertainty• Requires a few lines of code change in the existingidentified
software with epistemic uncertainty,
identified
with epistemic
uncertainty,
cannot bgr✓ f (✓)
gr✓ f (✓) + 355µ• Scales to very large problems with minimal effort For this we present a unified
Ourmethod(inred)ismuch22
Bayesianframework
whichusVadam
reduces
overfitting
forallows
smallerdata
we (a)present
aunified
Bayesiandeeps(1)s + gs
(1Fordeep)sthis+356
glearning
fasterthantheexistingfor
an increasing
data
size
(fromleftwith
to right
g aleatoric uncertainty
g357
pings from input
data
toand
composethese
together
epingsfrominput
datatoaleatoric
uncertaintpµµ↵methods(in magenta)
and Vadam doµµ ↵ p
RMSpropandAdamoverfit,
while
Our method also improves existing optimizers: reducetainty approximations. We
s + derive our framework+forbothregression
andclassificati
taintys358
approximations.
We
deriveourframew
also
requirefewer
samples
RMSprop and Adam behave like Vadam.
overfitting, avoid local minima, and encourages exploration.
and present results for per-pixel depth regression
semantic
segmentation
tasks (sand presentresultsfor per-pixel
depth regressi
359 andthe supplementary video for examples).
show
uncertain
360 how modelingtheWesupplementary
video foraleatoricexamples).
We s
can be used to learn loss attenuation, anda complimentary
approachthcan develop
be361
used tolearn loss attenuation,
andfordev
case. This demonstrates the efficacy ofcase.
our approachon difficult
large scale
362 demonstrates
Thistheandefficacyof ourtask
ap363
The
maincontributionsofthisworkare;Themaincontributions
ofthisworkare;
Graphical
model+
Deep
Structured
Inference
364
For Structured deep models
Our SAN
ModelsNetworks
365
methodis(shown in the right), we
1. We capture an accurate understandingofaleatoricandepistemicuncertainti
1. We capture an accurate understandin
asflexible366
withanovelapproachforclassification,
propose an efficient message
with a novel approach as
forVAEclassificati
but367
passing algorithm. Our
can also
368
2.
We
improvemodel
performance
by
1
3%
over
non-Bayesianbaselines
2.
We
improvemodel
performancebyalgorithm is a structured,cluster the369effecteffect
of
noisy
data
with
the
impliedattenuation
obtainedfromexplicit
of
noisy
data
with
the
impliedatasimilaramortized, and natural-gradient370
aleatoric
uncertainty,
toGMMaleatoric
uncertainty,(SAN) inference algorithm.
371372
3. We study the trade-offs between modelingaleatoricor epistemic
uncertain
3. We study
the trade-offs
betweenmodReferences
373 and comparing model performanc
izing the properties of each uncertainty
izing
theproperties
ofeachuncertai
1. Conjugate-Computation Variational Inference, (AISTATS 2017) M.E. KHAN AND W. LIN
374
time.
time.
2. Vprop: Variational Inference using RMSprop, (NIPS 2017, WORKSHOP ON BAYESIAN DEEP LEARNING), M.E. KHAN, Z. LIU, V. TANGKARATT
Y. GAL4. Vadam reduces overfitting on sm
375, ANDFigure
dataset.
3. Variational Adaptive-Newton Method for Explorative-Learning, (NIPS 2017, WORKSHOP ONAABI) M.E. KHAN, W. LIN, V. TANGKARATT,Z.
L
IU, AND D. NIELSEN
376
4. Variational Message Passing with Structured Inference Networks, (ICLR 2018) W. LIN, N. HUBACHER, AND M.E. KHAN
377 2378-10123

Fast Inference for Deep Learning

Encoder

Decoder

Scalable Inference in Structured Deep Models Using Message Passing