1AUTHOR(S): BAYESIAN SEGNET
000

Bayesian SegNet: Model Uncertainty in
003 Deep Convolutional Encoder-Decoder004
005 Architectures for Scene Understanding
001
002

006
007

BMVC 2017 Submission # 205

008
009
010
011

Abstract012

We present a deep learning framework for probabilistic pixel-wise semantic segmentation, which we term Bayesian SegNet. Semantic segmentation is an important toolfor visual scene understanding and a meaningful measure of uncertainty is essential for
decision making. Our contribution is a practical system which is able to predict pixelwise class labels with a measure of model uncertainty using Bayesian deep learning. We
achieve this by Monte Carlo sampling with dropout at test time to generate a posterior
distribution of pixel class labels. In addition, we show that modelling uncertainty improves segmentation performance by 2-3% across a number of datasets and architectures
such as SegNet, FCN, Dilation Network and DenseNet.

013
014
015
016
017
018
019
020
021
022

1

Introduction

023

Semantic segmentation requires an understanding of an image at a pixel level and is an im025 portant tool for scene understanding. Previous approaches to scene understanding used low
026 level visual features [24]. We are now seeing the emergence of machine learning techniques
027 for this problem [20, 25]. While deep learning sets the benchmark on many popular datasets[6, 9], we lack interpretability and understanding of these models. One way to understand028
what a model knows, or does not no, is a measure of model uncertainty.
029
Uncertainty should be a natural part of any predictive system’s output. Knowing the con030
fidence with which we can trust the semantic segmentation output is important for decision
031
making. For instance, a system on an autonomous vehicle may segment an object as a pedes032
trian. But it is desirable to know the model uncertainty with respect to other classes such as
033
street sign or cyclist as this can have a strong effect on behavioural decisions. Uncertainty is034
also immediately useful for other applications such as active learning [5], semi-supervised
035
learning, or label propagation [1].
036
The main contribution of this paper is extending deep convolutional encoder-decoder
037 neural network architectures [2] to Bayesian convolutional neural networks which can pro038 duce a probabilistic segmentation output [11]. In section 4 we propose Bayesian SegNet, a
039 probabilistic deep convolutional neural network framework for pixel-wise semantic segmen040 tation. We use dropout at test time which allows us to approximate epistemic uncertainty by
041 sampling from a Bernoulli distribution across the network’s weights. This is achieved with
042 no additional parametrisation. In particular, we analyse which part of deep encoder decoder
043 models benefit from Bayesian modelling.
024

044

c 2017. The copyright of this document resides with its authors.
045 It may be distributed unchanged freely in print or electronic forms.

2

AUTHOR(S): BAYESIAN SEGNET
Input

Convolutional Encoder-Decoder

Stochastic Dropout
Samplesmean

Segmentation

046
047
048

Model Uncertainty
variance

RGB Image

Conv + Batch Normalisation + ReLU
Dropout
Pooling/Upsampling

049
050
051

Softmax

052
053

Figure 1: A schematic of the Bayesian SegNet architecture. This diagram shows the 054
entire pipeline for the system which is trained end-to-end in one step with stochastic gradient 055
descent. The encoders are based on the 13 convolutional layers of the VGG-16 network [27], 056
with the decoder placing them in reverse. The probabilistic output is obtained from Monte 057
Carlo samples of the model with dropout at test time. We take the variance of these softmax 058
059
samples as the model uncertainty for each class.
060
061

In section 5, we demonstrate that our Bayesian approach improves performance of a 062
number of baseline models on prominent scene understanding datasets, CamVid [3], SUN
063
RGB-D [28] and Pascal VOC [9]. In particular, we find a larger performance improvement064
on smaller datasets such as CamVid where the Bayesian Neural Network is able to cope065
with the additional uncertainty from a smaller amount of data. Moreover, we show that066
this technique is broadly applicable across a number of state of the art architectures andachieves a 2-3% improvement in segmenation accuracy when applied to SegNet [2], FCN 067[20], Dilation Network [30] and DenseNet [15]. Finally in section 5.1 we demonstrate the 068
effectiveness of model uncertainty. We explore what factors contribute to Bayesian SegNet 069
070
making an uncertain prediction.
071
072

2

Related Work

073
074

Semantic pixel labelling was initially approached with TextonBoost [24], TextonForest [23] 075
and Random Forest Based Classifiers [25]. Deep learning architectures are now the standard 076
approach for pixel-wise segmentation, such as SegNet [2] Fully Convolutional Networks 077(FCN) [20] and Dilation Network [30]. FCN is trained using stochastic gradient descent 078
with a stage-wise training scheme. SegNet was the first architecture proposed that can be 079
trained end-to-end in one step, due to its lower parametrisation. We have also seen methods 080
which improve on these core architectures by adding post processing tools. HyperColumn
081[13] and DeConvNet [22] use region proposals to bootstrap their core segmentation engine.
082
DeepLab [4] post-processes with conditional random fields (CRFs) and CRF-RNN [32] use
083
recurrent neural networks. These methods improve performance by smoothing the output
084
and ensuring label consistency. However none of these segmentation methods generate a
085
probabilistic output with a measure of model uncertainty.
Neural networks which model uncertainty are known as Bayesian neural networks [7, 08621]. They offer a probabilistic interpretation of deep learning models by inferring distribu- 087
tions over the networks’ weights. They are often computationally very expensive, increasing 088
the number of model parameters without increasing model capacity significantly. Perform- 089
ing inference in Bayesian neural networks is a difficult task, and approximations to the model 090
091
posterior are often used, such as variational inference [12].

AUTHOR(S): BAYESIAN SEGNET

3

On the other hand, the already significant parametrization of convolutional network architectures leaves them particularly susceptible to over-fitting without large amounts of train094 ing data. A technique known as dropout is commonly used as a regularizer in convolutional
095 neural networks to prevent over-fitting and co-adaptation of features [29]. During training
096 with stochastic gradient descent, dropout randomly removes units within a network. By do097 ing this it samples from a number of thinned networks with reduced width. At test time,
098 standard dropout approximates the effect of averaging the predictions of all these thinned
099 networks by using the weights of the unthinned network – referred to as weight averaging.
Gal and Ghahramani [11] have interpreted dropout as approximate Bayesian inference
100
over
the network’s weights. [10] shows that dropout can be used at test time to impose a101
Bernoulli
distribution over the convolutional net filter’s weights, without requiring any addi102
tionalmodel
parameters. This is achieved by sampling the network with randomly dropped
103
out units at test time. We can consider these as Monte Carlo samples obtained from the pos104
terior distribution over models. This technique has seen success in modelling uncertainty for105
camera relocalisation [17]. Here we apply it to pixel-wise semantic segmentation.
106
In particular, MC dropout is able to capture epistemic uncertainty, which accounts for un107
certainty in the model parameters – uncertainty which captures our ignorance about which
108
model generated our collected data [18]. Semantic segmentation models can typically only109
capture aleatoric uncertainty, from the entropy of the class logits, which measures noise
110
inherent in the observations. Bayesian SegNet models epistemic uncertainty which is impor111 tant for safety applications because it is required to understand examples which are different
112 from training data [18].
092
093

113
114
115

3

SegNet Architecture

116

We briefly review the SegNet architecture [2] which we extend to produce Bayesian SegNet.
SegNet is a deep convolutional encoder decoder architecture which consists of a sequence
119 of non-linear processing layers (encoders) and a corresponding set of decoders followed by
120 a pixel-wise classifier. Typically, each encoder consists of one or more convolutional lay121 ers with batch normalisation and a ReLU non-linearity, followed by non-overlapping max122 pooling and sub-sampling. The sparse encoding due to the pooling process is upsampled in
123 the decoder using the max-pooling indices in the encoding sequence. This has the important
124 advantage of retaining class boundary details in the segmented images and also reducing the
125 total number of model parameters. The model is trained end to end using stochastic gradientdescent.
126
We take both SegNet [2] and a smaller variant termed SegNet-Basic [2] as our base127
models.
SegNet’s encoder is based on the 13 convolutional layers of the VGG-16 network
128[27] followed by 13 corresponding decoders. SegNet-Basic is a much smaller network with
129
only four layers each for the encoder and decoder with a constant feature size of 64. We
130
use SegNet-Basic as a smaller model for our analysis since it conceptually mimics the larger
131
architecture.
117
118

132
133
134
135
136
137

4

Bayesian Semantic Segmentation Model

To produce a probabilistic segmentation with Bayesian SegNet, we are interested in finding
the posterior distribution over the convolutional weights, W, given our observed training data

4

AUTHOR(S): BAYESIAN SEGNET

X and labels Y.

138

p(W | X, Y)(1) 139140

In general, this posterior distribution is not tractable, therefore we need to approximate the141
distribution of these weights [7].
We use Monte Carlo dropout samples to approximate inference in a Bayesian neural 142
network [10]. Typically, dropout [29] was used during training to sample thinner models 143
to regularise the network. During inference, these models were combined with weight av- 144
eraging. In this work, we propose to use dropout during inference to obtain samples from 145
the posterior distribution of models. Gal and Ghahramani [10] link this technique to varia- 146
tional inference in Bayesian convolutional neural networks, with Bernoulli distributions over 147
the network’s weights. We leverage this method to perform probabilistic inference over our 148
149
segmentation model.
This technique allows us to learn the distribution over the network’s weights, q(W), by 150
minimising the Kullback-Leibler (KL) divergence between this approximating distribution 151
and the full posterior;152
KL(q(W) || p(W | X, Y)).
(2) 153
where the approximating variational distribution q(Wi ) for every convolutional layer i, with 154
units wi, j , is defined with Bernoulli distributed random variables and variational parameters, 155ŵ, as: wi, j ∼ ŵi, j Bernoulli(pi ) for all units j. At the extreme case, if we have infinite units 156
for each layer our approximate model approaches a Gaussian process [10]. The dropout 157
probabilities, pi , could be optimised. However we fix them to the standard probability of 158
159
dropping a connection as 50%, i.e. pi = 0.5 [29].
In [10] it was shown that minimising the cross entropy loss objective function has the 160
effect of minimising the Kullback-Leibler divergence term. We use this loss and train the 161
network with stochastic gradient descent. This will encourage the model to learn a distribu- 162
163
tion of weights which explains the data well while preventing over-fitting.
We train the model with dropout and sample the posterior distribution over the weights at 164
test time using dropout to obtain the posterior distribution of softmax class probabilities. We 165
take the mean of these samples for our segmentation prediction and use the variance to output 166
model uncertainty for each class. We take the mean of the per-class variance measurements 167
as an overall measure of model uncertainty. We also explored using the variation ratio as a 168
measure of uncertainty (i.e. the percentage of samples which agree with the class prediction) 169
however we found this to qualitatively produce a more binary measure of model uncertainty. 170
Fig. 1 shows a schematic of the segmentation prediction and model uncertainty estimate 171
process.
172
173

4.1

Probabilistic Variants

174
175

A fully Bayesian network should be trained with dropout after every convolutional layer.
176
However we found in practice that this was too strong a regulariser, causing the network tolearn very slowly. We therefore explored a number of variants that have different config- 177
urations of Bayesian or deterministic encoder and decoder units. We note that an encoder 178
unit contains one or more convolutional layers followed by a max pooling layer. A decoder 179
unit contains one or more convolutional layers followed by an upsampling layer. The vari- 180
ants are: Bayesian Encoder dropout after each encoder unit, Bayesian Decoder dropout 181
after each decoder unit, Bayesian Encoder-Decoder dropout after each encoder and de- 182
coder unit, Bayesian Center dropout after the deepest encoder, before the decoder stage, 183

5

AUTHOR(S): BAYESIAN SEGNET
184
185

Probabilistic Variants
No Dropout
Dropout Encoder
Dropout Decoder
Dropout Enc-Dec
Dropout Central Enc-Dec
Dropout Center
Dropout Classifier

186
187
188
189
190
191

WeightAveragingGC
I/U
82.9 62.4 46.4
80.6 68.9 53.4
82.4 64.5 48.8
79.9 69.0 54.2
81.1 70.6 55.7
82.9 68.9 53.1
84.2 62.6 46.9

Monte Carlo
Sampling
G
C
I/U
n/a
n/a
n/a
81.6 69.4 54.0
82.6 62.4 46.1
79.8 68.8 54.0
81.6 70.6 55.8
82.7 68.9 53.2
84.2 62.6 46.8

Training
Fit
G
C
I/U
94.7 96.2 92.7
90.6 92.5 86.3
94.6 96.0 92.4
88.9 89.0 80.6
90.4 92.3 85.9
93.3 95.4 91.2
94.9 96.0 92.3

Table 1: Architecture Variants for SegNet-Basic on the CamVid dataset [3]. We compare
the performance of weight averaging against 50 Monte Carlo samples. We quantify perfor194 mance with three metrics; global accuracy (G), class average accuracy (C) and intersection
195 over union (I/U). Results are shown as percentages (%). We observe that dropping out every
196 encoder and decoder is too strong a regulariser and results in a lower training fit. The optimal
197 result across all classes is when only the central encoder and decoders are dropped out.
192
193

198
199

Bayesian Central Four Encoder-Decoder dropout after the central four encoder and decoder units and Bayesian Classifier dropout after the last decoder unit, before the classifier.
201
For analysis we use the smaller eight layer SegNet-Basic architecture [2] and test these202 Bayesian variants on the CamVid dataset [3]. We observe qualitatively that all four variants
203 produce similar looking model uncertainty output. That is, they are uncertain near the border
204 of segmentations and with visually ambiguous objects, such as cyclist and pedestrian classes.
205 However, Table 1 shows a difference in quantitative segmentation performance.
206
We observe using dropout after all the encoder and decoder units results in a lower train207 ing fit and poorer test performance as it is too strong a regulariser on the model. We find
208 that dropping out half of the encoder or decoder units is the optimal configuration. The best
209 configuration is dropping out the deepest half of the encoder and decoder units. We therefore
210 benchmark our Bayesian SegNet results on the Central Enc-Dec variant. For the full 26 layer
211 Bayesian SegNet, we add dropout to the central six encoders and decoders. This is illustrated
212 in Fig. 1.
In the lower layers of convolutional networks basic features are extracted, such as edges
213
andcorners [31]. These results show that applying Bayesian weights to these layers does214
notresultin a better performance. We believe this is because these low level features are
215
consistentacross the distribution of models because they are better modelled with determin216
istic
weights.
However, the higher level features that are formed in the deeper layers, such217
as shape and contextual relationships, are more effectively modelled with Bayesian weights.
200

218
219
220
221

4.2

Comparing Weight Averaging and Monte Carlo Dropout Sampling

Monte Carlo dropout sampling qualitatively allows us to understand the model uncertaintyof the result. However, for segmentation, we also want to understand the quantitative dif223
ference between sampling with dropout and using the weight averaging technique proposed224
by [29]. Weight averaging proposes to remove dropout at test time and scale the weights
225 proportionally to the dropout percentage. Fig. 2 shows that Monte Carlo sampling with226 dropout performs better than weight averaging after approximately 6 samples. We also ob227 serve no additional performance improvement beyond approximately 40 samples. Therefore
228 the weight averaging technique produces poorer segmentation results, in terms of global ac229 curacy, in addition to being unable to provide a measure of model uncertainty. However,222

6

AUTHOR(S): BAYESIAN SEGNET

81.5
81
80.5
80
79.5
79
0

230

87.5

Monte Carlo Dropout Sampling
Weight Averaging
10

20

30

40

Number of Samples(a) SegNet Basic

50

60

Global Accuracy (%)

Global Accuracy (%)

82

231
87

232
86.5

233

86

234

85.5

85
0

Monte Carlo Dropout Sampling
Weight Averaging
10

20

30

40

Number of Samples(b) SegNet

50

60

235
236
237
238
239

Figure 2: Global segmentation accuracy against number of Monte Carlo samples for240
both SegNet and SegNet-Basic. Results averaged over 5 trials, with two standard deviation
error bars, are shown for the CamVid dataset. This shows that Monte Carlo sampling out- 241
performs the weight averaging technique after approximately 6 samples, and converges after 242
243
approx. 40 samples.
244
245

sampling comes at the expense of inference time, but when computed in parallel on a GPU 246
this cost can be reduced for practical applications.
247
248
249

5

Experiments

250
251

We implement Bayesian SegNet using the Caffe library [16]. We train the whole system 252
end-to-end using stochastic gradient descent with a base learning rate of 0.001 and weight 253
decay parameter equal to 0.0005. Following [2] we train SegNet with median frequency class 254
balancing using the formula proposed by Eigen and Fergus [8]. We use batch normalisation 255
after every convolutional layer [14].
256
We quantify the performance of Bayesian SegNet on three different benchmarks using 257
our Caffe implementation. Through this process we demonstrate the efficacy of Bayesian 258
SegNet for a wide variety of scene segmentation tasks which have practical applications. 259
CamVid [3] is a road scene understanding dataset which has applications for autonomous 260
driving. SUN RGB-D [28] is a very challenging and large dataset of indoor scenes which is261
important for domestic robotics. Finally, Pascal VOC 2012 [9] is a RGB dataset for object
262
segmentation.
263
CamVid is a road scene understanding dataset with 367 training images and 233 testing
264
images of day and dusk scenes [3] with 11 classes. We resize images to 360x480 pixels for265
training and testing of our system. We show our Bayesian method outperforms other models
266
in Table 2 with qualitative results in Fig. 5.
267
SUN RGB-D [28] is a challenging and large dataset of indoor scenes with 5285 training
and 5050 testing images. The images are captured by different sensors and are labelled with 268
37 indoor semantic classes. Table 2 and Fig. 4 compare to other models, including those 269
which use depth input. Our method outperforms all of these other techniques. We also note 270271that an earlier benchmark dataset, NYUv2 [26], is included as part of this dataset.
Pascal VOC12 segmentation challenge [9] consists of segmenting 20 salient object 272
classes from widely varying backgrounds. We train on the 12031 training images and 1456 273
testing images. Table 3 shows our results compared to other methods, with qualitative results 274
275
in Fig. 5.

AUTHOR(S): BAYESIAN SEGNET

7

276
277
278
279
280
281
282
283
284
285
286
287
288
289
290

Figure 3: Bayesian SegNet results on CamVid dataset [3]. From top: input image, groundtruth, Bayesian SegNet’s segmentation prediction, and overall model uncertainty averaged
292 across all classes (with darker colours indicating more uncertain predictions).
291
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307

Figure 4: Bayesian SegNet results on the SUN RGB-D dataset [28]. Bayesian SegNet uses only RGB input
308 and is able to accurately segment 37 classes in this challenging dataset.
309
310
311
312
313
314
315
316
317
318
319
320
321

Figure 5: Pascal VOC 2012 dataset [9]. Ground truth is not publicly available for these test images.

8

AUTHOR(S): BAYESIAN SEGNET
CamVidG
C
SegNet-Basic [2]62.3 82.8
SegNet [2]65.9 88.6
FCN 8 [20]64.2 83.1
DeconvNet [22]62.1 85.9
DeepLab-LargeFOV-DenseCRF [4] 60.7 89.7
DenseNet [15]
Bayesian SegNet Models in this work:
Bayesian SegNet-Basic
70.5 81.6
Bayesian SegNet
76.3 86.9
Bayesian DenseNet

I/U
46.3
50.2
52.0
48.9
54.7
66.9
55.8
63.1
67.2

SUN RGB-D

G

C

I/U

n/a
68.2
66.1
67.0
70.3
71.2

9.3
38.4
32.3
33.0
35.6
45.9

n/a
27.4
22.6
24.1
22.1
30.7

RGB
Liu et al. [19]FCN 8 [20]DeconvNet [22]DeepLab-LargeFOV-CRF [4]SegNet [2]Bayesian SegNet (this work)

Table 2: Quantitative results for CamVid [3] (left) and SUN RGB-D [28] (right).

322
323
324
325
326
327
328
329
330
331
332

To demonstrate the general applicability of this method, we also apply it to other deep333
learning architectures trained with dropout; FCN [20] and Dilation Network [30]. We selectthese state-of-the-art methods as they are already trained by their respective authors using 334
dropout. We take their trained, open source models off the shelf, and evaluate them using 50 335
Monte Carlo dropout samples. Table 3 shows the mean IoU result of these methods evaluated 336
as Bayesian Neural Networks, as computed by the online evaluation server. This shows the 337
general applicability of our method. By leveraging this underlying Bayesian framework our 338
339
method obtains 2-3% improvement across this range of architectures.
340
341
Parameters
Pascal VOC Test IoU
342
Method(Millions) Non-Bayesian Bayesian343
Dilation Network [30]
140.8
71.3
73.1
344
FCN-8 [20]
134.5
62.2
65.4
345
SegNet [2]
29.45
59.1
60.5
Table 3: Pascal VOC12 [9] test results evaluated from the online evaluation server. We 346
compare to competing deep learning architectures. Bayesian SegNet is considerably smaller 347
but achieves a competitive accuracy to other methods. We also evaluate FCN [20] and Di- 348
lation Network (front end) [30] with Monte Carlo dropout sampling. We observe a 2-3% 349
improvement in segmentation performance across all three deep learning models when us- 350
351
ing the Bayesian approach.
352
353

5.1

Understanding Model Uncertainty

354
355
356

Qualitative observations. Fig. 5 shows segmentations and model uncertainty results from
Bayesian SegNet on CamVid Road Scenes [3]. Fig. 4 shows SUN RGB-D Indoor Scene 357
Understanding [28] results and Fig. 5 has Pascal VOC [9] results. These figures show the 358
qualitative performance of Bayesian SegNet. We observe that segmentation predictions are 359
smooth, with a sharp segmentation around object boundaries. Also, when the model predicts 360
an incorrect label, the model uncertainty is generally very high. More generally, we observe 361
362that a high model uncertainty is predominantly caused by three situations.
Firstly, at class boundaries the model often displays a high level of uncertainty. This 363
reflects the ambiguity surrounding the definition of defining where these labels transition. 364
365
The Pascal results clearly illustrated this in Fig. 5.
Secondly, objects which are visually difficult to identify often appear uncertain to the 366367
model. This is often the case when objects are occluded or at a distance from the camera.

9

AUTHOR(S): BAYESIAN SEGNET
368
369
370
371
372
373
374
375
376
377

(a) Performance vs. mean model uncertainty

378(b) Class frequency vs. mean model uncertainty

379

Figure 6: Bayesian SegNet performance and frequency compared to mean model uncertainty for each class in CamVid road scene understanding dataset. These figures show
381 a strong inverse relationships. We observe in (a) that classes that Bayesian SegNet is more
382 confident at are more prevalent in the dataset. Conversely, for the more rare classes such
383 as Sign Symbol and Bicyclist, Bayesian SegNet has a much higher model uncertainty. (b)384 shows that the model is more confident with more accurate classes.
380

385
386
387

The third situation causing model uncertainty is when the object appears visually ambiguous to the model. As an example, cyclists in the CamVid results (Fig. 5) are visually
389 similar to pedestrians, and the model often displays uncertainty around them. We observe
390 similar results with visually similar classes in SUN (Fig. 4) such as chair and sofa, or bench
391 and table. In Pascal this is often observed between cat and dog, or train and bus classes.
388

392

Quantitative observations. To understand what causes the model to be uncertain, we
have plotted the relationship between uncertainty and accuracy in Fig. 6(a) and between un394 certainty and the frequency of each class in the dataset in Fig. 6(b). Uncertainty is calculated
395 as the mean uncertainty value for each pixel of that class in a test dataset. We observe an
396 inverse relationship between uncertainty and class accuracy or class frequency. This shows
397 that the model is more confident about classes which are easier or occur more often, and less
398 certain about rare and challenging classes.
393

399
400
401
402

6

Conclusions

403
404

We have presented Bayesian SegNet, the first probabilistic framework for semantic segmentation using deep learning, which outputs a measure of model uncertainty for each class.
406 We show that the model is uncertain at object boundaries and with difficult and visually
407 ambiguous objects.
405

408

We quantitatively show Bayesian SegNet produces a reliable measure of model uncertainty, improving segmentation performance by 2-3% across a number of state of the art
410 architectures such as SegNet, FCN and Dilation Network, while requiring no additional pa411 rameters. We demonstrate how to apply our knowledge of uncertainty to active learning
412 which significantly reduces the requirement for expensive labelled data. For future work we
413 intend to explore how video data can improve our model’s performance.
409

10

References

AUTHOR(S): BAYESIAN SEGNET
414
415[1] Vijay Badrinarayanan, Fabio Galasso, and Roberto Cipolla. Label propagation in video 416
sequences. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Confer- 417
ence on, pages 3265–3272. IEEE, 2010.
418[2] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep con- 419
volutional encoder-decoder architecture for image segmentation. arXiv preprint 420
421
arXiv:1511.00561, 2015.
422[3] Gabriel J Brostow, Julien Fauqueur, and Roberto Cipolla. Semantic object classes in 423
video: A high-definition ground truth database. Pattern Recognition Letters, 30(2): 424
88–97, 2009.
425[4] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L 426
Yuille. Semantic image segmentation with deep convolutional nets and fully connected 427
428
crfs. arXiv preprint arXiv:1412.7062, 2014.
429[5] David A Cohn, Zoubin Ghahramani, and Michael I Jordan. Active learning with statis430
tical models. Journal of artificial intelligence research, 1996.
431[6] Camille Couprie, Clément Farabet, Laurent Najman, and Yann LeCun. Indoor semantic 432
433
segmentation using depth information. arXiv preprint arXiv:1301.3572, 2013.
434[7] John Denker and Yann Lecun. Transforming neural-net output levels to probability
435
distributions. In Advances in Neural Information Processing Systems 3. Citeseer, 1991.
436[8] David Eigen and Rob Fergus. Predicting depth, surface normals and seman- 437
tic labels with a common multi-scale convolutional architecture. arXiv preprint 438
arXiv:1411.4734, 2014.
439
440[9] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew
441
Zisserman. The pascal visual object classes (voc) challenge. International journal of442
computer vision, 88(2):303–338, 2010.
443[10] Yarin Gal and Zoubin Ghahramani. Bayesian convolutional neural networks with 444
bernoulli approximate variational inference. arXiv:1506.02158, 2015.
445[11] Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Represent- 446
447
ing model uncertainty in deep learning. arXiv:1506.02142, 2015.
448[12] Alex Graves. Practical variational inference for neural networks. In Advances in Neural 449
Information Processing Systems, pages 2348–2356, 2011.
450[13] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Hypercolumns 451
for object segmentation and fine-grained localization. arXiv preprint arXiv:1411.5752, 452
453
2014.
454

[14] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network 455
training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015. 456[15] Simon Jégou, Michal Drozdzal, David Vazquez, Adriana Romero, and Yoshua Bengio. 457
The one hundred layers tiramisu: Fully convolutional densenets for semantic segmen- 458
459
tation. arXiv preprint arXiv:1611.09326, 2016.

AUTHOR(S): BAYESIAN SEGNET

11[16] Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross
Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture
for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014.
462
460
461
463
464
465
466
467

[17] Alex Kendall and Roberto Cipolla. Modelling uncertainty in deep learning for camera
relocalization. arXiv preprint arXiv:1509.05909, 2015.
[18] Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learningfor computer vision? arXiv preprint arXiv:1703.04977, 2017.

468[19] Ce Liu, Jenny Yuen, Antonio Torralba, Josef Sivic, and William T Freeman. Sift flow:
Dense correspondence across different scenes. In Computer Vision–ECCV 2008, pages
470
28–42. Springer, 2008.
469
471
472
473

[20] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks forsemantic segmentation. arXiv preprint arXiv:1411.4038, 2014.

474[21] David JC MacKay. A practical bayesian framework for backpropagation networks.
Neural computation, 4(3):448–472, 1992.
476
475
477
478

[22] Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han. Learning deconvolution network for semantic segmentation. arXiv preprint arXiv:1505.04366, 2015.

479[23] Jamie Shotton, Matthew Johnson, and Roberto Cipolla. Semantic texton forests forimage categorization and segmentation. In Computer vision and pattern recognition,
2008. CVPR 2008. IEEE Conference on, pages 1–8. IEEE, 2008.
482
480
481
483

[24] Jamie Shotton, John Winn, Carsten Rother, and Antonio Criminisi. Textonboost forimage understanding: Multi-class object recognition and segmentation by jointly mod485
eling texture, layout, and context. International Journal of Computer Vision, 81(1):486
2–23, 2009.
484

487

[25] Jamie Shotton, Toby Sharp, Alex Kipman, Andrew Fitzgibbon, Mark Finocchio, Andrew Blake, Mat Cook, and Richard Moore. Real-time human pose recognition in parts489
from single depth images. Communications of the ACM, 56(1):116–124, 2013.
488
490
491

[26] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Computer Vision–ECCV 2012, pages
493
746–760. Springer, 2012.
492
494[27] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for largescale image recognition. arXiv preprint arXiv:1409.1556, 2014.
496
495
497

[28] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene
understanding benchmark suite. In Proceedings of the IEEE Conference on Computer
499
Vision and Pattern Recognition, pages 567–576, 2015.
498
500[29] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan
Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting.
502
The Journal of Machine Learning Research, 15(1):1929–1958, 2014.
503
501

504
505

[30] Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions.
In ICLR, 2016.

12

AUTHOR(S): BAYESIAN SEGNET[31] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional net- 506
works. In Computer Vision–ECCV 2014, pages 818–833. Springer, 2014.
507
508[32] Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet,
509
Zhizhong Su, Dalong Du, Chang Huang, and Philip Torr. Conditional random fields as
510
recurrent neural networks. arXiv preprint arXiv:1502.03240, 2015.
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551