Efficient Uncertainty Estimation in Semantic Segmentation via Distillation
Christopher J. Holder Muhammad Shafique
New York University Abu Dhabi
Abu Dhabi, United Arab Emirates{chris.holder, ms12713}@nyu.edu

Abstract
Deep neural networks typically make predictions withlittle regard for the probability that a prediction might beincorrect. Attempts to address this often involve input data
undergoing multiple forward passes, either of multiple
models or of multiple configurations of a single model, andconsensus among outputs is used as a measure ofconfidence. This can be computationally expensive, as thetime taken to process a single input sample increases
linearly with the number of output samples being
generated, an important consideration in real-timescenarios such as autonomous driving, and so we propose
Uncertainty Distillation as a more efficient method forquantifying prediction uncertainty. Inspired by the conceptof Knowledge Distillation, whereby the performance of acompact model is improved by training it to mimic theoutputs of a larger model, we train a compact model tomimic the output distribution of a large ensemble of models,such that for each output there is a prediction and a
predicted level of uncertainty for that prediction. We apply
Uncertainty Distillation in the context of a semantic
segmentation task for autonomous vehicle scene
understanding and demonstrate a capability to reliably
predict pixelwise uncertainty over the resultant classprobability map. We also show that the aggregate pixel
uncertainty across an image can be used as a metric forreliable detection of out-of-distribution data.

1. Introduction
Deep neural networks have come to dominate the field ofmachine learning, surpassing prior approaches in a
multitude of tasks across domains including natural
language processing and computer vision. Despite their
unrivalled performance in many of these tasks, neuralnetworks typically suffer from a lack of interpretability,
posing major challenges when it comes to analysing what
exactly a model has learned, how a given input maps to a
subsequent output or assigning any meaningful measure ofconfidence to outputs. This is an especially poignant
problem when neural networks are deployed in safety
critical systems, such as autonomous vehicles, where
undetected errors have the potential to result in loss of life.

Many proposed solutions to the problem of quantifying
the uncertainty of model outputs require the computation ofa distribution of outputs. Some methods by which this
distribution can be computed include Bayesian neuralnetworks [1], whereby model parameters are learned as
distributions rather than fixed points, Monte Carlo Dropout[2], in which model parameters are randomly set to zero
during each inference pass, Variational Autoencoders [3],wherein a sample’s latent representation is encoded as a
distribution rather than a point within the feature space, andEnsembles [4], which comprise multiple models eachtrained to perform the same task but with slightly different
parameters due to the stochasticity of the training process.
In each of these cases, sampling from a stochastic processprovides a distribution of outputs from a single input
sample, and the parameters of this distribution can be used
to compute some measure of uncertainty, however thissampling process is computationally expensive, due tomany forward passes being required to capture an adequate
sample size.
In this work we present a novel approach for the efficientcomputation of uncertainty, which we term ‘Uncertainty
Distillation’. It has been shown that through the paradigmof Knowledge Distillation, a compact student model canlearn to produce outputs that closely match those of a larger
teacher model, boosting its performance beyond that
achieved with conventional supervised learning. In the caseof Uncertainty Distillation, the teacher is an ensemble andthe student learns to output a distribution that closely
matches that of the outputs produced by the models thatcomprise the teacher ensemble. In the context of semantic
segmentation for autonomous driving, we demonstrate that
our approach results in a compact model capable of reliably
predicting the uncertainty of its own predictions.
This work makes the following contributions:
1. A novel method for the distillation of the uncertainty
quantification capability of deep ensemble networks into asingle compact model.
2. Generation of pixelwise uncertainty maps in thecontext of semantic segmentation from a single pass of asingle model.
3. Robust

detectionofout-of-distribution(OOD)samples via aggregation of uncertainty values across whole
images.

2. Related Work2.1. Uncertainty Quantification
Most neural networks addressing classification
problems, including semantic segmentation, use thesoftmax function to give a set of output probabilities foreach possible class, however it has been shown [5] thatthese probabilities do not reliably capture any meaningfulmeasure of confidence or uncertainty in a model’s
predictions. There is a significant body of work attempting
to address this problem of quantifying the uncertainty ofpredictions in a variety of ways.
Much prior work towards quantifying the uncertainty ofneural network outputs relies on computationally expensive
sampling processes. Bayesian neural networks [1] learn a
distribution over their parameters, which can be sampledfrom at inference time to create potentially infinite
ensembles. Several challenges exist in training suchmodels, particularly scalability and selection of a suitableprior, and there have been many works attempting toaddress these or to otherwise approximate a true Bayesianneural network. In [6], the concept of Bayes by Backprop is
introduced, in which parameter uncertainty is computed
during gradient updates. Probabilistic backpropagation is
proposed in [7], with the posterior over model weights
approximated using a product of Gaussians. A Bayesianneural network can be approximated using Monte Carlo
Dropout [2], in which a randomly selected set of model
parameters are ignored on each pass creating a huge numberof potential model permutations from which to sample.
Bayesian SegNet [8] applies this approach to semantic
segmentation, generating pixelwise uncertainty values thatcan subsequently be used to improve segmentation
accuracy. Another method for making a neural network
non-deterministic is that of the Variational Autoencoder[9], which learns to encode an input sample as a distribution
in latent feature space, which can subsequently be infinitely
sampled from to generate a distribution of outputs.
Ensemble models can be considered an approximation of a
Bayesian neural network, with the distribution ofcomponent model outputs used to compute uncertainty, as
in [4]. In all these cases, uncertainty is derived from the
distribution of outputs over multiple passes of the same
input sample, with the level of consensus achieved acrossoutputs taken as a measure of confidence. This iscomputationally expensive, and so we propose shifting thissampling process from inference to training time, resulting
in a single efficient deterministic model capable ofpredicting the uncertainty of its own predictions in a single
pass.
Work towards quantifying uncertainty without sampling

include [10] and [11], in which a model learns to predict theparameters of a Dirichlet distribution over outputs, however
training such models is challenging and requires a welldefined prior. In [12] a model learns to represent regression
outputs in the form of a normal inverse-gamma distribution,from which aleatoric and epistemic uncertainty can bemodelled, although the technique is not applicable toclassification problems. In [13] a variational Dirichlet
framework is proposed, with entropy of the learned
posterior used to quantify uncertainty, however OOD
training data is required.
In this work, we address the main limitation of samplingbased techniques, that is their computational expense, by
distilling the uncertainty quantification capability of an
ensemble into a single model. Our approach is relatively
straightforward to train, does not require OOD training
data, and has the potential to be adapted for any applicationof a neural network.

2.2. Knowledge Distillation
The concept of compressing the knowledge of a large
ensemble into a single compact neural network was first
proposed in [14], taking the logits of a teacher model astargets for a student. Building on this work, [15] presented
the approach now commonly known as Knowledge
Distillation, which replaces the logit targets with hightemperature softmax targets. In scenarios where thecomputational requirements of a large model are notavailable, knowledge distillation has demonstrated better
results than those of a small model trained using standard
ground truth labels.
Later work has built upon this idea via training thestudent to also match intermediate feature maps [16] orattention maps [17] to those of the teacher, or by applying
adversarial learning [18], whereby a discriminator network
learns to classify outputs as those of the teacher or student,
and the student aims to produce outputs indistinguishable
from those of its teacher.
Several works have applied knowledge distillation in thecontext of semantic segmentation: In [19] adversarial loss
is used so that the student learns to output segmentation
maps that match those of the teacher; In [20] the student istrained to match the teacher’s latent representation; In [21]the teacher is trained to predict depth as well as
segmentation, with the resulting depth-aware embedding
used to train the student for segmentation alone.
In this work we build on the concept of knowledge
distillation to transfer not just predictions from teacher tostudent, but the distributions of those predictions over ateacher ensemble so that the student learns to quantify theuncertainty of its own predictions.

Figure 1. The process for training the student model in Uncertainty Distillation. For input xi, ensemble models f1 to fF comprisingteacher ft output probability maps x’f1i to x’fFi, the elementwise mean and standard deviation of which populate Mi and ∑i respectively.
Student model fs outputs class probability map x’fsi, which is compared with ground truth segmentation yi to compute classification loss
Lcls and to Mi to compute distillation loss Ldist, and uncertainty map x*i, which is compared to ∑i to compute uncertainty loss Lunc.
These losses are then combined to optimise fs such that it learns to approximate the output distribution of ft.

the resulting distribution:′

3. Uncertainty Distillation
In a typical supervised learning scenario, we aim to learn
a function f that minimises some loss function L(f(xi),yi)over a dataset D comprising N input samples xi each pairedwith a ground truth target yi:
∑

,

(1)

In the case of a convolutional neural network (CNN)
performing semantic segmentation of images, xi is an RGB
input image of dimensions 3 × H × W, f is a CNN
parameterised by a set of weights w that outputs a classprobability map x’i of dimensions C × H × W, C being thenumber of classes present in D, yi is the ground truth
segmentation map pertaining to xi, and L is some measureof similarity between x’i and yi, commonly cross entropy.
At each training step, the gradient of L(x’i, yi) is computedwith respect to w, which are adjusted accordingly by somelearning scheme.
When Knowledge Distillation is applied, a teacher modelft is trained via the method described above, and the outputs
from this model are used to train a student model fs. The
softmax function is applied to the final layer of ft so that
outputs x’ are within the range (0, 1), with a single tunable
parameter, temperature T, determining the smoothness of

, ,

∑"
#

, ,

⁄

#, ,

⁄

(2)

Where c ∈ {1,…C}, v ∈ {1,…H} and u ∈ {1,…W}
denote the location of an element within our classprobability map, and is the logits class probability map
at the final layer before the softmax function is applied.
Subsequently fs, also with a softmax function at its final
layer, is trained to minimise the difference between its own
outputs and those of ft, Ldist( ′ % , ′ & , over each sample xi
in dataset D. As this is no longer a pure classification
problem, distillation loss Ldist can be some distance measuresuch as mean squared error or Kullback-Leibler (KL)divergence [22].
An additional classification loss function, Lcls( ′ % , ' ,may be used to compare fs outputs with ground truth labels,
usually with a lesser weighting than Ldist.
We build upon this with our proposed concept ofUncertainty Distillation, the training process for which is
described in Figure 1. Teacher ft is an ensemble comprising
F trained models. For a given input xi, ′ # is computed foreach model fj, and the mean and standard deviation acrossthe outputs of all models are calculated at each element in
the class probability map to generate mean map Mi andstandard deviation map Σi, each of dimensions C × H × W:

Figure 2. Example Images from the Cityscapes test set with corresponding (left to right) segmentation output of our trained student
model, binary accuracy map (white pixels are incorrectly labelled), uncertainty map output by our student model.

-

(
, ,

, ,

∑+
#

∑,123

.

)*

#

,

′#

, ,

, ,

4

,

−(

(3)
0

, ,

: 0 5; <7 + >5 9< + ?5 @(4)

Ldist for input sample xi is the KL divergence between
prediction head output ′ % and mean map Mi:
A5 B, C

5; <7

∑K×M×ND# ∙ FGH D# IFGH J##

A5O ′ % , ( P,

QR S, T

5 9<

Student model fs is modified such that the output layer isreplaced by a prediction head and an uncertainty head,
which respectively output prediction map ′ % anduncertainty map x*i, for each input image xi. Each headcomprises a 3 × 3 conv-BN-ReLU block with 256 output
channels, followed by a 1 × 1 convolution with C output
channels, and a sigmoid function bounds uncertainty
outputs between 0 and 1, while the softmax function (2) isapplied to prediction outputs. Both outputs are ofdimensions C × H × W, giving a predicted mean andstandard deviation describing a distribution for each classat each pixel.
Three loss functions are combined to optimise fs during
training: Distillation loss LDist, Classification loss Lcls, anduncertainty loss Lunc, with their respective contributions
weighted by parameters α and β, as shown in equation 4.
We determine α and β empirically, with values of 0.1 and
1.1 found to give the best results in our experiments.
5 6789

Lcls is the cross-entropy between prediction head output′ % and ground truth segmentation labels yi:

(5)

YZ [

−log X∑"

∑^ ∑]

YZ1[

#

QR
, ,'
%
_×`
)

\,

(6)

,

Where p is an array of C values denoting the output
probability of each class being present at pixel (v, u), and cis the ground truth label at pixel (v, u). Lunc is computed as
the mean squared error between the uncertainty head output
x*i and standard deviation map Σi:
5 @

∑e

∑d

∑"

∗

, ,

f×^×]

Ib

, ,

c

(7)

4. Experimental Setup
For our experiments, we use a Deeplab v3+ [23], awidely used semantic segmentation model that has
demonstrated state of the art results on many datasets, witha mobilenet [24] backbone, chosen for its compactness andsuitability for real time applications. We use the Cityscapes
dataset [25], a common benchmark for autonomous vehicle
scene understanding tasks, which consists of 2975 training
images and 500 validation images which we use for testing
as the ground truth labels of the official test set are notavailable. Each RGB image has dimensions of 2048 × 1024
and each pixel is assigned one of 19 class labels in thecorresponding ground truth. Ideally we would have 2
separate training sets – one to train ft, and a second unseen
set from which to extract the outputs of ft for training fs –however due to the small number of samples in theCityscapes dataset we perform both steps with the same

training dataset.
Teacher ensemble ft can be an infinite virtual ensemble
generated by sampling from a single Bayesian model,
however in our implementation ft is an ensemble of 25
identically initialised Deeplab models with varianceintroduced via the stochasticity of the training process. Thisconfiguration was chosen due to several unsolved
challenges in the training of Bayesian neural networks andfor the deterministic nature of such an ensemble once
training is completed. In training our ensemble, we use
different learning rates, optimisation algorithms and data
augmentation techniques, to increase the variability
between individual models.
Our student model fs is a single Deeplab model, identicalto those used in ft, and is pretrained to convergence withground truth labels and a standard output layer before thisis replaced with our prediction and uncertainty heads andtraining continues via Uncertainty Distillation. For
comparison, we take a single model from our ensemble toact as a baseline for segmentation performance.

Figure 3. Per Pixel standard deviation predicted by our trained
student model compared with that of the teacher ensemble. Red
points denote pixels that are incorrectly labelled by the student,
blue correctly labelled.

5. Results

student.

5.1. Semantic Segmentation Performance

5.2. Pixelwise Uncertainty Quantification

Table 1 lists the mean intersection over union (IoU) for
each class across the test dataset for a standard Deeplab
model (baseline), our ensemble of 25 such models(teacher), and a model that has been trained via Uncertainty
Distillation (student), as well as the mean uncertainty of
pixels predicted to belong each class as predicted by theteacher and the student. Overall, these results demonstratethat while the student does not match the teacher insegmentation performance, it does surpass the baseline, as
we would expect in a knowledge distillation scenario. In
particular, the distillation process appears to improve
results for some of the harder classes such as Motorbike andTraffic Light. Predicted uncertainty correlates well withperformance across classes, with harder classes
demonstrating greater uncertainty, from both teacher andFigure 2 displays example images from the test dataset
alongside corresponding predicted segmentation, binary
accuracy map of incorrect pixels within predictedsegmentation, and predicted uncertainty. The uncertainty
map U is computed as the mean predicted uncertainty
across all classes at each pixel:
g

∑"

,

∗

h

, ,

(8)

We can see that those pixels assigned an incorrect label(white in the binary accuracy map) generally have a highpredicted uncertainty.
Figure 3 plots pixelwise uncertainty predicted by ourstudent model (y axis) against that computed from theoutputs of our teacher ensemble (x axis) across the test set.

Mean

BicycleMotorbike

Train

Bus

Truck

Car

Rider

PersonSky

Terrain

Vegetation

Tr. Sign

Tr. Light

Pole

Fence

Wall

Building

SidewalkRoad

Baseline
0.976 0.814 0.901 0.415 0.544 0.55 0.482 0.698 0.904 0.569 0.927 0.739 0.44 0.927 0.654 0.686 0.622 0.289 0.706 0.676
IoU
Teacher
0.961 0.794 0.914 0.432 0.563 0.584 0.62 0.73 0.917 0.596 0.937 0.782 0.553 0.935 0.668 0.793 0.677 0.534 0.743 0.723
IoU
Student
0.964 0.772 0.9 0.426 0.547 0.476 0.511 0.652 0.904 0.564 0.919 0.739 0.498 0.921 0.617 0.723 0.625 0.493 0.689 0.681
IoU
Teacher
0.029 0.097 0.055 0.21 0.147 0.1 0.128 0.108 0.028 0.129 0.03 0.068 0.125 0.03 0.176 0.155 0.257 0.165 0.082 0.111
Uncertainty
Student
0.032 0.086 0.077 0.155 0.141 0.133 0.135 0.111 0.055 0.127 0.046 0.097 0.127 0.046 0.108 0.100 0.127 0.135 0.130 0.104
Uncertainty

Table 1. Per-class results - mean intersection over union and mean predicted uncertainty. Red denotes lower IoU, higher uncertainty

(a)

(b)

softmax class probability from the outputs of the baselinemodel. Student predicted uncertainty demonstrates asimilar distribution to the ensemble uncertainty used to train
it. While a minority of correctly labelled pixels are assigned
relatively high uncertainty, the distribution of incorrectly
labelled pixels tails off quite steeply towards 0, meaningthat very few incorrectly labelled pixels are assigned an
uncertainty of below 0.05. With softmax probabilities, thecorrectly labelled histogram is a long-tail distribution withmost pixels near to 1, however the incorrectly labelled
histogram appears almost to resemble a uniform
distribution, with a very slight peak also near to 1,
suggesting softmax is a poor predictor of uncertainty. This
demonstrates that our measure of uncertainty issignificantly more meaningful than softmax predictedprobability.

5.3. Out of Distribution Detection(c)Figure 4. Histograms displaying the frequency of output
uncertainty values for correctly (blue) and incorrectly (red)labelled pixels. (a) shows softmax predicted probabilities of thebaseline model, (b) plots uncertainty of teacher ensemble
outputs, (c) plots uncertainty predictions of our student model.

Each blue point represents a pixel that was assigned thecorrect class label by the student, and each red point
represents a pixel that was assigned an incorrect label. We
can see a clear correlation between prediction and target,
demonstrating that the student has learned to approximate
the uncertainty in the predictions of the ensemble. It canalso be observed that the ratio of red to blue points increases
as both target and predicted value increase, showing thatstandard deviation of ensemble predictions can be a usefulpredictor of output uncertainty.
Figure 4 plots normalised histograms of uncertainty
values computed for test set pixels that are correctly (blue)
and incorrectly labelled (red), taken from ensemble outputs(b) and student model outputs (c). For comparison, (a) plots

We investigate the capability of a model trained via
Uncertainty Distillation to detect out of distribution
samples, that is data that is outside of the distribution of amodels training data. To do this, we test our model, trained
only using the standard Cityscapes training set, using threefurther datasets: Cityscapes Foggy [27] and Cityscapes
Rain [28], which modify the standard Cityscapes images
with synthetic weather effects, and the Audi Autonomous
Driving Dataset (A2D2) [29], which shares a similarlabelling scheme to Cityscapes but features different
scenarios and visual properties.
Table 2 shows the mean class intersection over union of
our student model’s predictions on each of these datasets,
demonstrating a significant drop in performance comparedto the in-distribution Cityscapes test set. Figure 5 shows
examples from each of these datasets. We can see that thebinary accuracy maps show a significantly higher numberof incorrectly labelled pixels in these OOD scenarios, andwhile the uncertainty maps appear to correlate less withpixel accuracy than they do for in-distribution data, overall
uncertainty values are higher.
To detect out of distribution samples, we assign a single
uncertainty value to the whole image with the goal ofsetting a threshold above which an image can be considered
OOD. To assign an image uncertainty value to image xi, we
consider taking the minimum, mean, median and maximum
across all pixel uncertainty values in Ui, with the ROC
curve for each plotted in Figure 6. Of these metrics, medianoffers the best discriminative performance, while maximum
DatasetCityscapes
CS Foggy
CS rain
A2D2

Mean IoU
0.681
0.589
0.348
0.344

Table 2. Segmentation performance over in- and out- of
distribution datasets

Figure 5. Example Images from (top to bottom) CS Foggy, CS Rain and A2D2 test sets, with corresponding (left to right)
segmentation output of our trained student model, binary accuracy map (white pixels are incorrectly labelled), uncertainty map
output by our student model.

gives the opposite of the expected result, with OOD images
generally having a lower maximum pixel uncertainty thanin-distribution images. We hypothesise that this may be
because the model is less confident in its own predicteduncertainty when faced with an OOD image, and so is lesslikely to assign high uncertainty values to noisy orotherwise difficult image regions than when faced with anij; 8@ki @in-distribution image. We find that the formula,i8
in blue on the ROC curve, offers the best discriminative
capability for detecting OOD samples.
Figure 7 plots the distribution of image uncertainty

values, assigned via the above formula, for the four datasets
tested. We can see that CS Rain and A2D2, both of which
demonstrate a significant drop in performance, feature
distributions far from that of the in-distribution data ofCityscapes, while CS Foggy, for which the performancedrop was less severe, has more overlap with Cityscapes.
Overall, this suggests that our image uncertainty value is agood metric for detecting OOD samples.

6. Discussion
In this work we have proposed a novel method fortraining a semantic segmentation model such that it canFigure 6. ROC curves comparing different statistical metrics for
assigning an image uncertainty value. True positives arecorrectly identified OOD samples, false positives are indistribution images incorrectly identified as OOD.

Figure 7. Image uncertainty values computed for all images ineach dataset. Cityscapes is the only dataset considered to be indistribution.

predict the uncertainty of its own predictions, via
distillation from an ensemble model. Our approachdemonstrates several advantages over existing work: The
ensemble training process is relatively straightforward,
albeit time consuming; Uncertainty predictions are made ina single pass with marginal additional computation required
over the baseline model, in contrast to prior techniques that
require computationally expensive sampling; And no outof-distribution data is required during training.
We have demonstrated that predicted pixel level
uncertainty values are a good predictor of incorrectly
labelled pixels, and that an image level uncertainty value
can be computed that is a robust discriminator for detectingout of distribution input samples.
In future work we plan to compare different teacher
models, such as Bayesian Neural Networks from which an
infinite ensemble can be generated, and explore the use ofUncertainty Distillation for other tasks such as depth
estimation and object detection.

References[1] D. J. MacKay, "A practical bayesian framework for
backpropagation networks," Neural Computation, vol. 4, no.
3, pp. 448-472, 1992.
[2] Y. Gal, and Z. Ghahramani, "Dropout as a bayesian
approximation: representing model uncertainty in deeplearning," ICML, pp. 1050-1059, 2016.
[3] J. Walker, C. Doersch, A. Gupta, and M. Hebert, "An
uncertain future: forecasting from static images using
variational autoencoders," ECCV, pp. 835-851, 2016.
[4] B. Lakshminarayanan, A. Pritzel, and C. Blundell, "Simple
and scalable predictive uncertainty estimation using deep
ensembles," ArXiv:1612.01474, 2016.
[5] M. Sensoy, L. Kaplan, and M. Kandemir, "Evidential deeplearning
toquantifyclassification
uncertainty,"
ArXiv:1806.01768, 2018.
[6] C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra,"Weight uncertainty in neural network," ICML, pp. 16131622, 2015.
[7] J. M. Hernández-Lobato, and R. Adams, "Probabilistic
backpropagation for scalable learning of bayesian neuralnetworks," ICML, pp. 1861-1869, 2015.
[8] A. Kendall, V. Badrinarayanan, and R. Cipolla, "Bayesiansegnet: model uncertainty in deep convolutional encoderdecoderarchitectures
for
scene
understanding,"
ArXiv:1511.02680, 2015.
[9] D. P. Kingma, and M. Welling, "Auto-encoding variational
bayes," ArXiv:1312.6114, 2013.
[10] T. Tsiligkaridis, "Information robust dirichlet networks forpredictive uncertainty estimation," ArXiv:1910.04819, 2019.
[11] A. Amini, W. Schwarting, A. Soleimany, and D. Rus, "Deepevidential regression," ArXiv:1910.02600, 2019.
[12] W. Chen, Y. Shen, H. Jin, and W. Wang, "A variational
dirichlet framework for out-of-distribution detection,"
ArXiv:1811.07308, 2018.

[13] C. Buciluǎ, R. Caruana, and A. Niculescu-Mizil, "Modelcompression," ACM Int. Conf. On Knowledge Discovery
And Data Mining, pp. 535-541, 2006.
[14] G. Hinton, O. Vinyals, and J. Dean, "Distilling theknowledge in a neural network," ArXiv:1503.02531, 2015.
[15] A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta,
and Y. Bengio, "Fitnets: hints for thin deep nets,"
ArXiv:1412.6550, 2014.
[16] N. Komodakis, and S. Zagoruyko, "Paying more attention toattention: improving the performance of convolutional neuralnetworks via attention transfer," ICLR, 2017.
[17] X. Wang, R. Zhang, Y. Sun, and J. Qi, "KDGAN: knowledge
distillation with generative adversarial networks.," NeurIPS,
pp. 783-794, 2018.
[18] Y. Liu, K. Chen, C. Liu, Z. Qin, Z. Luo, and J. Wang,"Structured
knowledge
distillationforsemantic
segmentation," CVPR, pp. 2604-2613, 2019.
[19] T. He, C. Shen, Z. Tian, D. Gong, C. Sun, and Y. Yan,
"Knowledge adaptation for efficient semantic segmentation,”
CVPR, pp. 578-587, 2019.
[20] J. Jiao, Y. Wei, Z. Jie, H. Shi, R. W. Lau, and T. S. Huang,"Geometry-aware distillation for indoor semantic
segmentation," in Proc. Of The IEEE/CVF Conf. On
Computer Vision And Pattern Recognition, pp. 2869-2878,
2019.
[21] S. Kullback, and R. A. Leibler, "On information andsufficiency," Annals Of Mathematical Statistics, vol. 22, no.
1, pp. 79-86, 1951.
[22] L. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam,"Encoder-decoder with atrous separable convolution forsemantic image segmentation," ECCV, pp. 801-818, 2018.
[23] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang,
T. Weyand, M. Andreetto, and H. Adam, "Mobilenets:
efficient convolutional neural networks for mobile visionapplications," ArXiv:1704.04861, 2017.
[24] M. Cordts, M. Omran, S. Ramos, T. Scharwächter, M.
Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele,"The cityscapes dataset," in CVPR Workshop On The Future
Of Datasets In Vision, vol. 2, 2015.
[25] D. P. Kingma, and J. Ba, "Adam: a method for stochastic
optimization," ArXiv:1412.6980, 2014.
[26] C. Sakaridis, D. Dai, and L. Van Gool, "Semantic foggy
scene understanding with synthetic data," Int. Journal Of
Computer Vision, vol. 126, no. 9, pp. 973-992, 2018.
[27] X. Hu, C. Fu, L. Zhu, and P. Heng, "Depth-attentionalfeatures for single-image rain removal," CVPR, pp. 80228031, 2019.
[28] J. Geyer, Y. Kassahun, M. Mahmudi, X. Ricou, R. Durgesh,
A. S. Chung, L. Hauswald, V. H. Pham, M. Mühlegg, and S.
Dorn, "A2d2: audi autonomous driving dataset,"
ArXiv:2004.06320, 2020.