Summary:
Semantic segmentation and object detection research have recently achieved
rapid progress. However, the former task has no notion of different instances
of the same object, and the latter operates at a coarse, bounding-box level. We
propose an Instance Segmentation system that produces a segmentation map where
each pixel is assigned an object class and instance identity label. Most
approaches adapt object detectors to produce segments instead of boxes. In
contrast, our method is based on an initial semantic segmentation module, which
feeds into an instance subnetwork. This subnetwork uses the initial
category-level segmentation, along with cues from the output of an object
detector, within an end-to-end CRF to predict instances. This part of our model
is dynamically instantiated to produce a variable number of instances per
image. Our end-to-end approach requires no post-processing and considers the
image holistically, instead of processing independent proposals. Therefore,
unlike some related work, a pixel cannot belong to multiple instances.
Furthermore, far more precise segmentations are achieved, as shown by our
state-of-the-art results (particularly at high IoU thresholds) on the Pascal
VOC and Cityscapes datasets.

CVPR 2017

Pixelwise Instance Segmentation with a Dynamically Instantiated Network
Anurag Arnab and Philip H.S Torr
University of Oxford
{anurag.arnab, philip.torr }@eng.ox.ac.uk
Abstract
Semantic segmentation and object detection research
have recently achieved rapid progress. However, the former
task has no notion of different instances of the same object,
and the latter operates at a coarse, bounding-box level. We
propose an Instance Segmentation system that produces a
segmentation map where each pixel is assigned an object
class and instance identity label. Most approaches adapt
object detectors to produce segments instead of boxes. In
contrast, our method is based on an initial semantic seg-
mentation module, which feeds into an instance subnetwork.
This subnetwork uses the initial category-level segmenta-
tion, along with cues from the output of an object detector,
within an end-to-end CRF to predict instances. This part
of our model is dynamically instantiated to produce a vari-
able number of instances per image. Our end-to-end ap-
proach requires no post-processing and considers the im-
age holistically, instead of processing independent propos-
als. Therefore, unlike some related work, a pixel cannot be-
long to multiple instances. Furthermore, far more precise
segmentations are achieved, as shown by our substantial
improvements at high APrthresholds.
1. Introduction
Semantic segmentation and object detection are well-
studied scene understanding problems, and have recently
witnessed great progress due to deep learning [ 22,13,7].
However, semantic segmentation – which labels every pixel
in an image with its object class – has no notion of different
instances of an object (Fig. 1). Object detection does lo-
calise different object instances, but does so at a very coarse,
bounding-box level. Instance segmentation localises ob-
jects at a pixel level, as shown in Fig. 1, and can be thought
of being at the intersection of these two scene understanding
tasks. Unlike the former, it knows about different instances
of the same object, and unlike the latter, it operates at a pixel
level. Accurate recognition and localisation of objects en-
ables many applications, such as autonomous driving [ 9],
image-editing [ 53] and robotics [ 17].
Many recent approaches to instance segmentation arebased on object detection pipelines where objects are ﬁrst
localised with bounding boxes. Thereafter, each bounding
box is reﬁned into a segmentation [ 19,20,32,37,30]. An-
other related approach [ 12,56] is to use segment-based re-
gion proposals [ 10,41,42] instead of box-based proposals.
However, these methods do not consider the entire image,
but rather independent proposals. As a result, occlusions
between different objects are not handled. Furthermore,
many of these methods cannot easily produce segmentation
maps of the image, as shown in Fig. 1, since they process
numerous proposals independently. There are typically far
more proposals than actual objects in the image, and these
proposals can overlap and be assigned different class labels.
Finally, as these methods are based on an initial detection
step, they cannot recover from false detections.
Our proposed method is inspired by the fact that instance
segmentation can be viewed as a more complex form of se-
mantic segmentation, since we are not only required to la-
bel the object class of each pixel, but also its instance iden-
tity. We produce a pixelwise segmentation of the image,
where each pixel is assigned both a semantic class and in-
stance label. Our end-to-end trained network, which out-
puts a variable number of instances per input image, be-
gins with an initial semantic segmentation module. The
following, dynamic part of the network, then uses infor-
mation from an object detector and a Conditional Random
Field (CRF) model to distinguish different instances. This
approach is robust to false-positive detections, as well as
poorly localised bounding boxes which do not cover the
entire object, in contrast to detection-based methods to in-
stance segmentation. Moreover, as it considers the entire
image when making predictions, it attempts to resolve oc-
clusions between different objects and can produce segmen-
tation maps as in Fig. 1without any post-processing.
Furthermore, we note that the Average Precision (AP)
metric [ 14] used in evaluating object detection systems, and
itsAPrvariant [ 19] used for instance segmentation, consid-
ers individual, potentially overlapping, object predictions in
isolation, as opposed to the entire image. To evaluate meth-
ods such as ours, which produce complete segmentation
maps and reason about occlusions, we also evaluate using
1
arXiv:1704.02386v1  [cs.CV]  7 Apr 2017
(a) Object Detection (b) Semantic Segmentation (c) Instance Segmentation
Figure 1: Object detection (a) localises the different people, but at a coarse, bounding-box level. Semantic segmentation
(b) labels every pixel, but has no notion of instances. Instance segmentation (c) labels each pixel of each person uniquely.
Our proposed method jointly produces both semantic and instance segmentations. Our method uses the output of an object
detector as a cue to identify instances, but is robust to false positive detections, poor bounding box localisation and occlusions.
Best viewed in colour.
the “Matching Intersection over Union” metric.
Our system, which is based on an initial semantic seg-
mentation subnetwork, produces sharp and accurate in-
stance segmentations. This is reﬂected by the substan-
tial improvements we achieve over state-of-the-art methods
at highAPrthresholds on the Pascal VOC and Semantic
Boundaries datasets. Furthermore, our network improves
on the semantic segmentation task while being trained for
the related task of instance segmentation.
2. Related Work
An early work on instance segmentation was by Winn
and Shotton [ 51]. A per-pixel unary classiﬁer was trained
to predict parts of an object. These parts were then encour-
aged to maintain a spatial ordering, that is characteristic of
an instance, using asymmetric pairwise potentials in a Con-
ditional Random Field (CRF). Subsequent work [ 54], pre-
sented another approach where detection outputs of DPM
[15], with associated foreground masks, were assigned a
depth ordering using a generative, probabilistic model. This
depth ordering resolved occlusions.
However, instance segmentation has become more com-
mon after the “Simultaneous Detection and Segmentation”
(SDS) work of Hariharan et al. [19]. This system was based
on the R-CNN pipeline [ 16]: Region proposals, generated
by the method of [ 1], were classiﬁed into object categories
with a Convolutional Neural Network (CNN) before apply-
ing bounding-box regression as post-processing. A class-
speciﬁc segmentation was then performed in this bounding
box to simultaneously detect and segment the object. Nu-
merous works [ 20,8,30] have extended this pipeline. How-
ever, approaches that segment instances by reﬁning detec-
tions [ 19,20,8,11,30] are inherently limited by the qual-
ity of the initial proposals. This problem is exacerbated by
the fact that this pipeline consists of several different mod-
ules trained with different objective functions. Furthermore,numerous post-processing steps such as “superpixel projec-
tion” and rescoring are performed. Dai et al. [12] addressed
some of these issues by designing one end-to-end trained
network that generates box-proposals, creates foreground
masks from these proposals and then classiﬁes these masks.
This network can be seen as an extension of the end-to-end
Faster-RCNN [ 44] detection framework, which generates
box-proposals and classiﬁes them. Additionally, Liu et al.
[37] formulated an end-to-end version of the SDS network
[19], whilst [ 32] iteratively reﬁned object proposals.
On a separate track, algorithms have also been developed
that do not require object detectors. Zhang et al. [57,58]
segmented car instances by predicting the depth ordering
of each pixel in the image. Unlike the previous detection-
based approaches, this method reasoned globally about all
instances in the image simultaneously (rather than individ-
ual proposals) with an MRF-based formulation. However,
inference of this graphical model was not performed end-to-
end as shown to be possible in [ 60,2,5,34]. Furthermore,
although this method does not use object detections, it is
trained with ground truth depth and assumes a maximum
of nine cars in an image. Predicting all the instances in
an image simultaneously (rather than classifying individual
proposals) requires a model to be able to handle a variable
number of output instances per image. As a result, [ 45]
proposed a Recurrent Neural Network (RNN) for this task.
However, this model was only for a single object category.
Our proposed method not only outputs a variable number of
instances, but can also handle multiple object classes.
Liang et al. [33] developed another proposal-free method
based on the semantic segmentation network of [ 6]. The
category-level segmentation, along with CNN features, was
used to predict instance-level bounding boxes. The number
of instances of each class was also predicted to enable a ﬁnal
spectral clustering step. However, this additional informa-
tion predicted by Liang’s network could have been obtained
 Semantic Segmentation subnetwork  Instance Segmentation subnetwork Instance Unary Potentials Pixelwise
CNN Semantic CRF Detector Box
Global
Shape Instance CRF
Input Image Instance Segmentation
Semantic Segmentation End-to-end Network
K+1
D+1
Figure 2: Network overview: Our end-to-end trained network consists of semantic- and instance-segmentation modules. The
intermediate category-level segmentation, along with the outputs of an object detector, are used to reason about instances.
This is done by instance unary terms which use information from the detector’s bounding boxes, the initial semantic seg-
mentation and also the object’s shape. A ﬁnal CRF is used to combine all this information together to obtain an instance
segmentation. The output of the semantic segmentation module is a ﬁxed size W×H×(K+ 1) tensor where Kis the
number of object classes, excluding background, in the dataset. The ﬁnal output, however, is of a variable W×H×(D+ 1)
dimensions where Dis the number of detected objects (and one background label).
from an object detector. Arnab et al. [3] also started with an
initial semantic segmentation network [ 2], and combined
this with the outputs of an object detector using a CRF to
reason about instances. This method was not trained end-
to-end though, and could not really recover from errors in
bounding-box localisation or occlusion.
Our method also has an initial semantic segmentation
subnetwork, and uses the outputs of an object detector.
However, in contrast to [ 3] it is trained end-to-end to im-
prove on both semantic- and instance-segmentation perfor-
mance (to our knowledge, this is the ﬁrst work to achieve
this). Furthermore, it can handle detector localisation er-
rors and occlusions better due to the energy terms in our
end-to-end CRF. In contrast to detection-based approaches
[19,20,12,37], our network requires no additional post-
processing to create an instance segmentation map as in
Fig. 1(c) and reasons about the entire image, rather than
independent proposals. This global reasoning allows our
method to produce more accurate segmentations. Our pro-
posed system also handles a variable number of instances
per image, and thus does not assume a maximum number
of instances like [ 57,58].
3. Proposed Approach
Our network (Fig. 2) contains an initial semantic seg-
mentation module. We use the semantic segmentation re-
sult, along with the outputs of an object detector, to compute
the unary potentials of a Conditional Random Field (CRF)
deﬁned over object instances. We perform mean ﬁeld infer-
ence in this random ﬁeld to obtain the Maximum a Poste-
riori (MAP) estimate, which is our labelling. Although ournetwork consists of two conceptually different parts – a se-
mantic segmentation module, and an instance segmentation
network – the entire pipeline is fully differentiable, given
object detections, and trained end-to-end.
3.1. Semantic Segmentation subnetwork
Semantic Segmentation assigns each pixel in an image a
semantic class label from a given set, L. In our case, this
module uses the FCN8s architecture [ 38] which is based
on the VGG [ 47] ImageNet model. For better segmenta-
tion results, we include mean ﬁeld inference of a Condi-
tional Random Field as the last layer of this module. This
CRF contains the densely-connected pairwise potentials de-
scribed in [ 26] and is formulated as a recurrent neural net-
work as in [ 60]. Additionally, we include the Higher Order
detection potential described in [ 2]. This detection poten-
tial has two primary beneﬁts: Firstly, it improves semantic
segmentation quality by encouraging consistency between
object detections and segmentations. Secondly, it also re-
calibrates detection scores. This detection potential is sim-
ilar to the one previously proposed by [ 28], [48], [52] and
[55], but formulated for the differentiable mean ﬁeld infer-
ence algorithm. We employ this potential as we are already
using object detection information for identifying object in-
stances in the next stage. We denote the output at the se-
mantic segmentation module of our network as the tensor
Q, whereQi(l)denotes the probability (obtained by apply-
ing the softmax function on the network’s activations) of
pixelitaking on the label l∈ L.
(a) Semantic Segmentation (b) Instance Segmentation
Figure 3: Instance segmentation using only the “Box” unary
potential. This potential is effective when we have a good
initial semantic segmentation (a). Occlusions between ob-
jects of the same class can be resolved by the pairwise term
based on appearance differences. Note that we can ignore
the conﬁdent, false-positive “bottle” detections (b). This is
in contrast to methods such as [ 8,19,20,30] which cannot
recover from detection errors.
3.2. Instance Segmentation subnetwork
At the input to our instance segmentation subnetwork,
we assume that we have two inputs available: The semantic
segmentation predictions, Q, for each pixel and label, and a
set of object detections. For each input image, we assume
that there are Dobject detections, and that the ithdetection
is of the form (li,si,Bi)whereli∈ Lis the detected class
label,si∈[0,1]is the conﬁdence score and Biis the set of
indices of the pixels falling within the detector’s bounding
box. Note that the number Dvaries for every input image.
The problem of instance segmentation can then be
thought of as assigning every pixel to either a particular ob-
ject detection, or the background label. This is based on
the assumption that every object detection speciﬁes a po-
tential object instance. We deﬁne a multinomial random
variable,V, at each of the Npixels in the image, and
V= [V1V2...V N]T. Each variable at pixel i,Vi, is as-
signed a label corresponding to its instance. This label set,
{0,1,2,...,D }changes for each image since D, the number
of detections, varies for every image (0 is the background
label). In the case of instance segmentation of images, the
quality of a prediction is invariant to the permutations of the
instance labelling. For example, labelling the “blue person”
in Fig. 1(c) as “1” and the “purple person” as “2” is no dif-
ferent to labelling them as “2” and “1” respectively. This
condition is handled by our loss function in Sec. 3.4.
Note that unlike works such as [ 57] and [ 58] we do not
assume a maximum number of possible instances and keep
a ﬁxed label set. Furthermore, since we are considering ob-
ject detection outputs jointly with semantic segmentation
predictions, we have some robustness to high-scoring false
positive detections unlike methods such as [ 8,20,37] which
reﬁne object detections into segmentations.
We formulate a Conditional Random Field over our in-
stance variables, V, which consists of unary and pairwise
(a) Only Box term (b) Box and Global terms
Figure 4: The “Global” unary potential (b) is particularly
effective in cases where the input detection bounding box
does not cover the entire extent of the object. Methods
which are based on reﬁning bounding-box detections such
as [19,20,8,12] cannot cope with poorly localised detec-
tions. Note, the overlaid detection boxes are an additional
input to our system.
energies. The energy of the assignment vto all the vari-
ables,V, is
E(V=v) =/summationdisplay
iU(vi) +/summationdisplay
i<jP(vi,vj). (1)
The unary energy is a sum of three terms, which take into
account the object detection bounding boxes, the initial se-
mantic segmentation and shape information,
U(vi) =−ln[w1ψBox(vi) +w2ψGlobal (vi)+
w3ψShape (vi)], (2)
and are described further in Sections 3.2.1 through 3.2.3 .
w1,w2andw3are all weighting co-efﬁcients learned via
backpropagation.
3.2.1 Box Term
This potential encourages a pixel to be assigned to the in-
stance corresponding to the kthdetection if it falls within
the detection’s bounding box. This potential is proportional
to the probability of the pixel’s semantic class being equal
to the detected class Qi(lk)and the detection score, sk.
ψBox(Vi=k) =/braceleftBigg
Qi(lk)skifi∈Bk
0 otherwise(3)
As shown in Fig. 3, this potential performs well when the
initial semantic segmentation is good. It is robust to false
positive detections, unlike methods which reﬁne bounding
boxes [ 8,19,20] since the detections are considered in light
of our initial semantic segmentation, Q. Together with the
pairwise term (Sec. 3.2.4 ), occlusions between objects of
the same class can be resolved if there are appearance dif-
ferences in the different instances.
3.2.2 Global Term
This term does not rely on bounding boxes, but only the seg-
mentation prediction at a particular pixel, Qi. It encodes the
intuition that if we only know there are dpossible instances
of a particular object class, and have no further localisa-
tion information, each instance is equally probable, and this
potential is proportional to the semantic segmentation con-
ﬁdence for the detected object class at that pixel:
ψGlobal (Vi=k) =Qi(lk). (4)
As shown in Fig. 4, this potential overcomes cases where
the bounding box does not cover the entire extent of the ob-
ject, as it assigns probability mass to a particular instance
label throughout all pixels in the image. This is also ben-
eﬁcial during training, as it ensures that the ﬁnal output is
dependent on the segmentation prediction at all pixels in the
image, leading to error gradients that are more stable across
batches and thus more amenable to backpropagation.
3.2.3 Shape Term
We also incorporate shape priors to help us reason about oc-
clusions involving multiple objects of the same class, which
may have minimal appearance variation between them, as
shown in Fig. 5. In such cases, a prior on the expected shape
of an object category can help us to identify the foreground
instance within a bounding box. Previous approaches to in-
corporating shape priors in segmentation [ 23,8,50] have
involved generating “shape exemplars” from the training
dataset and, at inference time, matching these exemplars to
object proposals using the Chamfer distance [ 46,36].
We propose a fully differentiable method: Given a set of
shape templates, T, we warp each shape template using bi-
linear interpolation into ˜Tso that it matches the dimensions
of thekthbounding box, Bk. We then select the shape prior
which matches the segmentation prediction for the detected
class within the bounding box, QBk(lk), the best accord-
ing to the normalised cross correlation. Our shape prior is
then the Hadamard (elementwise) product ( ⊙) between the
segmentation unaries and the matched shape prior:
t∗= arg max
t∈˜T/summationtextQBk(lk)⊙t/vextenddouble/vextenddoubleQBk(lk)/vextenddouble/vextenddouble/bardblt/bardbl(5)
ψ(VBk=k) =QBk(lk)⊙t∗. (6)
Equations 5and6can be seen as a special case of max-
pooling, and the numerator of Eq. 5is simply a convolution
that produces a scalar output since the two arguments are of
equal dimension. Additionally, during training, we can con-
sider the shape priors Tas parameters of our “shape term”
layer and backpropagate through to the matched exemplar
t∗to update it. In practice, we initialised these parameters
(a) Without shape term (b) With Shape term
Figure 5: The “Shape” unary potential (b) helps us to distin-
guish between the green and purple sheep, which the other
two unary potentials cannot. Input detections are overlaid
on the images.
with the shape priors described in [ 50]. This consists of
roughly 250 shape templates for each of ﬁve different as-
pect ratios. These were obtained by clustering foreground
masks of object instances from the training set.
Here, we have only matched a single shape template to
a proposed instance. This method could be extended in fu-
ture to matching multiple templates to an instance, in which
case each shape exemplar would correspond to a part of the
object such as in DPM [ 15].
3.2.4 Pairwise term
The pairwise term consists of densely-connected Gaussian
potentials [ 26] and encourages appearance and spatial con-
sistency. The weights governing the importance of these
terms are also learnt via backpropagation, as in [ 60]. We
ﬁnd that these priors are useful in the case of instance seg-
mentation as well, since nearby pixels that have similar ap-
pearance often belong to the same object instance. They are
often able to resolve occlusions based on appearance differ-
ences between objects of the same class (Fig. 3).
3.3. Inference of our Dynamic Instance CRF
We use mean ﬁeld inference to approximately minimise
the Gibbs Energy in Eq. 1which corresponds to ﬁnding the
Maximum a Posteriori (MAP) labelling of the correspond-
ing probability distribution, P(V=v) =1
Zexp (−E(v))
whereZis the normalisation factor. Mean ﬁeld inference is
differentiable, and this iterative algorithm can be unrolled
and seen as a recurrent neural network [ 60]. Following this
approach, we can incorporate mean ﬁeld inference of a CRF
as a layer of our neural network. This enables us to train our
entire instance segmentation network end-to-end.
Because we deal with a variable number of instances for
every image, our CRF needs to be dynamically instantiated
to have a different number of labels for every image, as ob-
served in [ 3]. Therefore, unlike [ 60], none of our weights
are class-speciﬁc. This weight-sharing not only allows us to
deal with variable length inputs, but class-speciﬁc weights
(a) Original
ground truth, G(b) Prediction, P (c) “Matched”
ground truth, G∗
Figure 6: Due to the problem of label permutations, we
“match” the ground truth with our prediction before com-
puting the loss when training.
also do not make sense in the case of instance segmentation
since a class label has no particular semantic meaning.
3.4. Loss Function
When training for instance segmentation, we have a
single loss function which we backpropagate through our
instance- and semantic-segmentation modules to update all
the parameters. As discussed previously, we need to deal
with different permutations of our ﬁnal labelling which
could have the same ﬁnal result. The works of [ 57] and [ 58]
order instances by depth to break this symmetry. However,
this requires ground-truth depth maps during training which
we do not assume that we have. Proposal-based methods
[12,19,20,37] do not have this issue since they consider a
single proposal at a time, rather than the entire image. Our
approach is similar to [ 45] in that we match the original
ground truth to our instance segmentation prediction based
on the Intersection over Union (IoU) [ 14] of each instance
prediction and ground truth, as shown in Fig. 6.
More formally, we denote the ground-truth labelling of
an image, G, to be a set of rsegments, {g1,g2,...,g r},
where each segment (set of pixels) is an object instance
and has an associated semantic class label. Our predic-
tion, which is the output of our network, P, is a set of s
segments, {p1,p2,...,p s}, also where each segment corre-
sponds to an instance label and also has an associated class
label. Note that randsmay be different since we may pre-
dict greater or fewer instances than actually present. Let
Mdenote the set of all permutations of the ground-truth,
G. As can be seen in Fig. 6, different permutations of the
ground-truth correspond to the same qualitative result. We
deﬁne the “matched” ground-truth, G∗, as the permutation
of the original ground-truth labelling which maximises the
IoU between the prediction, P, and ground truth:
G∗= arg max
m∈MIoU(m,P). (7)
Once we have the “matched” ground truth, G∗, (Fig. 6) foran image, we can apply any loss function to train our net-
work for segmentation. In our case, we use the common
cross-entropy loss function. We found that this performed
better than the approximate IoU loss proposed in [ 27,45].
Crucially, we do not need to evaluate all permutations
of the ground truth to compute Eq. 7, since it can be for-
mulated as a maximum-weight bipartite matching problem.
The edges in our bipartite graph connect ground-truth and
predicted segments. The edge weights are given by the
IoU between the ground truth and predicted segments if
they share the same semantic class label, and zero other-
wise. Leftover segments are matched to “dummy” nodes
with zero overlap.
Additionally, the ordering of the instances in our network
are actually determined by the object detector, which re-
mains static during training. As a result, the ordering of
our predictions does not ﬂuctuate much during training – it
only changes in cases where there are multiple detections
overlapping an object.
3.5. Network Training
We ﬁrst train a network for semantic segmentation with
the standard cross-entropy loss. In our case, this network is
FCN8s [ 38] with a CRF whose inference is unrolled as an
RNN and trained end-to-end, as described in [ 60] and [ 2].
To this pretrained network, we append our instance segmen-
tation subnetwork, and ﬁnetune with instance segmentation
annotations and only the loss detailed in Sec. 3.4. For the
semantic segmentation subnetwork, we train with an initial
learning rate of 10−8, momentum of 0.9 and batch size of
20. The learning rate is low since we do not normalise the
loss by the number of pixels. This is so that images with
more pixels contribute a higher loss. The normalised learn-
ing rate is approximately 2×10−3. When training our in-
stance segmentation network as well, we lower the learning
rate to 10−12and use a batch size of 1 instead. Decreas-
ing the batch size gave empirically better results. We also
clipped gradients (a technique common in training RNNs
[40]) withℓ2norms above 109. This threshold was set by
observing “normal” gradient magnitudes during training.
The relatively high magnitude is due to the fact that our
loss is not normalised. In our complete network, we have
two CRF inference modules which are RNNs (one each in
the semantic- and instance-segmentation subnetworks), and
gradient clipping facilitated successful training.
3.6. Discussion
Our network is able to compute a semantic and instance
segmentation of the input image in a single forward pass.
We do not require any post-processing, such as the patch
aggregation of [ 37], “mask-voting” of [ 12], “superpixel
projection” of [ 19,20,30] or spectral clustering of [ 33].
The fact that we compute an initial semantic segmentation
means that we have some robustness to errors in the ob-
ject detector (Fig. 3). Furthermore, we are not necessarily
limited by poorly localised object detections either (Fig. 4).
Our CRF model allows us to reason about the entire image
at a time, rather than consider independent object proposals,
as done in [ 19,20,12,37,30]. Although we do not train our
object detector jointly with the network, it also means that
our segmentation network and object detector do not suc-
cumb to the same failure cases. Moreover, it ensures that
our instance labelling does not “switch” often during train-
ing, which makes learning more stable. Finally, note that
although we perform mean ﬁeld inference of a CRF within
our network, we do not optimise the CRF’s likelihood, but
rather a cross-entropy loss (Sec 3.4).
4. Experimental Evaluation
Sections 4.1to4.6describe our evaluation on the Pas-
cal VOC Validation Set [ 14] and the Semantic Boundaries
Dataset (SBD) [ 18] (which provides per-pixel annotations
to 11355 previously unlaballed images from Pascal VOC).
Section 4.7details results on Cityscapes [ 9].
4.1. Experimental Details
We ﬁrst train a network for semantic segmentation, ther-
after we ﬁnetune it to the task of instance segmentation,
as described in Sec. 3.5. Our training data for the seman-
tic segmentation pretraining consists of images from Pas-
cal VOC [ 14], SBD [ 18] and Microsoft COCO [ 35]. Fi-
nally, when ﬁnetuning for instance segmentation, we use
only training data from either the VOC dataset, or from the
SBD dataset. We train separate models for evaluating on
the VOC Validation Set, and the SBD Validation Set. In
each case, we remove validation set images from the initial
semantic segmentation pretraining set. We use the publicly
available R-FCN object detection framework [ 13], and en-
sure that the images used to train the detector do not fall
into our test sets for instance segmentation.
4.2. Evaluation Metrics
We report the mean Average Precision over regions
(APr) as deﬁned by [ 19]. The difference between APrand
the AP metric used in object detection [ 14] is that the Inter-
section over Union (IoU) is computed over predicted and
ground-truth regions instead of bounding boxes. Further-
more, the standard AP metric uses an IoU threshold of 0.5
to determine whether a prediction is correct or not. Here,
we use a variety of IoU thresholds since larger thresholds
require more precise segmentations. Additionally, we re-
port theAPr
volwhich is the average of the APrfor 9 IoU
thresholds ranging from 0.1 to 0.9 in increments of 0.1.
However, we also observe that the APrmetric requires
an algorithm to produce a ranked list of segments and their
object class. It does not require, nor evaluate, the ability ofTable 1: The effect of the different CRF unary potentials,
and end-to-end training with them, on the VOC 2012 Vali-
dation Set.
APr
APr
volmatch
IoU 0.5 0.7 0.9
Box Term
(piecewise)60.0 47.3 21.2 54.9 42.6
Box+Global
(piecewise)59.1 46.1 23.4 54.6 43.0
Box+Global+Shape
(piecewise)59.5 46.4 23.3 55.2 44.8
Box Term
(end-to-end)60.7 47.4 24.6 56.2 46.9
Box+Global
(end-to-end)60.9 48.1 25.5 56.7 47.1
Box+Global+Shape
(end-to-end)61.7 48.6 25.1 57.5 48.3
an algorithm to produce a globally coherent segmentation
map of the image, for example Fig. 1c. To measure this, we
propose the “Matching IoU” which matches the predicted
image and ground truth, and then calculates the correspond-
ing IoU as deﬁned in [ 14]. This matching procedure is the
same as described in Sec. 3.4. This measure was originally
proposed in [ 54], but has not been used since in evaluating
instance segmentation systems.
4.3. Effect of Instance Potentials and End-to-End
training
We ﬁrst perform ablation studies on the VOC 2012 Val-
idation set. This dataset, consisting of 1464 training and
1449 validation images has very high-quality annotations
with detailed object delineations which makes it the most
suited for evaluating pixel-level segmentations.
In Tab. 1, we examine the effect of each of our unary po-
tentials in our Instance subnetwork on overall performance.
Furthermore, we examine the effect of end-to-end training
the entire network as opposed to piecewise training. Piece-
wise training refers to freezing the pretrained semantic seg-
mentation subnetwork’s weights and only optimising the in-
stance segmentation subnetwork’s parameters. Note that
when training with only the “Box” (Eq. 3) unary poten-
tial and pairwise term, we also have to add in an additional
“Background” detection which encompasses the entire im-
age. Otherwise, we cannot classify the background label.
We can see that each unary potential improves overall in-
stance segmentation results, both in terms of APr
voland the
Matching IoU. The “Global” term (Eq. 4) shows particular
improvement over the “Box” term at the high APrthresh-
old of 0.9. This is because it can overcome errors in bound-
ing box localisation (Fig. 4) and leverage our semantic seg-
mentation network’s accurate predictions to produce precise
Table 2: Comparison of Instance Segmentation perfor-
mance to recent methods on the VOC 2012 Validation Set
MethodAPr
APr
vol0.5 0.6 0.7 0.8 0.9
SDS [ 19] 43.8 34.5 21.3 8.7 0.9 –
Chen et al. [8] 46.3 38.2 27.0 13.5 2.6 –
PFN [ 33] 58.7 51.3 42.5 31.2 15.7 52.3
Arnab et al. [3] 58.3 52.4 45.4 34.9 20.1 53.1
MPA 1-scale [ 37] 60.3 54.6 45.9 34.3 17.3 54.5
MPA 3-scale [ 37]62.1 56.6 47.4 36.1 18.5 56.5
Ours 61.7 55.5 48.6 39.5 25.1 57.5
labellings. The “Shape” term’s improvement in the APr
volis
primarily due to an improvement in the APrat low thresh-
olds. By using shape priors, we are able to recover instances
which were occluded and missed out. End-to-end training
also improves results at all APrthresholds. Training with
just the “Box” term shows a modest improvement in the
APr
volof 1.3%. Training with the “Global” and “Shape”
terms shows larger improvements of 2.1% and 2.3% respec-
tively. This may be because the “Box” term only considers
the semantic segmentation at parts of the image covered by
object detections. Once we include the “Global” term, we
consider the semantic segmentation over the entire image
for the detected class. Training makes more efﬁcient use of
images, and error gradients are more stable in this case.
4.4. Results on VOC Validation Set
We then compare our best instance segmentation model
to recent methods on the VOC Validation Set in Tab. 2. The
fact that our algorithm achieves the highest APrat thresh-
olds above 0.7 indicates that our method produces more de-
tailed and accurate segmentations.
At an IoU threshold of 0.9, our improvement over the
previous state-of-the-art (MPA [ 37]) is 6.6%, which is a
relative improvement of 36%. Unlike [ 37,19,8], our
network performs an initial semantic segmentation which
may explain our more accurate segmentations. Other
segmentation-based approaches, [ 3,33] are not fully end-
to-end trained. We also achieve the best APr
volof 57.5%.
The relatively small difference in APr
volto MPA [ 37] de-
spite large improvements at high IoU thresholds indicates
that MPA performs better at low IoU thresholds. Proposal-
based methods, such as [ 37,19] are more likely to perform
better at low IoU thresholds since they output more propos-
als than actual instances in an image (SDS evaluates 2000
proposals per image). Furthermore, note that whilst MPA
takes 8.7s to process an image [ 37], our method requires ap-
proximately 1.5s on the same Titan X GPU. More detailed
qualitative and quantitative results, including success and
failure cases, are included in the supplementary material.Table 3: Comparison of Instance Segmentation perfor-
mance on the SBD Dataset
MethodAPr
APr
volmatch
0.5 0.7 IoU
SDS [ 19] 49.7 25.3 41.4 –
MPA 1-scale [ 37] 55.5 – 48.3 –
Hypercolumn [ 20] 56.5 37.0 – –
IIS [30] 60.1 38.7 – –
CFM [ 11] 60.7 39.6 – –
Hypercolumn rescore [ 20] 60.0 40.4 – –
MPA 3-scale rescore [ 37] 61.8 – 52.0 –
MNC [ 12] 63.5 41.5 – 39.0
MNC, Instance FCN [ 10] 61.5 43.0 – –
IIS sp. projection, rescore [ 30]63.6 43.3 – –
Ours (piecewise) 59.1 42.1 52.3 41.8
Ours (end-to-end) 62.0 44.8 55.4 47.3
4.5. Results on SBD Dataset
We also evaluate our model on the SBD dataset, which
consists of 5623 training and 5732 validation images, as
shown in Tab. 3. Following other works, we only report
APrresults at IoU thresholds of 0.5 and 0.7. However, we
provide more detailed results in our supplementary mate-
rial. Once again, we show signiﬁcant improvements over
other work at high APrthresholds. Here, our APrat 0.7
improves by 1.5% over the previous state-of-the-art [ 30].
Note that [ 30,37,20] perform additional post-processing
where their results are rescored using an additional object
detector. In contrast, our results are obtained by a single
forward pass through our network. We have also improved
substantially on the APr
volmeasure (3.4%) compared to
other works which have reported it. We also used the pub-
licly available source code1, model and default parameters
of MNC [ 12] to evaluate the “Matching IoU”. Our method
improves this by 8.3%. This metric is a stricter measure of
segmentation performance, and our method, which is based
on an initial semantic segmentation and includes a CRF as
part of training therefore performs better.
4.6. Improvement in Semantic Segmentation
Finetuning our network for instance segmentation, with
the loss described in Sec. 3.4improves semantic segmen-
tation performance on both the VOC and SBD dataset, as
shown in Tab. 4. The improvement is 0.9% on VOC, and
1% on SBD. The tasks of instance segmentation and se-
mantic segmentation are highly related – in fact, instance
segmentation can be thought of as a more speciﬁc case of
semantic segmentation. As a result, ﬁnetuning for one task
improves the other.
1https://github.com/daijifeng001/MNC
Table 4: Semantic Segmentation performance before and
after ﬁnetuning for Instance Segmentation on the VOC and
SBD Validation Sets
DatasetMean IoU [%] before
Instance ﬁnetuningMean IoU [%] after
Instance ﬁnetuning
VOC 74.2 75.1
SBD 71.5 72.5
4.7. Results on Cityscapes
Finally, we evaluate our algorithm on the Cityscapes
road-scene understanding dataset [ 9]. This dataset consists
of 2975 training images, and the held-out test set consisting
of 1525 images are evaluated on an online server. None of
the 500 validation images were used for training. We use
an initial semantic segmentation subnetwork that is based
on the ResNet-101 architecture [ 59], and all of the instance
unary potentials described in Sec. 3.2.
As shown in Tab. 5, our method sets a new state-of-the-
art on Cityscapes, surpassing concurrent work [ 21] and the
best previous published work [ 49] by signiﬁcant margins.
Table 5: Results on Cityscapes Test Set. Evaluation met-
rics and results of competing methods obtained from the
online server. The “AP” metric of Cityscapes is similar to
ourAPr
volmetric.
Method AP AP at 0.5 AP 100m AP 50m
Ours 20.0 38.8 32.6 37.6
SAIS [ 21] 17.4 36.7 29.3 34.0
DWT [ 4] 15.6 30.0 26.2 31.8
InstanceCut [ 24] 13.0 27.9 22.1 26.1
Graph Decomp. [ 29] 9.8 23.2 16.8 20.3
RecAttend [ 43] 9.5 18.9 16.8 20.9
Pixel Encoding [ 49] 8.9 21.1 15.3 16.7
R-CNN [ 9] 4.6 12.9 7.7 10.3
5. Conclusion and Future Work
We have presented an end-to-end instance segmentation
approach that produces intermediate semantic segmenta-
tions, and shown that ﬁnetuning for instance segmentation
improves our network’s semantic segmentations. Our ap-
proach differs from other methods which derive their archi-
tectures from object detection networks [ 12,37,20] in that
our approach is more similar to a semantic segmentation
network. As a result, our system produces more accurate
and detailed segmentations as shown by our substantial im-
provements at high APrthresholds. Moreover, our system
produces segmentation maps naturally, and in contrast to
other published work, does not require any post-processing.
Finally, our network produces a variable number of outputs,depending on the number of instances in the image. Our fu-
ture work is to incorporate an object detector into the end-
to-end training of our system to create a network that per-
forms semantic segmentation, object detection and instance
segmentation jointly. Possible techniques for doing this are
suggested by [ 25] and [ 39].
Acknowledgements We thank Bernardino Romera-
Paredes and Stuart Golodetz for insightful discussions
and feedback. This work was supported by the EPSRC,
Clarendon Fund, ERC grant ERC-2012-AdG 321162-
HELIOS, EPRSRC grant Seebibyte EP/M013774/1 and
EPSRC/MURI grant EP/N019474/1.
References
[1] P. Arbelaez, J. Pont-Tuset, J. Barron, F. Marques, and J. Ma-
lik. Multiscale combinatorial grouping. In CVPR , pages
328–335. IEEE, 2014. 2
[2] A. Arnab, S. Jayasumana, S. Zheng, and P. H. S. Torr. Higher
order conditional random ﬁelds in deep neural networks. In
ECCV , 2016. 2,3,6
[3] A. Arnab and P. H. S. Torr. Bottom-up instance segmentation
with deep higher order crfs. In BMVC , 2016. 3,5,8,20
[4] M. Bai and R. Urtasun. Deep watershed transform for in-
stance segmentation. In arXiv preprint arXiv:1611.08303 ,
2016. 9
[5] L. Chen, A. Schwing, A. Yuille, and R. Urtasun. Learning
deep structured models. In ICML , Lille, France, 2015. 2
[6] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and
A. L. Yuille. Semantic image segmentation with deep con-
volutional nets and fully connected crfs. ICLR , 2015. 2
[7] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and
A. L. Yuille. Deeplab: Semantic image segmentation with
deep convolutional nets, atrous convolution, and fully con-
nected crfs. arXiv preprint arXiv:1606.00915 , 2016. 1
[8] Y.-T. Chen, X. Liu, and M.-H. Yang. Multi-instance ob-
ject segmentation with occlusion handling. In CVPR , pages
3470–3478, 2015. 2,4,5,8,20
[9] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler,
R. Benenson, U. Franke, S. Roth, and B. Schiele. The
cityscapes dataset for semantic urban scene understanding.
InCVPR , 2016. 1,7,9
[10] J. Dai, K. He, Y. Li, S. Ren, and J. Sun. Instance-sensitive
fully convolutional networks. In ECCV , 2016. 1,8
[11] J. Dai, K. He, and J. Sun. Convolutional feature masking for
joint object and stuff segmentation. In CVPR , 2015. 2,8
[12] J. Dai, K. He, and J. Sun. Instance-aware semantic segmen-
tation via multi-task network cascades. In CVPR , 2016. 1,2,
3,4,6,7,8,9,11,14,15
[13] J. Dai, Y. Li, K. He, and J. Sun. R-fcn: Object detection via
region-based fully convolutional networks. In NIPS , 2016.
1,7
[14] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and
A. Zisserman. The pascal visual object classes (voc) chal-
lenge. IJCV , 2010. 1,6,7
[15] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ra-
manan. Object detection with discriminatively trained part-
based models. PAMI , 32(9):1627–1645, 2010. 2,5
[16] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich fea-
ture hierarchies for accurate object detection and semantic
segmentation. In CVPR , 2014. 2
[17] S. Gupta, R. Girshick, P. Arbelaez, and J. Malik. Learning
rich features from RGB-D images for object detection and
segmentation. In ECCV . 2014. 1
[18] B. Hariharan, P. Arbel ´aez, L. Bourdev, S. Maji, and J. Malik.
Semantic contours from inverse detectors. In ICCV , pages
991–998. IEEE, 2011. 7
[19] B. Hariharan, P. Arbel ´aez, R. Girshick, and J. Malik. Simul-
taneous detection and segmentation. In ECCV , pages 297–
312. Springer, 2014. 1,2,3,4,6,7,8,20
[20] B. Hariharan, P. Arbel ´aez, R. Girshick, and J. Malik. Hyper-
columns for object segmentation and ﬁne-grained localiza-
tion. CVPR , 2015. 1,2,3,4,6,7,8,9
[21] Z. Hayder, X. He, and M. Salzmann. Shape-aware instance
segmentation. In arXiv preprint arXiv:1612.03129 , 2016. 9
[22] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning
for image recognition. In CVPR , 2016. 1
[23] X. He and S. Gould. An Exemplar-based CRF for Multi-
instance Object Segmentation. In CVPR , 2014. 5
[24] A. Kirillov, E. Levinkov, B. Andres, B. Savchynskyy, and
C. Rother. Instancecut: from edges to instances with multi-
cut. In arXiv preprint arXiv:1611.08272 , 2016. 9
[25] I. Kokkinos. Ubernet: Training a universal convolutional
neural network for low-, mid-, and high-level vision using
diverse datasets and limited memory. In CVPR , 2017. 9
[26] P. Kr ¨ahenb ¨uhl and V. Koltun. Efﬁcient inference in fully
connected CRFs with Gaussian edge potentials. In NIPS ,
2011. 3,5
[27] P. Kr ¨ahenb ¨uhl and V. Koltun. Parameter learning and con-
vergent inference for dense random ﬁelds. In ICML , 2013.
6
[28] L. Ladick ´y, P. Sturgess, K. Alahari, C. Russell, and P. H. S.
Torr. What, where and how many? combining object detec-
tors and crfs. In ECCV , pages 424–437, 2010. 3
[29] E. Levinkov, S. Tang, E. Insafutdinov, and B. Andres. Joint
graph decomposition and node labeling by local search.
arXiv preprint arXiv:1611.04399 , 2016. 9
[30] K. Li, B. Hariharan, and J. Malik. Iterative Instance Segmen-
tation. In CVPR , 2016. 1,2,4,6,7,8,21
[31] Y. Li, H. Qi, J. Dai, X. Ji, and Y. Wei. Fully convolutional
instance-aware semantic segmentation. In CVPR , 2017. 11,
16
[32] X. Liang, Y. Wei, X. Shen, Z. Jie, J. Feng, L. Lin, and S. Yan.
Reversible recursive instance-level object segmentation. In
CVPR , 2016. 1,2
[33] X. Liang, Y. Wei, X. Shen, J. Yang, L. Lin, and S. Yan.
Proposal-free network for instance-level object segmenta-
tion. arXiv preprint arXiv:1509.02636 , 2015. 2,6,8,18,
20
[34] G. Lin, C. Shen, and I. Reid. Efﬁcient piecewise training of
deep structured models for semantic segmentation. In CVPR ,
2016. 2[35] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ra-
manan, P. Doll ´ar, and C. L. Zitnick. Microsoft coco: Com-
mon objects in context. In ECCV , pages 740–755. Springer,
2014. 7,16
[36] M.-Y. Liu, O. Tuzel, A. Veeraraghavan, and R. Chellappa.
Fast directional chamfer matching. In CVPR , pages 1696–
1703. IEEE, 2010. 5
[37] S. Liu, X. Qi, J. Shi, H. Zhang, and J. Jia. Multi-scale patch
aggregation (mpa) for simultaneous detection and segmenta-
tion. In CVPR , 2016. 1,2,3,4,6,7,8,9,18,20
[38] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional
networks for semantic segmentation. In CVPR , 2015. 3,6
[39] I. Misra, A. Shrivastava, A. Gupta, and M. Hebert. Cross-
stitch Networks for Multi-task Learning. In CVPR , 2016. 9
[40] R. Pascanu, T. Mikolov, and Y. Bengio. On the difﬁculty of
training recurrent neural networks. In ICML , pages 1310–
1318, 2013. 6
[41] P. O. Pinheiro, R. Collobert, and P. Dollar. Learning to seg-
ment object candidates. In NIPS , 2015. 1
[42] P. O. Pinheiro, T.-Y. Lin, R. Collobert, and P. Dollar. Learn-
ing to reﬁne object segments. In ECCV , 2016. 1
[43] M. Ren and R. S. Zemel. End-to-end instance segmenta-
tion and counting with recurrent attention. In arXiv preprint
arXiv:1605.09410 , 2016. 9
[44] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: To-
wards real-time object detection with region proposal net-
works. In NIPS , 2015. 2
[45] B. Romera-Paredes and P. H. Torr. Recurrent instance seg-
mentation. In ECCV , 2016. 2,6
[46] J. Shotton, A. Blake, and R. Cipolla. Contour-based learning
for object detection. In ICCV , volume 1, pages 503–510.
IEEE, 2005. 5
[47] K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. In ICLR , 2015.
3
[48] M. Sun, B.-s. Kim, P. Kohli, and S. Savarese. Relating things
and stuff via object property interactions. PAMI , 36(7):1370–
1383, 2014. 3
[49] J. Uhrig, M. Cordts, U. Franke, and T. Brox. Pixel-level en-
coding and depth layering for instance-level semantic label-
ing. In German Conference on Pattern Recognition (GCPR) ,
2016. 9
[50] D. Weiss and B. Taskar. Scalpel: Segmentation cascades
with localized priors and efﬁcient learning. In CVPR , pages
2035–2042, 2013. 5
[51] J. Winn and J. Shotton. The layout consistent random ﬁeld
for recognizing and segmenting partially occluded objects.
InCVPR , 2006. 2
[52] C. Wojek and B. Schiele. A dynamic conditional random
ﬁeld model for joint labeling of object and scene classes. In
ECCV , pages 733–747. Springer, 2008. 3
[53] N. Xu, B. Price, S. Cohen, J. Yang, and T. Huang. Deep
interactive object selection. In CVPR , 2016. 1
[54] Y. Yang, S. Hallman, D. Ramanan, and C. C. Fowlkes. Lay-
ered object models for image segmentation. PAMI , 2012. 2,
7
[55] J. Yao, S. Fidler, and R. Urtasun. Describing the Scene as
a Whole: Joint Object Detection, Scene Classiﬁcation and
Semantic Segmentation. In CVPR , pages 702–709, 2012. 3
[56] S. Zagoruyko, A. Lerer, T.-Y. Lin, P. O. Pinheiro, S. Gross,
S. Chintala, and P. Doll ´ar. A multipath network for object
detection. In BMVC , 2016. 1
[57] Z. Zhang, S. Fidler, and R. Urtasun. Instance-level segmen-
tation for autonomous driving with deep densely connected
mrfs. In CVPR , 2016. 2,3,4,6
[58] Z. Zhang, A. G. Schwing, S. Fidler, and R. Urtasun. Monoc-
ular object instance segmentation and depth ordering with
cnns. In ICCV , pages 2614–2622, 2015. 2,3,4,6
[59] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid scene
parsing network. In arXiv preprint arXiv:1612.01105 , 2017.
9
[60] S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet,
Z. Su, D. Du, C. Huang, and P. Torr. Conditional random
ﬁelds as recurrent neural networks. In ICCV , 2015. 2,3,5,6
Appendix
In this supplementary material, we include more detailed
qualitative and quantitative results on the VOC and SBD
datasets. Furthermore, we also show the runtime of our al-
gorithm.
Figures 7and8show success and failure cases of our
algorithm. Figure 9compares the results of our algorithm
to the publicly available model for MNC [ 12]. Figure 10
compares our results to those of FCIS [ 31], concurrent work
which won the COCO 2016 challenge. Figure 11presents
some qualitative results on the Cityscapes dataset.
Section Ashows more detailed results on the VOC
dataset. Figure 12shows a visualisation of our results at
differentAPrthresholds, and Tables 7to9show per-class
APrresults at thresholds of 0.5, 0.7 and 0.9.
Section Bshows more detailed results on the SBD
dataset. Table 6shows our mean APrresults at thresh-
olds from 0.5 to 0.9, whilst Tables 10and11show per-class
APrresults at thresholds of 0.7 and 0.5 respectively.
Input image Semantic Segmentation Instance Segmentation Ground truth
Figure 7: Success cases of our method. First and second row: Our algorithm can leverage good initial semantic segmen-
tations, and detections, to produce an instance segmentation. Third row: Notice that we have ignored three false-positive
detections. Additionally, the red bounding box does not completely encompass the person, but our algorithm is still able to
associate pixels “outside-the-box” with the correct detection (also applies to row 2). Fourth row: Our system is able to deal
with the heavily occluded sheep, and ignore the false-positive detection. Fifth row: We have not been able to identify one
bicycle on the left since it was not detected, but otherwise have performed well. Sixth row: Although subjective, the train has
not been annotated in the dataset, but both our initial semantic segmentation and object detection networks have identiﬁed it.
Note that the ﬁrst three images are from the VOC dataset, and the last three from SBD. Annotations in the VOC dataset are
more detailed, and also make more use of the grey “ignore” label to indicate uncertain areas in the image. The ﬁrst column
shows the input image, and the results of our object detector which are another input to our network. Best viewed in colour.
Input image Semantic Segmentation Instance Segmentation Ground truth
Figure 8: Failure cases of our method. First row: Both our initial detector, and semantic segmentation system did not
identify a car in the background. Additionally, the “brown” person prediction actually consists of two people that have been
merged together. This is because the detector did not ﬁnd the background person. Second row: Our initial semantic segmen-
tation identiﬁed the table, but it is not there in the Instance Segmentation. This is because there was no “table detection” to
associate these pixels with. Using heuristics, we could propose additional detections in cases like these. However, we have
not done this in our work. Third row: A difﬁcult case where we have segmented most of the people. However, sometimes
two people instances are joined together as one person instance. This problem is because we do not have a detection for
each person in the image. Fourth row: Due to our initial semantic segmentation, we have not been able to segment the green
person and table correctly. Fifth row: We have failed to segment a bird although it was detected. Sixth row: The occluding
cows, which all appear similar, pose a challenge, even with our shape priors. The ﬁrst column shows the input image, and
the results of our object detector which are another input to our network. Best viewed in colour.
Input image MNC [ 12] Ours Ground truth
Figure 9: Comparison to MNC [ 12]The above examples emphasise the advantages in our method over MNC [ 12]. Unlike
proposal-based approaches such as MNC, our method can handle false-positive detections, poor bounding box localisation,
reasons globally about the image and also produces more precise segmentations due to the initial semantic segmentation
module which includes a differentiable CRF. Row 1 shows a case where MNC, which scores segment-based proposals, is
fooled by a false-positive detection and segments an imaginary human (yellow segment). Our method is robust to false-
positive detections due to the initial semantic segmentation module which does not have the same failure modes as the
detector. Rows 2, 3 and 4 show how MNC [ 12] cannot deal with poorly localised bounding boxes. The horizontal boundaries
of the red person in Row 2, and light-blue person in Row 4 correspond to the limits of the proposal processed by MNC. Our
method, in contrast, can segment “outside the detection bounding box” due to the global instance unary potential (Eq. 4). As
MNC does not reason globally about the image, it cannot handle cases of overlapping bounding boxes well, and produces
more instances than there actually are. The ﬁrst column shows the input image, and the results of our object detector which
are another input to our network. MNC does not use these detections, but does internally produce box-based proposals which
are not shown. Best viewed in colour.
Input image MNC [ 12] Ours Ground truth
Figure 9 continued: Comparison to MNC [ 12]The above examples show that our method produces more precise segmen-
tations than MNC, that adhere to the boundaries of the objects. However, in Rows 3, 4 and 5, we see that MNC is able
to segment instances that our method misses out. In Row 3 , our algorithm does not segment the baby, although there is a
detection for it. This suggests that our shape prior which was formulated to overcome such occlusions could be better. As
MNC processes individual instances, it does not have a problem with dealing with small, occluding instances. In Row 4 ,
MNC has again identiﬁed a person that our algorithm could not. However, this is because we did not have a detection for
this person. In Row 5 , MNC has segmented the horses on the right better than our method. The ﬁrst column shows the input
image, and the results of our object detector which are another input to our network. MNC does not use these detections, but
does internally produce box-based proposals which are not shown. We used the publicly available code, models and default
parameters of MNC to produce this ﬁgure. Best viewed in colour.
Input image FCIS [ 31] Ours
Figure 10: Comparison to FCIS [ 31]The above images compare our method to the concurrent work, FCIS [ 31], which was
trained on COCO [ 35] and won the COCO 2016 challenge. Unlike proposal-based methods such as FCIS, our method can
handle false-positive detections and poor bounding-box localisation. Furthermore, as our method reasons globally about the
image, one pixel can only be assigned to a single instance, which is not the case with FCIS. Our method also produces more
precise segmentations, as it includes a differentiable CRF, and it is based off a semantic segmentation network. The results of
FCIS are obtained from their publicly available results on the COCO test set ( https://github.com/daijifeng001/
TA-FCN ). Note that FCIS is trained on COCO, and our model is trained on Pascal VOC which does not have as many classes
as COCO, such as “umbrella” and “suitcase” among others. As a result, we are not able to detect these objects. The ﬁrst
column shows the input image, and the results of our object detector which are another input to our network. FCIS does not
use these detections, but does internally produce proposals which are not shown. Best viewed in colour.
Input image Semantic Segmentation Instance Segmentation
Figure 11: Sample results on the Cityscapes dataset The above images show how our method can handle the large numbers
of instances present in the Cityscapes dataset. Unlike other recent approaches, our algorithm can deal with objects that are
not continuous – such as the car in the ﬁrst row which is occluded by a pole. Best viewed in colour.
A. Detailed results on the VOC dataset
Figure 12shows a visualisation of the AProbtained by
our method for each class across nine different thresholds.
Each “column” of Fig. 12corresponds to the APrfor each
class at a given IoU threshold. It is therefore an alternate
representation for the results tables (Tables 7to9). We can
see that our method struggles with classes such as “bicy-
cle”, “chair”, “dining table” and “potted plant”. This may
be explained by the fact that current semantic segmentation
systems (including ours) struggle with these classes. All re-
cent methods on the Pascal VOC leaderboard2obtain an
IoU for these classes which is lower than the mean IoU for
all classes. In fact the semantic segmentation IoU for the
“chair” class is less than half of the mean IoU for all the
classes for 16 out of the 20 most recent submissions on the
VOC leaderboard at the time of writing.
Tables 7to9show per-class instance segmentation re-
sults on the VOC dataset, at IoU thresholds of 0.9, 0.7 and
0.5 respectively. At an IoU threshold of 0.9, our method
achieves the highest APrfor 16 of the 20 object classes.
At the threshold of 0.7, we achieve the highest APrin 15
classes. Finally, at an IoU threshold of 0.5, our method,
MPA 3-scale [ 37] and PFN [ 33] each achieve the highest
APrfor 6 categories.
B. Detailed results on the SBD dataset
Once again, we show a visualisation of the AProbtained
by our method for each class across nine different thresh-
olds (Fig. 13). The trend is quite similar to the VOC dataset
in that our algorithm struggles on the same object classes
(“chair”, “dining table”, “potted plant”, “bottle”). Note that
ourAPrfor the “bicycle” class has improved compared to
the VOC dataset. This is probably because the VOC dataset
has more detailed annotations. In the VOC dataset, each
spoke of a bicycle’s wheel is often labelled, whilst in SBD,
the entire wheel is labelled as a single circle with the “bi-
cycle” label. Therefore, the SBD dataset’s coarser labelling
makes it easier for an algorithm to perform well on objects
with ﬁne details.
Table 6shows our mean AProver all classes at thresh-
olds ranging from 0.5 to 0.9. Our APrat 0.9 is low com-
pared to the result which we obtained on the VOC dataset.
This could be for a number of reasons: As the SBD dataset
is not as ﬁnely annotated as the VOC dataset, it might not
be suited for measuring the APrat such high thresholds.
Additionally, the training data is not as good for training
our system which includes a CRF and is therefore able to
delineate sharp boundaries. Finally, as the SBD dataset has
5732 validation images (compared to the 1449 in VOC), it
leaves less data for pretraining our initial semantic segmen-
2http://host.robots.ox.ac.uk:8080/leaderboard/
displaylb.php?challengeid=11&compid=6tation module. This may hinder our network in being able
to produce precise segmentations.
Table 6: Comparison of Instance Segmentation perfor-
mance at multiple APrthesholds on the VOC 2012 Vali-
dation Set
MethodAPr
APr
vol0.5 0.6 0.7 0.8 0.9
Ours (piecewise) 59.1 51.9 42.1 29.4 12.0 52.3
Ours (end-to-end ) 62.0 54.0 44.8 32.3 13.8 55.4
Tables 10and11show per-class instance segmentation
results on the SBD dataset, at IoU thresholds of 0.7 and
0.5 respectively. We can only compare results at these two
thresholds since these are the only thresholds which other
work has reported.
Figure 12: A visualisation of the AProbtained for each of the 20 classes on the VOC dataset, at nine different IoU thresholds.
The x-axis represents the IoU threshold, and the y-axis each of the Pascal classes. Therefore, each “column” of this ﬁgure
corresponds to the APrper class at a particular threshold, and is thus an alternate representation to the results tables. Best
viewed in colour.
Figure 13: A visualisation of the AProbtained for each of the 20 classes on the SBD dataset, at nine different IoU thresholds.
The x-axis represents the IoU threshold, and the y-axis each of the Pascal classes. Therefore, each “column” of this ﬁgure
corresponds to the APrper class at a particular threshold, and is thus an alternate representation to the results tables. Best
viewed in colour.
Table 7: Comparison of mean APr, achieved by different published methods, at an IoU threshold of 0.9, for all twenty classes in the VOC dataset.
MethodMean
APr(%)aero-
planebike bird boatbot-
tlebus car cat chair cow table dog horse mbikeper-
sonplant sheep sofa train tv
Our method 25.1 56.6 0.03 36.8 14.4 9.9 39.0 17.2 47.1 1.3 29.0 9.5 47.2 29.8 20.0 14.8 2.3 25.9 23.8 45.7 32.3
MPA 3-scale [ 37] 18.5 – – – – – – – – – – – – – – – – – – – –
MPA 1-scale [ 37] 17.3 – – – – – – – – – – – – – – – – – – – –
Arnab et al. [3] 20.1 43.7 0.03 30.0 13.2 11.4 47.3 10.9 34.5 0.7 19.6 12.1 35.6 24.3 13.3 10.7 0.4 20.7 20.9 35.0 17.4
PFN [ 33] 15.7 43.9 0.1 24.5 7.8 4.1 32.5 6.3 42.0 0.6 25.7 3.2 31.8 13.4 8.1 5.9 1.6 14.8 14.3 25.0 8.5
Chen et al. [8] 2.6 0.6 0 0.6 0.5 4.9 9.8 1.1 8.3 0.1 1.1 1.2 1.7 0.3 0.8 0.6 0.3 0.8 7.6 4.3 6.2
SDS [ 19] 0.9 0 0 0.2 0.3 2.0 3.8 0.2 0.9 0.1 0.2 1.5 0 0 0 0.1 0.1 0 2.3 0.2 5.8
Table 8: Comparison of mean APr, achieved by different published methods, at an IoU threshold of 0.7, for all twenty classes in the VOC dataset.
MethodMean
APr(%)aero-
planebike bird boatbot-
tlebus car cat chair cow table dog horse mbikeper-
sonplant sheep sofa train tv
Our method 48.6 69.6 1.4 68.2 45.1 25.2 61.1 38.7 72.1 11.2 56.3 30.0 73.3 60.7 64.3 46.8 17.1 49.1 44.6 75.0 62.0
MPA 3-scale [ 37] 47.4 – – – – – – – – – – – – – – – – – – – –
MPA 1-scale [ 37] 45.9 – – – – – – – – – – – – – – – – – – – –
Arnab et al. [3] 45.4 68.9 0.84 65.1 38.3 26.3 64.7 31.8 72.7 6.7 45.4 32.9 67.9 60.0 63.7 41.1 13.4 43.9 41.1 74.6 48.1
PFN [ 33] 42.5 68.5 5.6 60.4 34.8 14.9 61.4 19.2 78.6 4.2 51.1 28.2 69.6 60.7 60.5 26.5 9.8 35.1 43.9 71.2 45.6
Chen et al. [8] 27.0 40.8 0.07 40.1 16.2 19.6 56.2 26.5 46.1 2.6 25.2 16.4 36.0 22.1 20.0 22.6 7.7 27.5 19.5 47.7 46.7
SDS [ 19] 21.3 17.8 0 32.5 7.2 19.2 47.7 22.8 42.3 1.7 18.9 16.9 20.6 14.4 12.0 15.7 5.0 23.7 15.2 40.5 51.4
Table 9: Comparison of mean APr, achieved by different published methods, at an IoU threshold of 0.5, for all twenty classes in the VOC dataset.
MethodMean
APr(%)aero-
planebike bird boatbot-
tlebus car cat chair cow table dog horse mbikeper-
sonplant sheep sofa train tv
Our method 61.7 80.2 19.3 76.4 69.0 35.3 74.5 50.8 84.5 22.8 70.9 43.3 87.7 71.3 76.2 65.6 37.2 61.3 50.3 90.5 67.2
MPA 3-scale [ 37] 62.1 79.7 11.5 71.6 54.6 44.7 80.9 62.0 85.4 26.5 64.5 46.6 87.6 71.7 77.9 72.1 48.8 57.4 48.8 78.9 70.8
MPA 1-scale [ 37] 60.3 79.2 13.4 71.6 59.0 41.5 73.8 52.3 87.3 23.3 61.2 42.5 83.1 70.0 77.0 67.6 50.7 56.0 45.9 80.0 70.5
Arnab et al. [3] 58.4 80.4 7.9 74.4 59.8 32.7 76.6 39.6 84.6 19.3 62.7 44.1 81.0 74.7 72.0 58.6 32.0 59.6 50.5 87.4 68.4
PFN [ 33] 58.7 76.4 15.6 74.2 54.1 26.3 73.8 31.4 92.1 17.4 73.7 48.1 82.2 81.7 72.0 48.4 23.7 57.7 64.4 88.9 72.3
Chen et al. [8] 46.3 63.6 0.3 61.5 43.9 33.8 67.3 46.9 74.4 8.6 52.3 31.3 63.5 48.8 47.9 48.3 26.3 40.1 33.5 66.7 67.8
SDS [ 19] 43.8 58.8 0.5 60.1 34.4 29.5 60.6 40.0 73.6 6.5 52.4 31.7 62.0 49.1 45.6 47.9 22.6 43.5 26.9 66.2 66.1
Table 10: Comparison of mean APr, achieved by different published methods, at an IoU threshold of 0.7, for all twenty classes in the SBD dataset.
MethodMean
APr(%)aero-
planebike bird boatbot-
tlebus car cat chair cow table dog horse mbikeper-
sonplant sheep sofa train tv
Our method 44.8 69.0 27.4 52.7 26.4 22.4 70.3 46.0 74.7 9.6 46.8 16.9 71.6 48.4 46.3 40.3 14.8 47.6 36.5 69.7 58.2
IIS sp, rescore[
30]43.3 61.9 35.1 44.4 26.4 29.6 74.0 48.7 66.8 10.9 48.4 13.6 64.0 53.0 46.8 33.0 19.0 51.0 23.7 62.2 53.9
IIS raw [ 30] 38.7 61.8 31.5 42.0 22.0 22.7 72.4 44.8 65.4 7.2 37.6 10.4 60.4 39.6 41.9 32.5 12.0 40.9 19.9 58.8 50.8
Table 11: Comparison of mean APr, achieved by different published methods, at an IoU threshold of 0.5, for all twenty classes in the SBD dataset.
MethodMean
APr(%)aero-
planebike bird boatbot-
tlebus car cat chair cow table dog horse mbikeper-
sonplant sheep sofa train tv
Our method 62.0 80.3 52.8 68.5 47.4 39.5 79.1 61.5 87.0 28.1 68.3 35.5 86.1 73.9 66.1 63.8 32.9 65.3 50.4 81.4 71.4
IIS sp, rescore[
30]63.6 79.2 67.9 70.0 47.9 45.3 81.6 68.8 84.1 30.4 65.5 31.8 83.6 75.5 74.5 66.6 37.7 70.6 44.7 77.7 68.7
IIS raw [ 30] 60.1 77.3 65.3 65.5 42.5 35.4 80.3 62.2 83.9 27.2 61.6 32.4 82.3 70.9 71.4 63.1 31.3 63.6 44.9 78.3 62.4

1. Multiscale Combinatorial Grouping

P. Arbelaez, J. Pont-Tuset, J. Barron, F. Marques, J. Malik

2. Higher Order Conditional Random Fields in Deep Neural Networks

A. Arnab, S. Jayasumana, S. Zheng, P. H. S. Torr

3. Bottom-up Instance Segmentation with Deep Higher Order CRFs

4. Deep Watershed Transform for Instance Segmentation

5. Learning Deep Structured Models

L. Chen, A. Schwing, A. Yuille, R. Urtasun

6. Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs

L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. L. Yuille