Summary:
Convolutional networks are powerful visual models that yield hierarchies of
features. We show that convolutional networks by themselves, trained
end-to-end, pixels-to-pixels, exceed the state-of-the-art in semantic
segmentation. Our key insight is to build "fully convolutional" networks that
take input of arbitrary size and produce correspondingly-sized output with
efficient inference and learning. We define and detail the space of fully
convolutional networks, explain their application to spatially dense prediction
tasks, and draw connections to prior models. We adapt contemporary
classification networks (AlexNet, the VGG net, and GoogLeNet) into fully
convolutional networks and transfer their learned representations by
fine-tuning to the segmentation task. We then define a novel architecture that
combines semantic information from a deep, coarse layer with appearance
information from a shallow, fine layer to produce accurate and detailed
segmentations. Our fully convolutional network achieves state-of-the-art
segmentation of PASCAL VOC (20% relative improvement to 62.2% mean IU on 2012),
NYUDv2, and SIFT Flow, while inference takes one third of a second for a
typical image.

Jonathan Long
Evan Shelhamer
Trevor Darrell

to appear in CVPR (2015)

Fully Convolutional Networks for Semantic Segmentation
Jonathan LongEvan ShelhamerTrevor Darrell
UC Berkeley
fjonlong,shelhamer,trevor g@cs.berkeley.edu
Abstract
Convolutional networks are powerful visual models that
yield hierarchies of features. We show that convolu-
tional networks by themselves, trained end-to-end, pixels-
to-pixels, exceed the state-of-the-art in semantic segmen-
tation. Our key insight is to build “fully convolutional”
networks that take input of arbitrary size and produce
correspondingly-sized output with efﬁcient inference and
learning. We deﬁne and detail the space of fully convolu-
tional networks, explain their application to spatially dense
prediction tasks, and draw connections to prior models. We
adapt contemporary classiﬁcation networks (AlexNet [19],
the VGG net [31], and GoogLeNet [32]) into fully convolu-
tional networks and transfer their learned representations
by ﬁne-tuning [4] to the segmentation task. We then de-
ﬁne a novel architecture that combines semantic informa-
tion from a deep, coarse layer with appearance information
from a shallow, ﬁne layer to produce accurate and detailed
segmentations. Our fully convolutional network achieves
state-of-the-art segmentation of PASCAL VOC (20% rela-
tive improvement to 62.2% mean IU on 2012), NYUDv2,
and SIFT Flow, while inference takes less than one ﬁfth of a
second for a typical image.
1. Introduction
Convolutional networks are driving advances in recog-
nition. Convnets are not only improving for whole-image
classiﬁcation [19, 31, 32], but also making progress on lo-
cal tasks with structured output. These include advances in
bounding box object detection [29, 12, 17], part and key-
point prediction [39, 24], and local correspondence [24, 9].
The natural next step in the progression from coarse to
ﬁne inference is to make a prediction at every pixel. Prior
approaches have used convnets for semantic segmentation
[27, 2, 8, 28, 16, 14, 11], in which each pixel is labeled with
the class of its enclosing object or region, but with short-
comings that this work addresses.
Authors contributed equally
963842564096409621
21backward/learningforward/inference
pixelwise predictionsegmentation g.t.
256384Figure 1. Fully convolutional networks can efﬁciently learn to
make dense predictions for per-pixel tasks like semantic segmen-
tation.
We show that a fully convolutional network (FCN),
trained end-to-end, pixels-to-pixels on semantic segmen-
tation exceeds the state-of-the-art without further machin-
ery. To our knowledge, this is the ﬁrst work to train FCNs
end-to-end (1) for pixelwise prediction and (2) from super-
vised pre-training. Fully convolutional versions of existing
networks predict dense outputs from arbitrary-sized inputs.
Both learning and inference are performed whole-image-at-
a-time by dense feedforward computation and backpropa-
gation. In-network upsampling layers enable pixelwise pre-
diction and learning in nets with subsampled pooling.
This method is efﬁcient, both asymptotically and abso-
lutely, and precludes the need for the complications in other
works. Patchwise training is common [27, 2, 8, 28, 11], but
lacks the efﬁciency of fully convolutional training. Our ap-
proach does not make use of pre- and post-processing com-
plications, including superpixels [8, 16], proposals [16, 14],
or post-hoc reﬁnement by random ﬁelds or local classiﬁers
[8, 16]. Our model transfers recent success in classiﬁca-
tion [19, 31, 32] to dense prediction by reinterpreting clas-
siﬁcation nets as fully convolutional and ﬁne-tuning from
their learned representations. In contrast, previous works
have applied small convnets without supervised pre-training
[8, 28, 27].
Semantic segmentation faces an inherent tension be-
tween semantics and location: global information resolves
what while local information resolves where. Deep feature
1arXiv:1411.4038v2  [cs.CV]  8 Mar 2015
hierarchies jointly encode location and semantics in a local-
to-global pyramid. We deﬁne a novel “skip” architecture
to combine deep, coarse, semantic information and shallow,
ﬁne, appearance information in Section 4.2 (see Figure 3).
In the next section, we review related work on deep clas-
siﬁcation nets, FCNs, and recent approaches to semantic
segmentation using convnets. The following sections ex-
plain FCN design and dense prediction tradeoffs, introduce
our architecture with in-network upsampling and multi-
layer combinations, and describe our experimental frame-
work. Finally, we demonstrate state-of-the-art results on
PASCAL VOC 2011-2, NYUDv2, and SIFT Flow.
2. Related work
Our approach draws on recent successes of deep nets
for image classiﬁcation [19, 31, 32] and transfer learning
[4, 38]. Transfer was ﬁrst demonstrated on various visual
recognition tasks [4, 38], then on detection, and on both
instance and semantic segmentation in hybrid proposal-
classiﬁer models [12, 16, 14]. We now re-architect and ﬁne-
tune classiﬁcation nets to direct, dense prediction of seman-
tic segmentation. We chart the space of FCNs and situate
prior models, both historical and recent, in this framework.
Fully convolutional networks To our knowledge, the
idea of extending a convnet to arbitrary-sized inputs ﬁrst
appeared in Matan et al. [25], which extended the classic
LeNet [21] to recognize strings of digits. Because their net
was limited to one-dimensional input strings, Matan et al.
used Viterbi decoding to obtain their outputs. Wolf and Platt
[37] expand convnet outputs to 2-dimensional maps of de-
tection scores for the four corners of postal address blocks.
Both of these historical works do inference and learning
fully convolutionally for detection. Ning et al. [27] deﬁne
a convnet for coarse multiclass segmentation of C. elegans
tissues with fully convolutional inference.
Fully convolutional computation has also been exploited
in the present era of many-layered nets. Sliding window
detection by Sermanet et al . [29], semantic segmentation
by Pinheiro and Collobert [28], and image restoration by
Eigen et al. [5] do fully convolutional inference. Fully con-
volutional training is rare, but used effectively by Tompson
et al. [35] to learn an end-to-end part detector and spatial
model for pose estimation, although they do not exposit on
or analyze this method.
Alternatively, He et al . [17] discard the non-
convolutional portion of classiﬁcation nets to make a
feature extractor. They combine proposals and spatial
pyramid pooling to yield a localized, ﬁxed-length feature
for classiﬁcation. While fast and effective, this hybrid
model cannot be learned end-to-end.
Dense prediction with convnets Several recent works
have applied convnets to dense prediction problems, includ-
ing semantic segmentation by Ning et al. [27], Farabet et al.[8], and Pinheiro and Collobert [28]; boundary prediction
for electron microscopy by Ciresan et al. [2] and for natural
images by a hybrid neural net/nearest neighbor model by
Ganin and Lempitsky [11]; and image restoration and depth
estimation by Eigen et al. [5, 6]. Common elements of these
approaches include
small models restricting capacity and receptive ﬁelds;
patchwise training [27, 2, 8, 28, 11];
post-processing by superpixel projection, random ﬁeld
regularization, ﬁltering, or local classiﬁcation [8, 2,
11];
input shifting and output interlacing for dense output
[28, 11] as introduced by OverFeat [29];
multi-scale pyramid processing [8, 28, 11];
saturating tanh nonlinearities [8, 5, 28]; and
ensembles [2, 11],
whereas our method does without this machinery. However,
we do study patchwise training 3.4 and “shift-and-stitch”
dense output 3.2 from the perspective of FCNs. We also
discuss in-network upsampling 3.3, of which the fully con-
nected prediction by Eigen et al. [6] is a special case.
Unlike these existing methods, we adapt and extend deep
classiﬁcation architectures, using image classiﬁcation as su-
pervised pre-training, and ﬁne-tune fully convolutionally to
learn simply and efﬁciently from whole image inputs and
whole image ground thruths.
Hariharan et al. [16] and Gupta et al. [14] likewise adapt
deep classiﬁcation nets to semantic segmentation, but do
so in hybrid proposal-classiﬁer models. These approaches
ﬁne-tune an R-CNN system [12] by sampling bounding
boxes and/or region proposals for detection, semantic seg-
mentation, and instance segmentation. Neither method is
learned end-to-end.
They achieve state-of-the-art results on PASCAL VOC
segmentation and NYUDv2 segmentation respectively, so
we directly compare our standalone, end-to-end FCN to
their semantic segmentation results in Section 5.
3. Fully convolutional networks
Each layer of data in a convnet is a three-dimensional
array of size hwd, wherehandware spatial dimen-
sions, anddis the feature or channel dimension. The ﬁrst
layer is the image, with pixel size hw, anddcolor chan-
nels. Locations in higher layers correspond to the locations
in the image they are path-connected to, which are called
their receptive ﬁelds .
Convnets are built on translation invariance. Their ba-
sic components (convolution, pooling, and activation func-
tions) operate on local input regions, and depend only on
relative spatial coordinates. Writing xijfor the data vector
at location (i;j)in a particular layer, and yijfor the follow-
ing layer, these functions compute outputs yijby
yij=fks(fxsi+i;sj +jg0i;jk)
wherekis called the kernel size, sis the stride or subsam-
pling factor, and fksdetermines the layer type: a matrix
multiplication for convolution or average pooling, a spatial
max for max pooling, or an elementwise nonlinearity for an
activation function, and so on for other types of layers.
This functional form is maintained under composition,
with kernel size and stride obeying the transformation rule
fksgk0s0= (fg)k0+(k1)s0;ss0:
While a general deep net computes a general nonlinear
function, a net with only layers of this form computes a
nonlinear ﬁlter , which we call a deep ﬁlter orfully convolu-
tional network . An FCN naturally operates on an input of
any size, and produces an output of corresponding (possibly
resampled) spatial dimensions.
A real-valued loss function composed with an FCN de-
ﬁnes a task. If the loss function is a sum over the spatial
dimensions of the ﬁnal layer, `(x;) =P
ij`0(xij;), its
gradient will be a sum over the gradients of each of its spa-
tial components. Thus stochastic gradient descent on `com-
puted on whole images will be the same as stochastic gradi-
ent descent on `0, taking all of the ﬁnal layer receptive ﬁelds
as a minibatch.
When these receptive ﬁelds overlap signiﬁcantly, both
feedforward computation and backpropagation are much
more efﬁcient when computed layer-by-layer over an entire
image instead of independently patch-by-patch.
We next explain how to convert classiﬁcation nets into
fully convolutional nets that produce coarse output maps.
For pixelwise prediction, we need to connect these coarse
outputs back to the pixels. Section 3.2 describes a trick that
OverFeat [29] introduced for this purpose. We gain insight
into this trick by reinterpreting it as an equivalent network
modiﬁcation. As an efﬁcient, effective alternative, we in-
troduce deconvolution layers for upsampling in Section 3.3.
In Section 3.4 we consider training by patchwise sampling,
and give evidence in Section 4.3 that our whole image train-
ing is faster and equally effective.
3.1. Adapting classiﬁers for dense prediction
Typical recognition nets, including LeNet [21], AlexNet
[19], and its deeper successors [31, 32], ostensibly take
ﬁxed-sized inputs and produce nonspatial outputs. The fully
connected layers of these nets have ﬁxed dimensions and
throw away spatial coordinates. However, these fully con-
nected layers can also be viewed as convolutions with ker-
nels that cover their entire input regions. Doing so casts
them into fully convolutional networks that take input of
any size and output classiﬁcation maps. This transformation
`tabby cat" `
96256384384256409640961000
96384256409640961000
256384tabby cat heatmapconvolutionalizationFigure 2. Transforming fully connected layers into convolution
layers enables a classiﬁcation net to output a heatmap. Adding
layers and a spatial loss (as in Figure 1) produces an efﬁcient ma-
chine for end-to-end dense learning.
is illustrated in Figure 2. (By contrast, nonconvolutional
nets, such as the one by Le et al. [20], lack this capability.)
Furthermore, while the resulting maps are equivalent to
the evaluation of the original net on particular input patches,
the computation is highly amortized over the overlapping
regions of those patches. For example, while AlexNet takes
1:2ms (on a typical GPU) to produce the classiﬁcation
scores of a 227227image, the fully convolutional ver-
sion takes 22ms to produce a 1010grid of outputs from
a500500image, which is more than 5times faster than
the na ¨ıve approach1.
The spatial output maps of these convolutionalized mod-
els make them a natural choice for dense problems like se-
mantic segmentation. With ground truth available at ev-
ery output cell, both the forward and backward passes are
straightforward, and both take advantage of the inherent
computational efﬁciency (and aggressive optimization) of
convolution.
The corresponding backward times for the AlexNet ex-
ample are 2:4ms for a single image and 37ms for a fully
convolutional 1010output map, resulting in a speedup
similar to that of the forward pass. This dense backpropa-
gation is illustrated in Figure 1.
While our reinterpretation of classiﬁcation nets as fully
convolutional yields output maps for inputs of any size, the
output dimensions are typically reduced by subsampling.
The classiﬁcation nets subsample to keep ﬁlters small and
computational requirements reasonable. This coarsens the
output of a fully convolutional version of these nets, reduc-
ing it from the size of the input by a factor equal to the pixel
stride of the receptive ﬁelds of the output units.
1Assuming efﬁcient batching of single image inputs. The classiﬁcation
scores for a single image by itself take 5.4 ms to produce, which is nearly
25times slower than the fully convolutional version.
3.2. Shift-and-stitch is ﬁlter rarefaction
Input shifting and output interlacing is a trick that yields
dense predictions from coarse outputs without interpola-
tion, introduced by OverFeat [29]. If the outputs are down-
sampled by a factor of f, the input is shifted (by left and top
padding)xpixels to the right and ypixels down, once for
every value of (x;y)2f0;:::;f1gf 0;:::;f1g.
Thesef2inputs are each run through the convnet, and the
outputs are interlaced so that the predictions correspond to
the pixels at the centers of their receptive ﬁelds.
Changing only the ﬁlters and layer strides of a convnet
can produce the same output as this shift-and-stitch trick.
Consider a layer (convolution or pooling) with input stride
s, and a following convolution layer with ﬁlter weights fij
(eliding the feature dimensions, irrelevant here). Setting the
lower layer’s input stride to 1upsamples its output by a fac-
tor ofs, just like shift-and-stitch. However, convolving the
original ﬁlter with the upsampled output does not produce
the same result as the trick, because the original ﬁlter only
sees a reduced portion of its (now upsampled) input. To
reproduce the trick, rarefy the ﬁlter by enlarging it as
f0
ij=fi=s;j=s ifsdivides both iandj;
0 otherwise,
(withiandjzero-based). Reproducing the full net output
of the trick involves repeating this ﬁlter enlargement layer-
by-layer until all subsampling is removed.
Simply decreasing subsampling within a net is a tradeoff:
the ﬁlters see ﬁner information, but have smaller receptive
ﬁelds and take longer to compute. We have seen that the
shift-and-stitch trick is another kind of tradeoff: the output
is made denser without decreasing the receptive ﬁeld sizes
of the ﬁlters, but the ﬁlters are prohibited from accessing
information at a ﬁner scale than their original design.
Although we have done preliminary experiments with
shift-and-stitch, we do not use it in our model. We ﬁnd
learning through upsampling, as described in the next sec-
tion, to be more effective and efﬁcient, especially when
combined with the skip layer fusion described later on.
3.3. Upsampling is backwards strided convolution
Another way to connect coarse outputs to dense pixels
is interpolation. For instance, simple bilinear interpolation
computes each output yijfrom the nearest four inputs by a
linear map that depends only on the relative positions of the
input and output cells.
In a sense, upsampling with factor fis convolution with
afractional input stride of 1=f. So long as fis integral, a
natural way to upsample is therefore backwards convolution
(sometimes called deconvolution ) with an output stride of
f. Such an operation is trivial to implement, since it simply
reverses the forward and backward passes of convolution.Thus upsampling is performed in-network for end-to-end
learning by backpropagation from the pixelwise loss.
Note that the deconvolution ﬁlter in such a layer need not
be ﬁxed (e.g., to bilinear upsampling), but can be learned.
A stack of deconvolution layers and activation functions can
even learn a nonlinear upsampling.
In our experiments, we ﬁnd that in-network upsampling
is fast and effective for learning dense prediction. Our best
segmentation architecture uses these layers to learn to up-
sample for reﬁned prediction in Section 4.2.
3.4. Patchwise training is loss sampling
In stochastic optimization, gradient computation is
driven by the training distribution. Both patchwise train-
ing and fully-convolutional training can be made to pro-
duce any distribution, although their relative computational
efﬁciency depends on overlap and minibatch size. Whole
image fully convolutional training is identical to patchwise
training where each batch consists of all the receptive ﬁelds
of the units below the loss for an image (or collection of
images). While this is more efﬁcient than uniform sampling
of patches, it reduces the number of possible batches. How-
ever, random selection of patches within an image may be
recovered simply. Restricting the loss to a randomly sam-
pled subset of its spatial terms (or, equivalently applying a
DropConnect mask [36] between the output and the loss)
excludes patches from the gradient computation.
If the kept patches still have signiﬁcant overlap, fully
convolutional computation will still speed up training. If
gradients are accumulated over multiple backward passes,
batches can include patches from several images.2
Sampling in patchwise training can correct class imbal-
ance [27, 8, 2] and mitigate the spatial correlation of dense
patches [28, 16]. In fully convolutional training, class bal-
ance can also be achieved by weighting the loss, and loss
sampling can be used to address spatial correlation.
We explore training with sampling in Section 4.3, and do
not ﬁnd that it yields faster or better convergence for dense
prediction. Whole image training is effective and efﬁcient.
4. Segmentation Architecture
We cast ILSVRC classiﬁers into FCNs and augment
them for dense prediction with in-network upsampling and
a pixelwise loss. We train for segmentation by ﬁne-tuning.
Next, we build a novel skip architecture that combines
coarse, semantic and local, appearance information to re-
ﬁne prediction.
For this investigation, we train and validate on the PAS-
CAL VOC 2011 segmentation challenge [7]. We train with
2Note that not every possible patch is included this way, since the re-
ceptive ﬁelds of the ﬁnal layer units lie on a ﬁxed, strided grid. However,
by shifting the image left and down by a random value up to the stride,
random selection from all possible patches may be recovered.
a per-pixel multinomial logistic loss and validate with the
standard metric of mean pixel intersection over union, with
the mean taken over all classes, including background. The
training ignores pixels that are masked out (as ambiguous
or difﬁcult) in the ground truth.
4.1. From classiﬁer to dense FCN
We begin by convolutionalizing proven classiﬁcation ar-
chitectures as in Section 3. We consider the AlexNet3ar-
chitecture [19] that won ILSVRC12, as well as the VGG
nets [31] and the GoogLeNet4[32] which did exception-
ally well in ILSVRC14. We pick the VGG 16-layer net5,
which we found to be equivalent to the 19-layer net on this
task. For GoogLeNet, we use only the ﬁnal loss layer, and
improve performance by discarding the ﬁnal average pool-
ing layer. We decapitate each net by discarding the ﬁnal
classiﬁer layer, and convert all fully connected layers to
convolutions. We append a 11convolution with chan-
nel dimension 21to predict scores for each of the PAS-
CAL classes (including background) at each of the coarse
output locations, followed by a deconvolution layer to bi-
linearly upsample the coarse outputs to pixel-dense outputs
as described in Section 3.3. Table 1 compares the prelim-
inary validation results along with the basic characteristics
of each net. We report the best results achieved after con-
vergence at a ﬁxed learning rate (at least 175 epochs).
Fine-tuning from classiﬁcation to segmentation gave rea-
sonable predictions for each net. Even the worst model
achieved75% of state-of-the-art performance. The
segmentation-equippped VGG net (FCN-VGG16) already
appears to be state-of-the-art at 56.0 mean IU on val, com-
pared to 52.6 on test [16]. Training on extra data raises
performance to 59.4 mean IU on a subset of val7. Training
details are given in Section 4.3.
Despite similar classiﬁcation accuracy, our implementa-
tion of GoogLeNet did not match this segmentation result.
4.2. Combining what and where
We deﬁne a new fully convolutional net (FCN) for seg-
mentation that combines layers of the feature hierarchy and
reﬁnes the spatial precision of the output. See Figure 3.
While fully convolutionalized classiﬁers can be ﬁne-
tuned to segmentation as shown in 4.1, and even score
highly on the standard metric, their output is dissatisfyingly
coarse (see Figure 4). The 32 pixel stride at the ﬁnal predic-
tion layer limits the scale of detail in the upsampled output.
We address this by adding links that combine the ﬁnal
prediction layer with lower layers with ﬁner strides. This
3Using the publicly available CaffeNet reference model.
4Since there is no publicly available version of GoogLeNet, we use
our own reimplementation. Our version is trained with less extensive data
augmentation, and gets 68.5% top-1 and 88.4% top-5 ILSVRC accuracy.
5Using the publicly available version from the Caffe model zoo.Table 1. We adapt and extend three classiﬁcation convnets to seg-
mentation. We compare performance by mean intersection over
union on the validation set of PASCAL VOC 2011 and by infer-
ence time (averaged over 20 trials for a 500500 input on an
NVIDIA Tesla K40c). We detail the architecture of the adapted
nets as regards dense prediction: number of parameter layers, re-
ceptive ﬁeld size of output units, and the coarsest stride within the
net. (These numbers give the best performance obtained at a ﬁxed
learning rate, not best performance possible.)
FCN-
AlexNetFCN-
VGG16FCN-
GoogLeNet4
mean IU 39.8 56.0 42.5
forward time 50 ms 210 ms 59 ms
conv. layers 8 16 22
parameters 57M 134M 6M
rf size 355 404 907
max stride 32 32 32
turns a line topology into a DAG, with edges that skip ahead
from lower layers to higher ones (Figure 3). As they see
fewer pixels, the ﬁner scale predictions should need fewer
layers, so it makes sense to make them from shallower net
outputs. Combining ﬁne layers and coarse layers lets the
model make local predictions that respect global structure.
By analogy to the multiscale local jet of Florack et al. [10],
we call our nonlinear local feature hierarchy the deep jet .
We ﬁrst divide the output stride in half by predicting
from a 16 pixel stride layer. We add a 11convolution
layer on top of pool4 to produce additional class predic-
tions. We fuse this output with the predictions computed
on top of conv7 (convolutionalized fc7) at stride 32 by
adding a 2upsampling layer and summing6both predic-
tions. (See Figure 3). We initialize the 2upsampling to
bilinear interpolation, but allow the parameters to be learned
as described in Section 3.3. Finally, the stride 16 predictions
are upsampled back to the image. We call this net FCN-16s.
FCN-16s is learned end-to-end, initialized with the param-
eters of the last, coarser net, which we now call FCN-32s.
The new parameters acting on pool4 are zero-initialized so
that the net starts with unmodiﬁed predictions. The learning
rate is decreased by a factor of 100.
Learning this skip net improves performance on the val-
idation set by 3.0 mean IU to 62.4. Figure 4 shows im-
provement in the ﬁne structure of the output. We compared
this fusion with learning only from the pool4 layer (which
resulted in poor performance), and simply decreasing the
learning rate without adding the extra link (which results
in an insigniﬁcant performance improvement, without im-
proving the quality of the output).
We continue in this fashion by fusing predictions from
pool3 with a 2upsampling of predictions fused from
pool4 andconv7 , building the net FCN-8s. We obtain
6Max fusion made learning difﬁcult due to gradient switching.
FCN-32s FCN-16s FCN-8s Ground truth
Figure 4. Reﬁning fully convolutional nets by fusing information
from layers with different strides improves segmentation detail.
The ﬁrst three images show the output from our 32, 16, and 8
pixel stride nets (see Figure 3).
Table 2. Comparison of skip FCNs on a subset of PASCAL
VOC2011 validation7. Learning is end-to-end, except for FCN-
32s-ﬁxed, where only the last layer is ﬁne-tuned. Note that FCN-
32s is FCN-VGG16, renamed to highlight stride.
pixel
acc.mean
acc.mean
IUf.w.
IU
FCN-32s-ﬁxed 83.0 59.7 45.4 72.0
FCN-32s 89.1 73.3 59.4 81.4
FCN-16s 90.0 75.7 62.4 83.0
FCN-8s 90.3 75.9 62.7 83.2
a minor additional improvement to 62.7 mean IU, and ﬁnd
a slight improvement in the smoothness and detail of our
output. At this point our fusion improvements have met di-
minishing returns, both with respect to the IU metric which
emphasizes large-scale correctness, and also in terms of the
improvement visible e.g. in Figure 4, so we do not continue
fusing even lower layers.
Reﬁnement by other means Decreasing the stride of
pooling layers is the most straightforward way to obtain
ﬁner predictions. However, doing so is problematic for our
VGG16-based net. Setting the pool5 layer to have stride 1
requires our convolutionalized fc6 to have a kernel size of1414in order to maintain its receptive ﬁeld size. In addi-
tion to their computational cost, we had difﬁculty learning
such large ﬁlters. We made an attempt to re-architect the
layers above pool5 with smaller ﬁlters, but were not suc-
cessful in achieving comparable performance; one possible
explanation is that the initialization from ImageNet-trained
weights in the upper layers is important.
Another way to obtain ﬁner predictions is to use the shift-
and-stitch trick described in Section 3.2. In limited exper-
iments, we found the cost to improvement ratio from this
method to be worse than layer fusion.
4.3. Experimental framework
Optimization We train by SGD with momentum. We
use a minibatch size of 20 images and ﬁxed learning rates of
103,104, and 55for FCN-AlexNet, FCN-VGG16, and
FCN-GoogLeNet, respectively, chosen by line search. We
use momentum 0:9, weight decay of 54or24, and dou-
bled the learning rate for biases, although we found training
to be insensitive to these parameters (but sensitive to the
learning rate). We zero-initialize the class scoring convo-
lution layer, ﬁnding random initialization to yield neither
better performance nor faster convergence. Dropout was in-
cluded where used in the original classiﬁer nets.
Fine-tuning We ﬁne-tune all layers by back-
propagation through the whole net. Fine-tuning the
output classiﬁer alone yields only 70% of the full ﬁne-
tuning performance as compared in Table 2. Training from
scratch is not feasible considering the time required to
learn the base classiﬁcation nets. (Note that the VGG net is
trained in stages, while we initialize from the full 16-layer
version.) Fine-tuning takes three days on a single GPU for
the coarse FCN-32s version, and about one day each to
upgrade to the FCN-16s and FCN-8s versions.
Patch Sampling As explained in Section 3.4, our full
image training effectively batches each image into a regu-
image pool4 pool5 pool1 pool2 pool332x upsampled
prediction (FCN-32s)2x upsampled
prediction16x upsampled
prediction (FCN-16s)8x upsampled
prediction (FCN-8s)
pool4
prediction2x upsampled
prediction
pool3
predictionP P
Figure 3. Our DAG nets learn to combine coarse, high layer information with ﬁne, low layer information. Layers are shown as grids that
reveal relative spatial coarseness. Only pooling and prediction layers are shown; intermediate convolution layers (including our converted
fully connected layers) are omitted. Solid line (FCN-32s): Our single-stream net, described in Section 4.1, upsamples stride 32 predictions
back to pixels in a single step. Dashed line (FCN-16s): Combining predictions from both the ﬁnal layer and the pool4 layer, at stride
16, lets our net predict ﬁner details, while retaining high-level semantic information. Dotted line (FCN-8s): Additional predictions from
pool3 , at stride 8, provide further precision.
500 1000 1500
iteration number0.40.60.81.01.2lossfull images
50% sampling
25% sampling
10000 20000 30000
relative time (num. images processed)0.40.60.81.01.2lossFigure 5. Training on whole images is just as effective as sampling
patches, but results in faster (wall time) convergence by making
more efﬁcient use of data. Left shows the effect of sampling on
convergence rate for a ﬁxed expected batch size, while right plots
the same by relative wall time.
lar grid of large, overlapping patches. By contrast, prior
work randomly samples patches over a full dataset [27, 2,
8, 28, 11], potentially resulting in higher variance batches
that may accelerate convergence [22]. We study this trade-
off by spatially sampling the loss in the manner described
earlier, making an independent choice to ignore each ﬁnal
layer cell with some probability 1p. To avoid changing the
effective batch size, we simultaneously increase the number
of images per batch by a factor 1=p. Note that due to the ef-
ﬁciency of convolution, this form of rejection sampling is
still faster than patchwise training for large enough values
ofp(e.g., at least for p > 0:2according to the numbers
in Section 3.1). Figure 5 shows the effect of this form of
sampling on convergence. We ﬁnd that sampling does not
have a signiﬁcant effect on convergence rate compared to
whole image training, but takes signiﬁcantly more time due
to the larger number of images that need to be considered
per batch. We therefore choose unsampled, whole image
training in our other experiments.
Class Balancing Fully convolutional training can bal-
ance classes by weighting or sampling the loss. Although
our labels are mildly unbalanced (about 3=4are back-
ground), we ﬁnd class balancing unnecessary.
Dense Prediction The scores are upsampled to the in-
put dimensions by deconvolution layers within the net. Fi-
nal layer deconvolutional ﬁlters are ﬁxed to bilinear inter-
polation, while intermediate upsampling layers are initial-
ized to bilinear upsampling, and then learned. Shift-and-
stitch (Section 3.2), or the ﬁlter rarefaction equivalent, are
not used.
Augmentation We tried augmenting the training data
by randomly mirroring and “jittering” the images by trans-
lating them up to 32 pixels (the coarsest scale of prediction)
in each direction. This yielded no noticeable improvement.
More Training Data The PASCAL VOC 2011 segmen-
tation challenge training set, which we used for Table 1,
labels 1112 images. Hariharan et al . [15] have collectedlabels for a much larger set of 8498 PASCAL training im-
ages, which was used to train the previous state-of-the-art
system, SDS [16]. This training data improves the FCN-
VGG16 validation score7by 3.4 points to 59.4 mean IU.
Implementation All models are trained and tested with
Caffe [18] on a single NVIDIA Tesla K40c. The models
and code will be released open-source on publication.
5. Results
We test our FCN on semantic segmentation and scene
parsing, exploring PASCAL VOC, NYUDv2, and SIFT
Flow. Although these tasks have historically distinguished
between objects and regions, we treat both uniformly as
pixel prediction. We evaluate our FCN skip architecture8
on each of these datasets, and then extend it to multi-modal
input for NYUDv2 and multi-task prediction for the seman-
tic and geometric labels of SIFT Flow.
Metrics We report four metrics from common semantic
segmentation and scene parsing evaluations that are varia-
tions on pixel accuracy and region intersection over union
(IU). Letnijbe the number of pixels of class ipredicted to
belong to class j, where there are ncldifferent classes, and
letti=P
jnijbe the total number of pixels of class i. We
compute:
pixel accuracy:P
inii=P
iti
mean accuraccy: (1=ncl)P
inii=ti
mean IU: (1=ncl)P
inii=
ti+P
jnjinii
frequency weighted IU:
(P
ktk)1P
itinii=
ti+P
jnjinii
PASCAL VOC Table 3 gives the performance of our
FCN-8s on the test sets of PASCAL VOC 2011 and 2012,
and compares it to the previous state-of-the-art, SDS [16],
and the well-known R-CNN [12]. We achieve the best re-
sults on mean IU9by a relative margin of 20%. Inference
time is reduced 114(convnet only, ignoring proposals and
reﬁnement) or 286(overall).
Table 3. Our fully convolutional net gives a 20% relative improve-
ment over the state-of-the-art on the PASCAL VOC 2011 and 2012
test sets, and reduces inference time.
mean IU mean IU inference
VOC2011 test VOC2012 test time
R-CNN [12] 47.9 - -
SDS [16] 52.6 51.6 50 s
FCN-8s 62.7 62.2 175 ms
NYUDv2 [30] is an RGB-D dataset collected using the
7There are training images from [15] included in the PASCAL VOC
2011 val set, so we validate on the non-intersecting set of 736 images. An
earlier version of this paper mistakenly evaluated on the entire val set.
8Our models and code are publicly available at
https://github.com/BVLC/caffe/wiki/Model-Zoo#fcn .
9This is the only metric provided by the test server.
Table 4. Results on NYUDv2. RGBD is early-fusion of the
RGB and depth channels at the input. HHA is the depth embed-
ding of [14] as horizontal disparity, height above ground, and
the angle of the local surface normal with the inferred gravity
direction. RGB-HHA is the jointly trained late fusion model
that sums RGB and HHA predictions.
pixel
acc.mean
acc.mean
IUf.w.
IU
Gupta et al. [14] 60.3 - 28.6 47.0
FCN-32s RGB 60.0 42.2 29.2 43.9
FCN-32s RGBD 61.5 42.4 30.5 45.5
FCN-32s HHA 57.1 35.2 24.2 40.4
FCN-32s RGB-HHA 64.3 44.9 32.8 48.0
FCN-16s RGB-HHA 65.4 46.1 34.0 49.5
Microsoft Kinect. It has 1449 RGB-D images, with pixel-
wise labels that have been coalesced into a 40 class seman-
tic segmentation task by Gupta et al. [13]. We report results
on the standard split of 795 training images and 654 testing
images. (Note: all model selection is performed on PAS-
CAL 2011 val.) Table 4 gives the performance of our model
in several variations. First we train our unmodiﬁed coarse
model (FCN-32s) on RGB images. To add depth informa-
tion, we train on a model upgraded to take four-channel
RGB-D input (early fusion). This provides little beneﬁt,
perhaps due to the difﬁcultly of propagating meaningful
gradients all the way through the model. Following the suc-
cess of Gupta et al. [14], we try the three-dimensional HHA
encoding of depth, training nets on just this information, as
well as a “late fusion” of RGB and HHA where the predic-
tions from both nets are summed at the ﬁnal layer, and the
resulting two-stream net is learned end-to-end. Finally we
upgrade this late fusion net to a 16-stride version.
SIFT Flow is a dataset of 2,688 images with pixel labels
for 33 semantic categories (“bridge”, “mountain”, “sun”),
as well as three geometric categories (“horizontal”, “verti-
cal”, and “sky”). An FCN can naturally learn a joint repre-
sentation that simultaneously predicts both types of labels.
We learn a two-headed version of FCN-16s with seman-
tic and geometric prediction layers and losses. The learned
model performs as well on both tasks as two independently
trained models, while learning and inference are essentially
as fast as each independent model by itself. The results in
Table 5, computed on the standard split into 2,488 training
and 200 test images,10show state-of-the-art performance on
both tasks.
10Three of the SIFT Flow categories are not present in the test set. We
made predictions across all 33 categories, but only included categories ac-
tually present in the test set in our evaluation. (An earlier version of this pa-
per reported a lower mean IU, which included all categories either present
or predicted in the evaluation.)Table 5. Results on SIFT Flow10with class segmentation
(center) and geometric segmentation (right). Tighe [33] is
a non-parametric transfer method. Tighe 1 is an exemplar
SVM while 2 is SVM + MRF. Farabet is a multi-scale con-
vnet trained on class-balanced samples (1) or natural frequency
samples (2). Pinheiro is a multi-scale, recurrent convnet, de-
noted RCNN 3(3). The metric for geometry is pixel accuracy.
pixel
acc.mean
acc.mean
IUf.w.
IUgeom.
acc.
Liuet al. [23] 76.7 - - - -
Tighe et al. [33] - - - - 90.8
Tighe et al. [34] 1 75.6 41.1 - - -
Tighe et al. [34] 2 78.6 39.2 - - -
Farabet et al. [8] 1 72.3 50.8 - - -
Farabet et al. [8] 2 78.5 29.6 - - -
Pinheiro et al. [28] 77.7 29.8 - - -
FCN-16s 85.2 51.7 39.5 76.1 94.3
FCN-8s SDS [16] Ground Truth Image
Figure 6. Fully convolutional segmentation nets produce state-
of-the-art performance on PASCAL. The left column shows the
output of our highest performing net, FCN-8s. The second shows
the segmentations produced by the previous state-of-the-art system
by Hariharan et al. [16]. Notice the ﬁne structures recovered (ﬁrst
row), ability to separate closely interacting objects (second row),
and robustness to occluders (third row). The fourth row shows a
failure case: the net sees lifejackets in a boat as people.
6. Conclusion
Fully convolutional networks are a rich class of mod-
els, of which modern classiﬁcation convnets are a spe-
cial case. Recognizing this, extending these classiﬁcation
nets to segmentation, and improving the architecture with
multi-resolution layer combinations dramatically improves
the state-of-the-art, while simultaneously simplifying and
speeding up learning and inference.
Acknowledgements This work was supported in part
by DARPA’s MSEE and SMISC programs, NSF awards IIS-
1427425, IIS-1212798, IIS-1116411, and the NSF GRFP,
Toyota, and the Berkeley Vision and Learning Center. We
gratefully acknowledge NVIDIA for GPU donation. We
thank Bharath Hariharan and Saurabh Gupta for their ad-
vice and dataset tools. We thank Sergio Guadarrama for
reproducing GoogLeNet in Caffe. We thank Jitendra Malik
for his helpful comments. Thanks to Wei Liu for pointing
out an issue wth our SIFT Flow mean IU computation and
an error in our frequency weighted mean IU formula.
A. Upper Bounds on IU
In this paper, we have achieved good performance on
the mean IU segmentation metric even with coarse semantic
prediction. To better understand this metric and the limits
of this approach with respect to it, we compute approximate
upper bounds on performance with prediction at various
scales. We do this by downsampling ground truth images
and then upsampling them again to simulate the best results
obtainable with a particular downsampling factor. The fol-
lowing table gives the mean IU on a subset of PASCAL
2011 val for various downsampling factors.
factor mean IU
128 50.9
64 73.3
32 86.1
16 92.8
896.4
498.5
Pixel-perfect prediction is clearly not necessary to
achieve mean IU well above state-of-the-art, and, con-
versely, mean IU is a not a good measure of ﬁne-scale ac-
curacy.
B. More Results
We further evaluate our FCN for semantic segmentation.
PASCAL-Context [26] provides whole scene annota-
tions of PASCAL VOC 2010. While there are over 400 dis-
tinct classes, we follow the 59 class task deﬁned by [26] that
picks the most frequent classes. We train and evaluate on
the training and val sets respectively. In Table 6, we com-
pare to the joint object + stuff variation of Convolutional
Feature Masking [3] which is the previous state-of-the-art
on this task. FCN-8s scores 35.1 mean IU for an 11% rela-
tive improvement.
Changelog
The arXiv version of this paper is kept up-to-date with
corrections and additional relevant material. The following
gives a brief history of changes.Table 6. Results on PASCAL-Context. CFM is the best result of
[3] by convolutional feature masking and segment pursuit with the
VGG net. O2Pis the second order pooling method [1] as reported
in the errata of [26]. The 59 class task includes the 59 most fre-
quent classes while the 33 class task consists of an easier subset
identiﬁed by [26].
59 classpixel
acc.mean
acc.mean
IUf.w.
IU
O2P - - 18.1 -
CFM - - 31.5 -
FCN-32s 63.8 42.7 31.8 48.3
FCN-16s 65.7 46.2 34.8 50.7
FCN-8s 65.9 46.5 35.1 51.0
33 class
O2P - - 29.2 -
CFM - - 46.1 -
FCN-32s 69.8 65.1 50.4 54.9
FCN-16s 71.8 68.0 53.4 57.5
FCN-8s 71.8 67.6 53.5 57.7
v2Add Appendix A giving upper bounds on mean IU and
Appendix B with PASCAL-Context results. Correct PAS-
CAL validation numbers (previously, some val images were
included in train), SIFT Flow mean IU (which used an in-
appropriately strict metric), and an error in the frequency
weighted mean IU formula. Add link to models and update
timing numbers to reﬂect improved implementation (which
is publicly available).
References
[1] J. Carreira, R. Caseiro, J. Batista, and C. Sminchisescu. Se-
mantic segmentation with second-order pooling. In ECCV ,
2012. 9
[2] D. C. Ciresan, A. Giusti, L. M. Gambardella, and J. Schmid-
huber. Deep neural networks segment neuronal membranes
in electron microscopy images. In NIPS , pages 2852–2860,
2012. 1, 2, 4, 7
[3] J. Dai, K. He, and J. Sun. Convolutional feature mask-
ing for joint object and stuff segmentation. arXiv preprint
arXiv:1412.1283 , 2014. 9
[4] J. Donahue, Y . Jia, O. Vinyals, J. Hoffman, N. Zhang,
E. Tzeng, and T. Darrell. DeCAF: A deep convolutional acti-
vation feature for generic visual recognition. In ICML , 2014.
1, 2
[5] D. Eigen, D. Krishnan, and R. Fergus. Restoring an image
taken through a window covered with dirt or rain. In Com-
puter Vision (ICCV), 2013 IEEE International Conference
on, pages 633–640. IEEE, 2013. 2
[6] D. Eigen, C. Puhrsch, and R. Fergus. Depth map prediction
from a single image using a multi-scale deep network. arXiv
preprint arXiv:1406.2283 , 2014. 2
[7] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn,
and A. Zisserman. The PASCAL Visual Object Classes
Challenge 2011 (VOC2011) Results. http://www.pascal-
network.org/challenges/VOC/voc2011/workshop/index.html.
4
[8] C. Farabet, C. Couprie, L. Najman, and Y . LeCun. Learning
hierarchical features for scene labeling. Pattern Analysis and
Machine Intelligence, IEEE Transactions on , 2013. 1, 2, 4,
7, 8
[9] P. Fischer, A. Dosovitskiy, and T. Brox. Descriptor matching
with convolutional neural networks: a comparison to SIFT.
CoRR , abs/1405.5769, 2014. 1
[10] L. Florack, B. T. H. Romeny, M. Viergever, and J. Koen-
derink. The gaussian scale-space paradigm and the multi-
scale local jet. International Journal of Computer Vision ,
18(1):61–75, 1996. 5
[11] Y . Ganin and V . Lempitsky. N4-ﬁelds: Neural network near-
est neighbor ﬁelds for image transforms. In ACCV , 2014. 1,
2, 7
[12] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich fea-
ture hierarchies for accurate object detection and semantic
segmentation. In Computer Vision and Pattern Recognition ,
2014. 1, 2, 7
[13] S. Gupta, P. Arbelaez, and J. Malik. Perceptual organization
and recognition of indoor scenes from RGB-D images. In
CVPR , 2013. 8
[14] S. Gupta, R. Girshick, P. Arbelaez, and J. Malik. Learning
rich features from RGB-D images for object detection and
segmentation. In ECCV . Springer, 2014. 1, 2, 8
[15] B. Hariharan, P. Arbelaez, L. Bourdev, S. Maji, and J. Malik.
Semantic contours from inverse detectors. In International
Conference on Computer Vision (ICCV) , 2011. 7
[16] B. Hariharan, P. Arbel ´aez, R. Girshick, and J. Malik. Simul-
taneous detection and segmentation. In European Confer-
ence on Computer Vision (ECCV) , 2014. 1, 2, 4, 5, 7, 8
[17] K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling
in deep convolutional networks for visual recognition. In
ECCV , 2014. 1, 2
[18] Y . Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Gir-
shick, S. Guadarrama, and T. Darrell. Caffe: Convolu-
tional architecture for fast feature embedding. arXiv preprint
arXiv:1408.5093 , 2014. 7
[19] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classiﬁcation with deep convolutional neural networks. In
NIPS , 2012. 1, 2, 3, 5
[20] Q. V . Le, R. Monga, M. Devin, K. Chen, G. S. Corrado,
J. Dean, and A. Y . Ng. Building high-level features using
large scale unsupervised learning. In ICML , 2012. 3
[21] Y . LeCun, B. Boser, J. Denker, D. Henderson, R. E. Howard,
W. Hubbard, and L. D. Jackel. Backpropagation applied to
hand-written zip code recognition. In Neural Computation ,
1989. 2, 3
[22] Y . A. LeCun, L. Bottou, G. B. Orr, and K.-R. M ¨uller. Ef-
ﬁcient backprop. In Neural networks: Tricks of the trade ,
pages 9–48. Springer, 1998. 7
[23] C. Liu, J. Yuen, and A. Torralba. Sift ﬂow: Dense correspon-
dence across scenes and its applications. Pattern Analysis
and Machine Intelligence, IEEE Transactions on , 33(5):978–
994, 2011. 8[24] J. Long, N. Zhang, and T. Darrell. Do convnets learn corre-
spondence? In NIPS , 2014. 1
[25] O. Matan, C. J. Burges, Y . LeCun, and J. S. Denker. Multi-
digit recognition using a space displacement neural network.
InNIPS , pages 488–495. Citeseer, 1991. 2
[26] R. Mottaghi, X. Chen, X. Liu, N.-G. Cho, S.-W. Lee, S. Fi-
dler, R. Urtasun, and A. Yuille. The role of context for object
detection and semantic segmentation in the wild. In Com-
puter Vision and Pattern Recognition (CVPR), 2014 IEEE
Conference on , pages 891–898. IEEE, 2014. 9
[27] F. Ning, D. Delhomme, Y . LeCun, F. Piano, L. Bottou, and
P. E. Barbano. Toward automatic phenotyping of developing
embryos from videos. Image Processing, IEEE Transactions
on, 14(9):1360–1371, 2005. 1, 2, 4, 7
[28] P. H. Pinheiro and R. Collobert. Recurrent convolutional
neural networks for scene labeling. In ICML , 2014. 1, 2,
4, 7, 8
[29] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus,
and Y . LeCun. Overfeat: Integrated recognition, localization
and detection using convolutional networks. In ICLR , 2014.
1, 2, 3, 4
[30] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. Indoor
segmentation and support inference from rgbd images. In
ECCV , 2012. 7
[31] K. Simonyan and A. Zisserman. Very deep convolu-
tional networks for large-scale image recognition. CoRR ,
abs/1409.1556, 2014. 1, 2, 3, 5
[32] C. Szegedy, W. Liu, Y . Jia, P. Sermanet, S. Reed,
D. Anguelov, D. Erhan, V . Vanhoucke, and A. Rabinovich.
Going deeper with convolutions. CoRR , abs/1409.4842,
2014. 1, 2, 3, 5
[33] J. Tighe and S. Lazebnik. Superparsing: scalable nonpara-
metric image parsing with superpixels. In ECCV , pages 352–
365. Springer, 2010. 8
[34] J. Tighe and S. Lazebnik. Finding things: Image parsing with
regions and per-exemplar detectors. In CVPR , 2013. 8
[35] J. Tompson, A. Jain, Y . LeCun, and C. Bregler. Joint training
of a convolutional network and a graphical model for human
pose estimation. CoRR , abs/1406.2984, 2014. 2
[36] L. Wan, M. Zeiler, S. Zhang, Y . L. Cun, and R. Fergus. Reg-
ularization of neural networks using dropconnect. In Pro-
ceedings of the 30th International Conference on Machine
Learning (ICML-13) , pages 1058–1066, 2013. 4
[37] R. Wolf and J. C. Platt. Postal address block location using
a convolutional locator network. Advances in Neural Infor-
mation Processing Systems , pages 745–745, 1994. 2
[38] M. D. Zeiler and R. Fergus. Visualizing and understanding
convolutional networks. In Computer Vision–ECCV 2014 ,
pages 818–833. Springer, 2014. 2
[39] N. Zhang, J. Donahue, R. Girshick, and T. Darrell. Part-
based r-cnns for ﬁne-grained category detection. In Com-
puter Vision–ECCV 2014 , pages 834–849. Springer, 2014.
1


Deep convolutional filter banks for texture recognition and segmentation

Summary:
Research in texture recognition often concentrates on the problem of material
recognition in uncluttered conditions, an assumption rarely met by
applications. In this work we conduct a first study of material and describable
texture at- tributes recognition in clutter, using a new dataset derived from
the OpenSurface texture repository. Motivated by the challenge posed by this
problem, we propose a new texture descriptor, D-CNN, obtained by Fisher Vector
pooling of a Convolutional Neural Network (CNN) filter bank. D-CNN
substantially improves the state-of-the-art in texture, mate- rial and scene
recognition. Our approach achieves 82.3% accuracy on Flickr material dataset
and 81.1% accuracy on MIT indoor scenes, providing absolute gains of more than
10% over existing approaches. D-CNN easily trans- fers across domains without
requiring feature adaptation as for methods that build on the fully-connected
layers of CNNs. Furthermore, D-CNN can seamlessly incorporate multi-scale
information and describe regions of arbitrary shapes and sizes. Our approach is
particularly suited at lo- calizing stuff categories and obtains
state-of-the-art re- sults on MSRC segmentation dataset, as well as promising
results on recognizing materials and surface attributes in clutter on the
OpenSurfaces dataset.

Mircea Cimpoi
Subhransu Maji
Andrea Vedaldi

Accepted to CVPR15

Deep Filter Banks for Texture Recognition and Segmentation
Mircea Cimpoi
University of Oxford
mircea@robots.ox.ac.ukSubhransu Maji
University of Massachusetts, Amherst
smaji@cs.umass.eduAndrea Vedaldi
University of Oxford
vedaldi@robots.ox.ac.uk
Abstract
Research in texture recognition often concentrates on the
problem of material recognition in uncluttered conditions,
an assumption rarely met by applications. In this work we
conduct a ﬁrst study of material and describable texture at-
tributes recognition in clutter, using a new dataset derived
from the OpenSurface texture repository. Motivated by the
challenge posed by this problem, we propose a new texture
descriptor, FV-CNN, obtained by Fisher Vector pooling of a
Convolutional Neural Network (CNN) ﬁlter bank. FV-CNN
substantially improves the state-of-the-art in texture, mate-
rial and scene recognition. Our approach achieves 79.8%
accuracy on Flickr material dataset and 81% accuracy on
MIT indoor scenes, providing absolute gains of more than
10% over existing approaches. FV-CNN easily transfers
across domains without requiring feature adaptation as for
methods that build on the fully-connected layers of CNNs.
Furthermore, FV-CNN can seamlessly incorporate multi-
scale information and describe regions of arbitrary shapes
and sizes. Our approach is particularly suited at localiz-
ing “stuff” categories and obtains state-of-the-art results
on MSRC segmentation dataset, as well as promising results
on recognizing materials and surface attributes in clutter on
the OpenSurfaces dataset.
1. Introduction
Texture is ubiquitous and provides useful cues of mate-
rial properties of objects and their identity, especially when
shape is not useful. Hence, a signiﬁcant amount of effort in
the computer vision community has gone into recognizing
texture via tasks such as texture perception [1, 2, 12, 13]
and description [7, 11], material recognition [26, 36, 37],
segmentation [20, 29], and even synthesis [9, 43].
Perhaps the most studied task in texture understanding
is the one of material recognition, as captured in bench-
marks such as CuRET [8], KTH-TIPS [5], and, more re-
cently, FMD [38]. However, while at least the FMD dataset
contains images collected from the Internet, vividly dubbed
“images in the wild”, all these datasets make the simplifying
banded
 blotchy
 chequered
 grid
marbled
 paisley
 paisley
 wrinkled
brick
 ceramic
 carpet
 fabric
Figure 1. Texture recognition in clutter . Example of top re-
trieved texture segments by attributes (top two rows) and materials
(bottom) in the OpenSurfaces dataset.
assumption that textures ﬁll images. Thus, they are not nec-
essarily representative of the signiﬁcantly harder problem
of recognising materials in natural images, where textures
appear in clutter. Building on a recent dataset collected by
the computer graphics community, the ﬁrst contribution of
this paper is a large-scale analysis of material and percep-
tual texture attribute recognition and segmentation in clut-
ter(Fig. 1 and Sect. 2).
Motivated by the challenge posed by recognising texture
in clutter, we develop a new texture descriptor. In the sim-
plest terms a texture is characterized by the arrangement of
local patterns, as captured in early works [26, 41] by the
distribution of local “ﬁlter bank” responses. These ﬁlter
banks were designed to capture edges, spots and bars at
different scales and orientations. Typical combinations of
the ﬁlter responses, identiﬁed by vector quantisation, were
used as the computational basis of the “textons” proposed
by Julesz [22]. Texton distributions were the early versions
of “bag-of-words” representations, a dominant approach in
recognition in the early 2000s, since then improved by new
pooling schemes such as soft-assignment [27, 42, 48] and
Fisher Vectors (FVs) [32]. Until recently, FV with SIFT
features [28] as a local representation was the state-of-the-
art method for recognition, not only for textures, but for
objects and scenes too.
Later, however, Convolutional Neural Networks (CNNs)
1arXiv:1411.6836v2  [cs.CV]  9 Jul 2015
have emerged as the new state-of-the-art for recognition, ex-
empliﬁed by remarkable results in image classiﬁcation [23],
detection [14] and segmentation [16] on a number of widely
used benchmarks. Key to their success is the ability to lever-
age large labelled datasets to learn increasingly complex
transformations of the input to capture invariances. Impor-
tantly, CNNs pre-trained on such large datasets have been
shown [6, 14, 30] to contain general-purpose feature extrac-
tors, transferrable to many other domains.
Domain transfer in CNNs is usually achieved by using
as features the output of a deep, fully-connected layer of
the network. From the perspective of textures, however,
this choice has three drawbacks. The ﬁrst one (I) is that,
while the convolutional layers are akin to non-linear ﬁlter
banks, the fully connected layers capture their spatial lay-
out. While this may be useful for representing the shape of
an object, it may not be as useful for representing texture. A
second drawback (II) is that the input to the CNN has to be
of ﬁxed size to be compatible with the fully connected lay-
ers, which requires an expensive resizing of the input image,
particularly when features are computed for many different
regions [14, 15]. A third and more subtle drawback (III) is
that deeper layers may be more domain-speciﬁc and there-
fore potentially less transferrable than shallower layers.
The second contribution of this paper is FV-CNN
(Sect. 3), a pooling method that overcomes these draw-
backs . The idea is to regard the convolutional layers of a
CNN as a ﬁlter bank and build an orderless representation
using FV as a pooling mechanism, as is commonly done
in the bag-of-words approaches. Although the suggested
change is simple, the approach is remarkably ﬂexible and
effective. First, pooling is orderless and multi-scale, hence
suitable for textures. Second, any image size can be pro-
cessed by convolutional layers, avoiding costly resizing op-
erations. Third, convolutional ﬁlters, pooled by FV-CNN,
are shown to transfer more easily than fully-connected ones
even without ﬁne-tuning. While other authors [15, 18] have
recently proposed alternative pooling strategies for CNNs,
we show that our method is more natural, faster and often
signiﬁcantly more accurate.
Thethird contribution of the paper is a thorough evalu-
ation of these descriptors on a variety of benchmarks, from
textures to objects (Sect. 4). In textures, we evaluate mate-
rial and describable attributes recognition and segmentation
on new datasets derived from OpenSurfaces (Sect. 2). When
used with linear SVMs, FV-CNN improves the state of the
art on texture recognition by a signiﬁcant margin. Like tex-
tures, scenes are also weakly structured and a bag-of-words
representation is effective. FV-CNN obtains 81.1% accu-
racy on the MIT indoor scenes dataset [34], signiﬁcantly
outperforming the current state-of-the-art of 70.8% [47].
What is remarkable is that, where [47] ﬁnds that CNNs
trained on scene recognition data perform better than CNNs
trained on an object domain (ImageNet), when used in FV-CNN not only is there an overall performance improve-
ment, but the domain-speciﬁc advantage is entirely removed
(Tab. 3). This indicates that FV-CNN are in fact better at do-
main transfer. Our method also matches the previous best in
PASCAL VOC 2007 classiﬁcation dataset providing mea-
surable boost over CNNs and closely approaches competi-
tor methods on CUB 2010-2011 datasets when ground-truth
object bounding boxes are given.
FV-CNN can be used for describing regions by simply
pooling across pixels within the region. Combined with
a low-level segmentation algorithm this sufﬁces to local-
ize textures within images. This approach is similar to a
recently proposed method called “R-CNN” for localizing
objects [14]. However, in contrast to it we do not need re-
peated evaluations of the CNN since the convolutional fea-
tures can be computed just once for the entire image and
pooled differently. This makes FV-CNN not only faster,
but also as experiments suggest, much more accurate at
texture localization. We achieve state of the art results on
the MSRC segmentation dataset using a simple scheme of
classifying “superpixels” obtaining an accuracy of 87.0%
(previous best 86.5%). The corresponding R-CNN obtains
57.7% accuracy. Segmentation results are promising in the
OpenSurfaces dataset [4] as well.
Finally, we analyze the utility of different network layers
and architectures as ﬁlter banks, concluding that: SIFT is
competitive only with the ﬁrst few layers of a CNN (Fig. 4)
and that signiﬁcant improvement to the underlying CNN ar-
chitecture, such as the ones achieved by the very deep mod-
els of Simonyan and Zisserman [39], directly translate into
much better ﬁlter banks for texture recognition.
2. Texture recognition in clutter
A contribution of this work is the analysis of materials
and texture attributes in realistic imaging conditions. Ear-
lier datasets such as KTH-TIPS were collected in controlled
conditions, which makes their applicability to natural im-
ages unclear. More recent datasets such as FMD and DTD
remove this limitation by building on images downloaded
from the Internet, dubbed images “in the wild”. However,
in these datasets texture always ﬁll the ﬁeld of view of the
camera. In this paper we remove this limitation by exper-
imenting for the ﬁrst time with a large dataset of textures
collected in the wild and in cluttered conditions.
In particular, we build on the Open Surfaces (OS) dataset
that was recently introduced by Bell et al. [4] in computer
graphics. OS comprises 25,357 images, each containing a
number of high-quality texture/material segments. Many of
these segments are annotated with additional attributes such
as the material name, the viewpoint, the BRDF, and the ob-
ject class. Not all segments have a complete set of annota-
tions; the experiments in this paper focus on the 58,928 that
contain material names. Since material classes are highly
unbalanced, only the materials that contain at least 400 ex-
amples are considered. This result in 53,915 annotated
material segments in 10,422 images spanning 22 different
classes.1Images are split evenly into training, validation,
and test subsets with 3,474 images each. Segment sizes are
highly variable, with half of them being relatively small,
with an area smaller than 6464pixels. While the lack
of exhaustive annotations makes it impossible to deﬁne a
complete background class, several less common materials
(including for example segments that annotators could not
assign to a material) are merged in an “other” class that acts
as pseudo-background.
In order to study perceptual properties as well as ma-
terials, we augment the OS dataset with some of the 47
attributes from the DTD [7]. Since the OS segments do
not trigger with sufﬁcient frequency all the 47 attributes,
the evaluation is restricted to eleven of them for which it
was possible to identify at least 100 matching segments.2
The attributes were manually labelled in the 53,915 seg-
ments retained for materials. We refer to this data as
OSA. The complete list of images, segments, labels, and
splits are available at http://www.robots.ox.ac.
uk/˜vgg/data/wildtex/ .
3. Method
This section describes the methodological contributions
of this paper: region description and segmentation.
3.1. Region description
This section introduces a number of visual descriptors
suitable to model the appearance of image regions. Texture
is traditionally described by orderless pooling of ﬁlter bank
responses as, unlike in objects, the overall shape informa-
tion is usually unimportant. However, small under-sampled
textures may beneﬁt if recognized in the context of an ob-
ject. Thus, the primacy of orderless pooling may not always
hold in the recognition of textures in natural conditions.
In order to explore the interplay between shape and or-
derless pooling, we evaluate two corresponding region de-
scriptors: FC-CNN for shape and FV-CNN for texture.
Both descriptors are based on the same CNN features [23]
obtained from an off-the-shelf CNN pre-trained on the Ima-
geNet ILSVRC 2012 data as suggested in [6, 21, 35]. Since
the underlying CNN is the same, it is meaningful to com-
pare FC- and FV-CNN directly.
1The classes and corresponding number of example segments are:
brick (610), cardboard (423), carpet/rug (1,975), ceramic (1,643), con-
crete (567), fabric/cloth (7,484), food (1,461), glass (4,571), granite/-
marble (1,596), hair (443), other (2,035), laminate (510), leather (957),
metal (4,941), painted (7,870), paper/tissue (1,226), plastic/clear (586),
plastic/opaque (1,800), stone (417), tile (3,085), wallpaper (483), wood
(9,232).
2These are: banded, blotchy, chequered, ﬂecked, gauzy, grid, marbled,
paisley, pleated, stratiﬁed, wrinkled.Object descriptor: FC-CNN. The FC-CNN descriptor is
obtained by extracting as features the output of the penul-
timate Fully-Connected (FC) layer of a CNN, including
the non-linear gating function, applied to the input image.
This can be considered an object descriptor because the
fully connected layers allow FC-CNN to capture the overall
shape of the object contained in the region. FC-CNN is ap-
plied to an image region R(which may be the whole image)
by warping the bounding box enclosing R(plus a 10% bor-
der) to a square of a ﬁxed size matching the default CNN
input geometry, obtaining the same R-CNN descriptor in-
troduced by Girshick et al. [14] as a state-of-the-art object
detector in the PASCAL VOC [10] data.
Texture descriptor: FV-CNN. The FV-CNN descriptor
is inspired by the state-of-the-art texture descriptors of [7]
based on the Fisher Vector (FV). Differently from FC-CNN,
FV pools local features densely within the described regions
removing global spatial information , and is therefore more
apt at describing textures than objects. Here FV is computed
on the output of a single (last) convolutional layer of the
CNN, but we compared features from other layers as well
(Sect 4.4). By avoiding the computation of the fully con-
nected layers, the input image does not need to be rescaled
to a speciﬁc size; in fact, the dense convolutional features
are extracted at multiple scales and pooled into a single FV
just like for SIFT. The pooled convolutional features are ex-
tracted immediately after the last linear ﬁltering operator
and are not otherwise normalised.
The FV-CNN descriptor is related to the one proposed
by Gong et al. [15]. There VLAD pooling, which is simi-
lar to FV , is applied to FC-CNN-like descriptors extracted
from densely sampled image windows. A key difference of
FV-CNN is that dense features are extracted from the con-
volutional rather than fully-connected layers. This is more
natural, signiﬁcantly more efﬁcient (as it does not require
recomputing the network for each extracted local descrip-
tor) and, as shown in Sect. 4, more accurate.
3.2. Region segmentation
This section discusses our method to automatically par-
tition an image into a number of recognisable regions. In-
spired by Cimpoi et al. [7] that successfully ported object
description methods to texture descriptors, here we propose
a segmentation technique inspired by object detection. An
increasingly popular method for object detection, followed
for example by R-CNN [14], is to ﬁrst propose a number of
candidate object regions using low-level image cues, and
then verifying a shortlist of such regions using a power-
ful classiﬁer. Applied to textures, this requires a low-level
mechanism to generate textured region proposals, followed
by a region classiﬁer. A key advantage of this approach
is that it allows applying object- (FC-CNN) and texture-
like (FV-CNN) descriptors alike. After proposal classiﬁca-
tion, each pixel can be assigned more than one label; this is
solved with simple voting schemes, also inspired by object
detections methods.
The paper explores two such region generation methods:
the crisp regions of [19] and the multi-scale combinatorial
grouping of [3]. In both cases, region proposals are gener-
ated using low-level image cues, such as colour or texture
consistency, as speciﬁed by the original methods. It would
of course be possible to incorporate FC-CNN and FV-CNN
among these energy terms to potentially strengthen the re-
gion generation mechanism itself. However, this contra-
dicts partially the logic of the scheme, which breaks down
the problem into cheaply generating tentative segmenta-
tions and then verifying them using a more powerful (and
likely expensive) model. Furthermore, and more impor-
tantly, these cues focus on separating texture instances , as
presented in each particular image, whereas FC-CNN and
FV-CNN are meant to identify a texture class. It is reason-
able to expect instance-speciﬁc cues (say the colour of a
painted wall) to be better for segmentation.
4. Results
This section evaluates the proposed region recognition
methods for classifying and segmenting materials, describ-
able texture properties, and higher-level object categories.
Sect. 4.1 evaluates the classiﬁcation task by assessing how
well regions can be classiﬁed given that their true extent is
known and Sect. 4.3 evaluates both classiﬁcation and seg-
mentation . The rest of the section introduces the evaluation
benchmarks and technical details of the representations.
Datasets. The evaluation considers three texture recogni-
tion benchmarks other than OS (Sect. 2). The ﬁrst one is the
Flickr Material Dataset (FMD) [37], a recent benchmark
containing 10 material classes. The second one is the De-
scribable Texture Datasets (DTD) [7], which contains tex-
ture images jointly annotated with 47 describable attributes
drawn from the psychological literature. Both FMD and
DTD contain images “in the wild”, i.e. collected in uncon-
trolled conditions. However, differently from OS, these im-
ages are uncluttered. The third texture dataset is KTH-TIPS-
2b[5, 17], containing a number of example images for each
of four samples of 11 material categories. For each ma-
terial, images of one sample are used for training and the
remaining for testing.
Object categorisation is evaluated in the PASCAL VOC
2007 [10] dataset, containing 20 object categories, any
combination of which may be associated to any of the
benchmark images. Scene categorisation uses the MIT In-
door [34] dataset, containing 67 indoor scene classes. Fine-
grained categorisation uses the Caltech/UCSD Bird dataset
(CUB) [45], containing images of 200 bird species.
Note that some of these datasets come with ground truth
region/object localisation. The +R sufﬁx will be appendedto a dataset to indicate that this information is used both at
training and testing time. For example, OS means that seg-
mentation is performed automatically at test time, whereas
OS+R means that ground-truth segmentations are used.
Evaluation measures. For each dataset the correspond-
ing standard evaluator protocols and accuracy measures are
used. In particular, for FMD, DTD, MIT Indoor, CUB,
and OS+R, evaluation uses average classiﬁcation accuracy,
per-segment/image and normalized for each class. When
evaluating the quality of a segmentation algorithm, how-
ever, one must account for the fact that in most datasets,
and in OS and MSRC in particular, not all pixels are la-
belled. In this case, accuracy is measured per-pixel rather
than per-segment, ignoring all pixels that are unlabelled in
the ground truth. For MSRC, furthermore, accuracy is nor-
malised across all pixels regardless of their category. For
OSA, since some segments may have more than one label,
we are reporting mAP, following the standard procedure for
multi-label datasets. Finally, PASCAL VOC 2007 classiﬁ-
cation uses mean average precision (mAP), computed using
the TRECVID 11-point interpolation [10].3
Descriptor details. FC-CNN and FV-CNN build on the
pre-trained VGG-M [6] model as this performs better than
other popular models such as [21] while having a sim-
ilar computational cost. This network results in 4096-
dimensional FC-CNN features and 512-dimensional local
features for FV-CNN computation. The latter are pooled
into a FV representation with 64 Gaussian components,
resulting in 65K-dimensional descriptors. While the FV-
CNN dimensionality is much higher than the 4K dimen-
sions of FC-CNN, the FV is known to be highly redun-
dant and can be typically compressed by one order of mag-
nitude without appreciable reduction in the classiﬁcation
performance [31], so the effective dimensionality of FC-
and FV-CNN is likely comparable. We veriﬁed that by
PCA-reducing FV to 4096 dimensions and observing only
a marginal reduction in classiﬁcation performance in the
PASCAL VOC object recognition task described below. In
addition to VGG-M, the recent state-of-the art VGG-VD
(very deep with 19 layers) model of Simonyan and Zisser-
man [39] is also evaluated.
Due to the similarity between FV-CNN and the dense
SIFT FV descriptors used for texture recognition in [7],
the latter is evaluated as well. Since SIFT descriptors
are smaller (128-dimensional) than the convolutional ones
(512-dimensional), a larger number of Gaussian compo-
nents (256) are used to obtain FV descriptors with a com-
parable dimensionality. The SIFT descriptors support is
3232pixels at the base scale.
In order to make results comparable to [7], we use
the same settings whenever possible. FV-CNN and D-
SIFT compute features after rescaling the image by factors
3The deﬁnition of AP was changed in later versions of the benchmark.
dataset meas.IFVVGG-M VGG-VD FV-SIFTSoA(%) FC FV FC+FV FC FV FC+FV FC+FV-VD
(a)KTH-T2b acc 70.82.7712.373.34.773.94.975.41.581.82.581.12.4 81.52.0 76.02.9[40]
FMD acc 59.81.670.31.873.52.076.61.977.41.879.81.882.41.5 82.21.4 57.71.7[33, 37]
OS+R acc 39.8 41.3 52.5 54.9 43.4 59.5 60.9 58.7 –
(b)DTD acc 58.61.258.80.866.81.669.81.162.90.872.31.074.71.0 75.50.8 –
OSA+R mAP 56.5 54.3 65.2 67.9 49.7 67.2 67.9 68.2 –
(d)MSRC+R acc 85.7 85.0 95.4 96.9 79.4 97.7 98.8 99.1 –
MSRC+R msrc-acc 92.0 84.0 97.6 98.1 82.0 99.2 99.6 99.5 –
VOC07 mAP11 59.9 76.8 76.4 79.5 81.7 84.9 85.1 84.5 85.2 [44]
MIT Indoor acc 54.9 62.5 74.2 74.4 67.6 81.0 80.3 80.0 70.8 [47]
CUB acc 17.5 46.1 49.9 54.9 54.6 66.7 67.3 65.4 73.9 (62.8) [46]
CUB+R acc 27.7 56.5 65.5 68.1 62.8 73.0 74.9 73.6 76.37 [46]
Table 1. Evaluation of texture descriptors. The table compares FC-CNN, FV-CNN on two networks trained on ImageNet – VGG-M and
VGG-VD, and IFV on dense SIFT. We evaluated these descriptors on (a) material datasets (FMD, KTH-T2b, OS+R), (b) texture attributes
(DTD, OSA+R) and (c) general categorisation datasets (MSRC+R,VOC07,MIT Indoor) and ﬁne grained categorisation (CUB, CUB+R).
For this experiment the region support is assumed to be known (and equal to the entire image for all the datasets except OS+R and MSRC+R
and for CUB+R, where it is set to the bounding box of a bird). () using a model without parts. Best results are marked in bold.
VGG-M VGG-VD
dataset measure (%) FC-CNN FV-CNN FV+FC-CNN FC-CNN FV-CNN FC+FV-CNN SoA
OS pp-acc 36.3 48.7 (46.9) 50.5 38.8 55.4 (55.7) 55.2 –
MSRC msrc-acc 56.1 82.3 75.5 57.7 87.0 80.4 86.5 [24]
Table 2. Segmentation and recognition using crisp region proposals of materials (OS) and things & stuff (MSRC). Per-pixel accuracies are
reported, using the MSRC variant (see text) for the MSRC dataset. Results using MCG proposals [3] are reported in brackets for FV-CNN.
2s; s=3;2:5; : : :1:5(but, for efﬁciency, discarding
scales that would make the image larger than 10242pix-
els). Before pooling descriptors with a FV , these are usually
de-correlated by using PCA. Here PCA is applied to SIFT,
additionally reducing its dimension to 80, as this was em-
pirically shown to improve the overall recognition perfor-
mance. However, PCA is not applied to the convolutional
features in FV-CNN as in this case results were worse.
Learning details. The region descriptors (FC-CNN, FV-
CNN, and D-SIFT) are classiﬁed using 1-vs-rest Support
Vector Machine (SVM) classiﬁers. Prior to learning, de-
scriptors are L2normalised and the learning constant set
toC= 1. This is motivated by the fact that, after data
normalisation, the exact choice of Chas a negligible effect
on performance. Furthermore, the accuracy of the 1-vs-rest
classiﬁcation scheme is improved by recalibrating the SVM
scores after training, by scaling the SVM weight vector and
bias such that the median scores of the negative and positive
training samples for each class are mapped respectively to
the values1and1.
4.1. Region recognition: textures
This and the following section evaluate region recogni-
tion assuming that the ground-truth region Ris known (Ta-
ble 1), for example because it ﬁlls the entire image. This
section focuses on textures (materials and perceptual at-
tributes), while the next on objects and scenes.
Texture recognition without clutter. This experimentevaluates the performance of FC-CNN, FV-CNN, D-SIFT,
and their combinations in standard texture recognition
benchmarks such as FMD, KTH-TIPS-2, and DTD. FC-
CNN is roughly equivalent to the DeCAF method used
in [7] for this data as regions ﬁll images; however, while the
performance of our FC-CNN is similar in KTH ( 70%),
it is substantially better in FMD ( 60:7%!70:4%accu-
racy) and DTD ( 54:8%!58:7%). This likely is caused by
the improved underlying CNN, an advantage which is more
obvious in FMD and DTD that are closer to object recogni-
tion than KTH. FV-CNN performs within 2%in FMD and
KTH but substantially better for DTD ( 58:7%!66:6%).
D-SIFT is comparable in performance to FC-CNN in DTD
and KTH, but substantially worse ( 70:4%!59:2%) in
FMD. Our conclusion is that, even when textures ﬁll the
input image as in these benchmarks, orderless pooling in
FV-CNN and D-SIFT can be either the same or substan-
tially better than the pooling operated in the fully-connected
layers by FC-CNN.
Combining FC- and FV-CNN improves performance in
all datasets by 13%. While this combination is al-
ready signiﬁcantly above the state-of-the-art in DTD and
FMD ( +2:6%=11:2%), the method of [7] still outperforms
these descriptors in KTH. However, replacing VGG-M with
VGG-VD signiﬁcantly improves the performance in all
cases – a testament to the power of deep features. In partic-
ular, the best method FC+FV-CNN-VD, improves the state
of the art by at least 6%in all datasets. Interestingly, this is
obtained by using a single low-level feature type as FC- and
FV-CNN build on the same convolutional features. Adding
D-SIFT results in at most 1%improvement, and in some
cases it slightly degrades performance.
Texture recognition in clutter. The advantage of FV-CNN
over FC-CNN is much larger when textures do not ﬁll the
image but are extracted from clutter. In OS+R (Sect. 2),
material recognition accuracy starts at about 46% for both
FC-CNN and D-SIFT; however, FV-CNN improves this by
more than 11% (46:5%!58:1%). The combination of FC-
and FV-CNN improves results further by 2%, but adding
SIFT deteriorates performance. With the very deep CNN
conclusions are similar; however, switching to VGG-VD
barely affects the FC-CNN performance ( 46:5!48:0%),
but strongly affects the one of FV-CNN ( 58:1%!65:1%).
This conﬁrms that FC-CNN, while excellent in object de-
tection, is not a very good descriptor for classifying textured
regions. Results in OSA+R for texture attribute recognition
(Sect. 2) and in MSRC+R for semantic segmentation are
analogous; it is worth noting that, when ground-truth seg-
ments are used in this experiment, the best model achieves
a nearly perfect 99:7%classiﬁcation rate in MSRC.
4.2. Region recognition: objects and scenes
This section shows that the FV-CNN descriptor, despite
its orderless nature that make it an excellent texture descrip-
tor, excels at object and scene recognition as well. In the re-
mainder of the section, and unless otherwise noted, region
descriptors are applied to images as a whole by considering
these single regions.
FV-CNN vs FC-CNN. As seen in Table 1, in PASCAL
VOC and MIT Indoor the FC-CNN descriptor performs
very well but in line with previous results for this class of
methods [6]. FV-CNN performs similarly to FC-CNN in
PASCAL VOC, but substantially better ( +5% for VGG-M
and +13% for VGG-VD) in MIT Indoor. As further dis-
cussed below, this is an example of the ability of FV-CNN
to transfer between domains better than FC-CNN. The gap
between FC-CNN and FV-CNN is the highest for the very
deep VGG-VD models (68.1% !81.1%), a trend also ex-
hibited by other datasets as seen in Tab. 1. In the CUB
dataset, FV-CNN signiﬁcantly outperforms FC-CNN both
whether the descriptor is computed from the whole image
(CUB) or from the bird bounding box (CUB+R). In the lat-
ter case, the difference is very large ( +1014%).
Comparison with alternative pooling methods. FV-CNN
is related to the method of [15], which uses a similar
underlying CNN and VLAD instead of FV for pooling.
Notably, however, FV-CNN results on MIT Indoor are
markedly better than theirs for both VGG-M and VGG-
VD ( 68:8%!73:5%=81:1%resp.) and marginally bet-
ter (68:8%!69:1%) when the same CAFFE CNN is used
(Tab. 3). The key difference is that FV-CNN pools convolu-Accuracy (%)
CNN FC-CNN FV-CNN FC+FV-CNN
PLACES 65.0 67.6 73.1
CAFFE 58.6 69.7 71.6
VGG-M 62.5 74.2 74.4
VGG-VD 67.6 81.0 80.3
Table 3. Accuracy of various CNNs on the MIT indoor dataset.
tional features, whereas [15] pools fully-connected descrip-
tors extracted from square image patches. Thus, even with-
out spatial information as used by [15], FV-CNN is not only
substantially faster, but at least as accurate. Using the same
settings, that is, the same net and the same three scales, our
approach results in an 8.5 speedup.
Comparison with the state-of-the-art. The best result ob-
tained in PASCAL VOC is comparable to the current state-
of-the-art set by the deep learning method of [44] ( 85:2%!
85:0%), but using a much more straightforward pipeline. In
MIT Places our best performance is also substantially supe-
rior ( +10% ) to the current state-of-the-art using deep con-
volutional networks learned on the MIT Place dataset [47]
(see also below). In the CUB dataset, our best performance
is a little short (3%) of the state-of-the-art results of [46].
However, [46] uses a category-speciﬁc part detector and
corresponding part descriptor as well as a CNN ﬁne-tuned
on the CUB data; by contrast, FV-CNN and FC-CNN are
used here as global image descriptors which, furthermore,
are the same for all the datasets considered . Compared to
the results of [46] without part-based descriptors (but still
using a part-based object detector), our global image de-
scriptors perform substantially better ( 62:1%!69:1%).
We conclude that FV-CNN is a very strong descriptor.
Results are usually as good, and often signiﬁcantly better,
than FC-CNN. In most applications, furthermore, FV-CNN
is many times faster as it does not require evaluating the
CNN for each target image region. Finally, FC- and FV-
CNN can be combined outperforming the state-of-the-art in
many benchmarks.
Domain transfer. So far, the same underlying CNN fea-
tures, trained on ImageNet’s ILSVCR, were used for all
datasets. Here we investigate the effect of using domain-
speciﬁc features. To do so, we consider the PLACES [47],
trained to recognize places on a dataset of about 2.5 mil-
lion labeled images. [47] showed that, applied to the task of
scene recognition in MIT Indoor, these features outperform
similar ones trained on ILSVCR (denoted CAFFE [21] be-
low) – a fact explained by the similarity of domains. Below,
we repeat this experiment using FC- and FV-CNN descrip-
tors on top of VGG-M, VGG-VD, PLACES, and CAFFE.
Results are shown in Table 3. The FC-CNN results are
in line with those reported in [47] – in scene recognition
with FC-CNN the same CNN architecture performs better
if trained on the Places dataset instead of the ImageNet data
(58:6%!65:0%accuracy4). Nevertheless, stronger CNN
architectures such as VGG-M and VGG-VD can approach
and outperform PLACES even if trained on ImageNet data
(65:0%!63:0%=68:1%).
However, when it comes to using the ﬁlter banks with
FV-CNN, conclusions are very different. First, FV-CNN
outperforms FC-CNN in all cases, with substantial gains
up to 20% in correspondence of a domain transfer from
ImageNet to MIT Indoor. Second, the advantage of us-
ing domain-speciﬁc CNNs disappears . In fact, the same
CAFFE model that is 6.4% worse than PLACES with FC-
CNN, is actually 1.5% better when used in FV-CNN. The
conclusion is that FV-CNN appears to be immune, or at
least substantially less sensitive, to domain shifts.
Our tentative explanation of this surprising phenomenon
is that the convolutional layers are less committed to a spe-
ciﬁc dataset than the fully ones. Hence, by using those,
FV-CNN tends to be a more general than FC-CNN.
4.3. Texture segmentation
The previous section considered the problem of region
recognition when the region support is known at test time.
This section studies the problem of recognising regions
when their extent Risnotknown and also be estimated.
The ﬁrst experiment (Tab. 2) investigates the sim-
plest possible scheme: combining the region descriptors
of Sect. 4.1 with a general-purpose image segmentation
method, namely the crisp regions of [19]. Two datasets
are evaluated: OS for material recognition and MSRC for
things & stuff. Compared to OS+R, classifying crisp re-
gions results in a drop of about 5%points for all descrip-
tors. As this dataset is fairly challenging with best achiev-
able performance is 55:4%, this is a satisfactory result. But
it also illustrates that there is ample space for future im-
provements. In MSRC, the best accuracy is 87:0%, just a
hair above the best published result 86:5%[25]. Remark-
ably, these algorithms not use any dataset-speciﬁc train-
ing, nor CRF-regularised semantic inference: they simply
greedily classify regions as obtained from a general-purpose
segmentation algorithms. Qualitative segmentation results
(sampled at random) are given in Fig. 2 and 3.
Unlike crisp regions, the proposals of [3] are overlap-
ping and a typical image contains thousands of them. We
propose a simple scheme to combine prediction from mul-
tiple proposals. For each proposal we set its label to the
highest scoring class, and score to the highest score. We
then sort the proposals in the increasing order of their score
divided by their area and paste them one by one. This has
the effect of considering larger regions before smaller ones
and more conﬁdent regions after less ones for regions of
the same area. Results using FV-CNN shown in Tab. 2 in
4[47] report 68:3%for PLACES applied to MIT Indoor, a small differ-
ence explained by implementation details such as the fact that, for all the
methods, we do not perform data augmentation by jittering.
40444852566064687276Performance (%)
conv1
[7 x  7]
 96x256
conv2
[27 x  27]
256x128
conv3
[75 x  75]
512x 64
conv4
[107 x  107]
512x 64
conv5
[139 x  139]
512x 64(VGG−M) Filterbank Analysis

VOC07 (mAP11)
MIT Indoor (acc)
FMD (acc)
DTD (acc)
SIFTFigure 4. CNN ﬁlterbank analysis for VGG-M. Performance of
ﬁlter banks extracted from various layers network are shown on
various datasets. For each layer convf1; : : : ; 5gwe show, the size
of the receptive ﬁeld [ NN], and FBD, where FBis the size
of ﬁlter bank and Dis the dictionary size in the FV representation.
The performance using SIFT is shown in black plus (+) marks.
brackets (FC-CNN was too slow for our experiments). The
results are comparable to those using crisp regions, and we
obtain 55:7%accuracy on the OS dataset. Our initial at-
tempts at schemes such as non-maximum suppression of
overlapping regions that are quite successful for object seg-
mentation [16] performed rather poorly. We believe this is
because unlike objects, material information is fairly local-
ized and highly irregularly shaped in an image. However,
there is room for improvement by combining evidence from
multiple segmentations.
4.4. Convolutional layer analysis
We study the performance of ﬁlter banks extracted from
different layers of a CNN in the FV-CNN framework. We
use the VGG-M network which has ﬁve convolutional lay-
ers. Results on various datasets, obtained as in Sect. 4.1
and 4.2, are shown in Fig. 4. In addition we also show the
performance using FVs constructed from dense SIFT using
a number of words such that the resulting FV is roughly the
same size of FV-CNN. The CNN ﬁlter banks from layer
3 and beyond signiﬁcantly outperform SIFT. The perfor-
mance monotonically improves from layer one to ﬁve.
5. Conclusions
We have conducted a range of experiments on material
and texture attribute recognition in a large dataset of tex-
tures in clutter. This benchmark was derived from OpenSur-
faces, an earlier contribution of the computer graphics com-
munity, highlights the potential for collaboration between
computer graphics and vision communities. We have also
evaluated a number of state-of-the-art texture descriptors on
these and many other benchmarks. Our main ﬁnding is that
orderless pooling of convolutional neural network features
is a remarkably good texture descriptor, versatile enough to
dubbed as a scene and object descriptor, resulting in new
state-of-the-art performance in several benchmarks.
brick cardboard carpet_rug ceramic concrete fabric_cloth food glass
granite_marble hair other laminate leather metal painted paper_tissue
plastic_clear plastic_opaque stone tile wallpaper wood(a) (b) (c) (d) (e) (f)
Figure 2. OS material recognition results. Example test image with material recognition and segmentation on the OS dataset. (a) original
image. (b) ground truth segmentations from the OpenSurfaces repository (note that not all pixels are annotated). (c) FC-CNN and crisp-
region proposals segmentation results. (d) incorrectly predicted pixels (restricted to the ones annotated). (e-f) the same, but for FV-CNN.
building grass tree cow sheep sky aeroplane water
face car bicycle flower sign bird book chair
road cat dog body boat
(a) (b) (c) (d) (e) (f)
Figure 3. MSRC object segmentation results. (a) image, (b) ground-truth, (c-d) FC-CNN, (d-e) FV-CNN segmentation and errors.
Acknowledgements M. Cimpoi was supported by ERC
grant VisRec no. 228180 and the XRCE UAC grant. S. Maji
acknowledges a faculty startup grant from UMass, Amherst.References
[1] E. H. Adelson. On seeing stuff: The perception of materials by hu-
mans and machines. SPIE , 4299, 2001. 1
[2] M. Amadasun and R. King. Textural features corresponding to tex-
tural properties. Systems, Man, and Cybernetics , 19(5), 1989. 1
[3] P. Arbel ´aez, J. Pont-Tuset, J. T. Barron, F. Marques, and J. Malik.
Multiscale combinatorial grouping. CVPR, 2014. 4, 5, 7
[4] S. Bell, P. Upchurch, N. Snavely, and K. Bala. Opensurfaces: A
richly annotated catalog of surface appearance. In Proc. SIGGRAPH ,
2013. 2
[5] B. Caputo, E. Hayman, and P. Mallikarjuna. Class-speciﬁc material
categorisation. In ICCV , 2005. 1, 4
[6] K. Chatﬁeld, K. Simonyan, A. Vedaldi, and A. Zisserman. Return
of the devil in the details: Delving deep into convolutional nets. In
Proc. BMVC , 2014. 2, 3, 4, 6
[7] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi.
Describing textures in the wild. In Proc. CVPR , 2014. 1, 3, 4, 5
[8] K. J. Dana, B. van Ginneken, S. K. Nayar, and J. J. Koenderink.
Reﬂectance and texture of real world surfaces. ACM Transactions on
Graphics , 18(1):1–34, 1999. 1
[9] A. Efros and T. Leung. Texture synthesis by non-parametric sam-
pling. In CVPR , volume 2, 1999. 1
[10] M. Everingham, A. Zisserman, C. Williams, and L. V . Gool. The
PASCAL visual obiect classes challenge 2007 (VOC2007) results.
Technical report, Pascal Challenge, 2007. 3, 4
[11] V . Ferrari and A. Zisserman. Learning visual attributes. In Proc.
NIPS , 2007. 1
[12] D. Forsyth. Shape from texture and integrability. In ICCV , volume 2,
pages 447–452. IEEE, 2001. 1
[13] J. G ˚arding. Shape from texture for smooth curved surfaces in per-
spective projection. Journal of Mathematical Imaging and Vision ,
2(4):327–350, 1992. 1
[14] R. B. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature
hierarchies for accurate object detection and semantic segmentation.
InProc. CVPR , 2014. 2, 3
[15] Y . Gong, L. Wang, R. Guo, and S. Lazebnik. Multi-scale orderless
pooling of deep convolutional activation features. In Proc. ECCV ,
2014. 2, 3, 6
[16] B. Hariharan, P. Arbel ´aez, R. Girshick, and J. Malik. Simultaneous
detection and segmentation. In Computer Vision–ECCV 2014 , pages
297–312. Springer, 2014. 2, 7
[17] E. Hayman, B. Caputo, M. Fritz, and J.-O. Eklundh. On the signif-
icance of real-world conditions for material classiﬁcation. ECCV ,
2004. 4
[18] K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling in deep
convolutional networks for visual recognition. In Proc. ECCV , 2014.
2
[19] P. Isola, D. Zoran, D. Krishnan, and E. H. Adelson. Crisp boundary
detection using pointwise mutual information. In Proc. ECCV , 2014.
4, 7
[20] A. Jain and F. Farrokhnia. Unsupervised texture segmentation using
gabor ﬁlters. Pattern recognition , 24(12):1167–1186, 1991. 1
[21] Y . Jia. Caffe: An open source convolutional architecture for fast
feature embedding. http://caffe.berkeleyvision.org/ ,
2013. 3, 4, 6
[22] B. Julesz and J. R. Bergen. Textons, the fundamental elements in
preattentive vision and perception of textures. Bell System Technical
Journal , 62(6, Pt 3):1619–1645, Jul-Aug 1983. 1
[23] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classiﬁca-
tion with deep convolutional neural networks. In Proc. NIPS , 2012.
2, 3
[24] L. Ladicky, C. Russell, P. Kohli, and P. Torr. Graph cut based infer-
ence with co-occurrence statistics. In Proc. ECCV , pages 239–253,
2010. 5
[25] L. Ladicky, P. Sturgess, K. Alahari, C. Russell, and P. Torr. What,
where and how many? combining object detectors and crfs. In Proc.
ECCV , 2010. 7
[26] T. Leung and J. Malik. Representing and recognizing the visual ap-
pearance of materials using three-dimensional textons. International
Journal of Computer Vision , 43(1):29–44, 2001. 1
[27] L. Liu, L. Wang, and X. Liu. In defense of soft-assignment coding.
InComputer Vision (ICCV), 2011 IEEE International Conference on ,
pages 2486–2493. IEEE, 2011. 1[28] D. G. Lowe. Object recognition from local scale-invariant features.
InProc. ICCV , 1999. 1
[29] B. Manjunath and R. Chellappa. Unsupervised texture segmentation
using markov random ﬁeld models. IEEE Transactions on Pattern
Analysis and Machine Intelligence , 13(5):478–482, 1991. 1
[30] M. Oquab, L. Bottou, I. Laptev, and J. Sivic. Learning and Transfer-
ring Mid-Level Image Representations using Convolutional Neural
Networks. In Proc. CVPR , 2014. 2
[31] O. M. Parkhi, K. Simonyan, A. Vedaldi, and A. Zisserman. A com-
pact and discriminative face track descriptor. In Proc. CVPR , 2014.
4
[32] F. Perronnin, J. S ´anchez, and T. Mensink. Improving the ﬁsher ker-
nel for large-scale image classiﬁcation. In Computer Vision–ECCV
2010 , pages 143–156. Springer, 2010. 1
[33] X. Qi, R. Xiao, C. G. Li, Y . Qiao, J. Guo, and X. Tang. Pair-
wise rotation invariant co-occurrence local binary pattern. PAMI ,
36(11):2199–2213, Nov 2014. 5
[34] A. Quattoni and A. Torralba. Recognizing indoor scenes. In Proc.
CVPR , 2009. 2, 4
[35] A. S. Razavin, H. Azizpour, J. Sullivan, and S. Carlsson. Cnn fea-
tures off-the-shelf: An astounding baseline for recognition. In Deep-
Vision workshop , 2014. 3
[36] G. Schwartz and K. Nishino. Visual material traits: Recognizing
per-pixel material context. In Proc. CVCP , 2013. 1
[37] L. Sharan, C. Liu, R. Rosenholtz, and E. H. Adelson. Recognizing
materials using perceptually inspired features. International Journal
of Computer Vision , 103(3):348–371, 2013. 1, 4, 5
[38] L. Sharan, R. Rosenholtz, and E. H. Adelson. Material perceprion:
What can you see in a brief glance? Journal of Vision , 9:784(8),
2009. 1
[39] K. Simonyan and A. Zisserman. Very deep convolutional networks
for large-scale image recognition. CoRR , abs/1409.1556, 2014. 2, 4
[40] M. Sulc and J. Matas. Fast features invariant to rotation and scale of
texture. Technical report, 2014. 5
[41] M. Varma and A. Zisserman. Texture classiﬁcation: Are ﬁlter banks
necessary? In CVPR , volume 2, pages II–691. IEEE, 2003. 1
[42] J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, and Y . Gong. Locality-
constrained linear coding for image classiﬁcation. In Computer Vi-
sion and Pattern Recognition (CVPR), 2010 IEEE Conference on ,
pages 3360–3367. IEEE, 2010. 1
[43] L. Wei and M. Levoy. Fast texture synthesis using tree-
structured vector quantization. In SIGGRAPH , pages 479–488. ACM
Press/Addison-Wesley Publishing Co., 2000. 1
[44] Y . Wei, W. Xia, J. Huang, B. Ni, J. Dong, Y . Zhao, and S. Yan. Cnn:
Single-label to multi-label. 2014. 5, 6
[45] P. Welinder, S. Branson, T. Mita, C. Wah, and F. Schroff. Caltech-
ucsd birds 200. Technical report, Caltech-UCSD, 2010. 4
[46] N. Zhang, J. Donahue, R. Girshickr, and T. Darrell. Part-based R-
CNNs for ﬁne-grained category detection. In Proc. ECCV , 2014. 5,
6
[47] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. Learning
deep features for scene recognition using places database. In Proc.
NIPS , 2014. 2, 5, 6, 7
[48] X. Zhou, K. Yu, T. Zhang, and T. S. Huang. Image classiﬁcation
using super-vector coding of local image descriptors. In Computer
Vision–ECCV 2010 , pages 141–154. Springer, 2010. 1