Summary:
Recognizing materials in real-world images is a challenging task. Real-world
materials have rich surface texture, geometry, lighting conditions, and
clutter, which combine to make the problem particularly difficult. In this
paper, we introduce a new, large-scale, open dataset of materials in the wild,
the Materials in Context Database (MINC), and combine this dataset with deep
learning to achieve material recognition and segmentation of images in the
wild.
  MINC is an order of magnitude larger than previous material databases, while
being more diverse and well-sampled across its 23 categories. Using MINC, we
train convolutional neural networks (CNNs) for two tasks: classifying materials
from patches, and simultaneous material recognition and segmentation in full
images. For patch-based classification on MINC we found that the best
performing CNN architectures can achieve 85.2% mean class accuracy. We convert
these trained CNN classifiers into an efficient fully convolutional framework
combined with a fully connected conditional random field (CRF) to predict the
material at every pixel in an image, achieving 73.1% mean class accuracy. Our
experiments demonstrate that having a large, well-sampled dataset such as MINC
is crucial for real-world material recognition and segmentation.

Sean Bell
Paul Upchurch
Noah Snavely
Kavita Bala

CVPR 2015. Sean Bell and Paul Upchurch contributed equally

Material Recognition in the Wild with the Materials in Context Database
Sean BellPaul UpchurchNoah Snavely Kavita Bala
Department of Computer Science, Cornell University
fsbell,paulu,snavely,kb g@cs.cornell.edu
Abstract
Recognizing materials in real-world images is a challeng-
ing task. Real-world materials have rich surface texture,
geometry, lighting conditions, and clutter, which combine
to make the problem particularly difﬁcult. In this paper, we
introduce a new, large-scale, open dataset of materials in
the wild, the Materials in Context Database (MINC ), and
combine this dataset with deep learning to achieve material
recognition and segmentation of images in the wild.
MINC is an order of magnitude larger than previous ma-
terial databases, while being more diverse and well-sampled
across its 23 categories. Using MINC , we train convolu-
tional neural networks (CNNs) for two tasks: classifying
materials from patches, and simultaneous material recogni-
tion and segmentation in full images. For patch-based clas-
siﬁcation on MINC we found that the best performing CNN
architectures can achieve 85.2% mean class accuracy. We
convert these trained CNN classiﬁers into an efﬁcient fully
convolutional framework combined with a fully connected
conditional random ﬁeld (CRF) to predict the material at
every pixel in an image, achieving 73.1% mean class ac-
curacy. Our experiments demonstrate that having a large,
well-sampled dataset such as MINC is crucial for real-world
material recognition and segmentation.
1. Introduction
Material recognition plays a critical role in our under-
standing of and interactions with the world. To tell whether
a surface is easy to walk on, or what kind of grip to use
to pick up an object, we must recognize the materials that
make up our surroundings. Automatic material recognition
can be useful in a variety of applications, including robotics,
product search, and image editing for interior design. But rec-
ognizing materials in real-world images is very challenging.
Many categories of materials, such as fabric or wood, are
visually very rich and span a diverse range of appearances.
Materials can further vary in appearance due to lighting and
shape. Some categories, such as plastic and ceramic, are of-
Authors contributed equally
CNN
Sliding
CNN“wood”
Transfer weights(trained)
(discretely
optimized)(fixed)AMT AMT AMT MINC
3 million
patches OpenSurfacesFlickr
Houzz(a) Constructing MINC
Material
Labels(b) Patch material classification
(c) Full scene material classificationP(material)
Dense
CRF
PatchFigure 1. Overview. (a) We construct a new dataset by combining
OpenSurfaces [ 1] with a novel three-stage Amazon Mechanical
Turk (AMT) pipeline. (b)We train various CNNs on patches from
MINC to predict material labels. (c)We transfer the weights to a
fully convolutional CNN to efﬁciently generate a probability map
across the image; we then use a fully connected CRF to predict the
material at every pixel.
ten smooth and featureless, requiring reasoning about subtle
cues or context to differentiate between them.
Large-scale datasets (e.g., ImageNet [ 21], SUN [ 31,19]
and Places [ 34]) combined with convolutional neural net-
works (CNNs) have been key to recent breakthroughs in
object recognition and scene classiﬁcation. Material recogni-
tion is similarly poised for advancement through large-scale
data and learning. To date, progress in material recognition
has been facilitated by moderate-sized datasets like the Flickr
Material Database (FMD) [ 26]. FMD contains ten material
categories, each with 100 samples drawn from Flickr photos.
These images were carefully selected to illustrate a wide
range of appearances for these categories. FMD has been
used in research on new features and learning methods for
material perception and recognition [ 17,10,20,25]. While
FMD was an important step towards material recognition, it
is not sufﬁcient for classifying materials in real-world im-
1arXiv:1412.0623v2  [cs.CV]  14 Apr 2015
agery. This is due to the relatively small set of categories,
the relatively small number of images per category, and also
because the dataset has been designed around hand-picked
iconic images of materials. The OpenSurfaces dataset [ 1]
addresses some of these problems by introducing 105,000
material segmentations from real-world images, and is sig-
niﬁcantly larger than FMD. However, in OpenSurfaces many
material categories are under-sampled, with only tens of
images.
A major contribution of our paper is a new, well-sampled
material dataset, called the Materials in Context Database
(MINC ), with 3 million material samples. MINC is more
diverse, has more examples in less common categories, and
is much larger than existing datasets. MINC draws data from
both Flickr images, which include many “regular” scenes,
as well as Houzz images from professional photographers of
staged interiors. These sources of images each have different
characteristics that together increase the range of materials
that can be recognized. See Figure 2 for examples of our
data. We make our full dataset available online at http:
//minc.cs.cornell.edu/.
We use this data for material recognition by training dif-
ferent CNN architectures on this new dataset. We perform
experiments that illustrate the effect of network architec-
ture, image context, and training data size on subregions
(i.e., patches) of a full scene image. Further, we build on
our patch classiﬁcation results and demonstrate simultane-
ous material recognition and segmentation of an image by
performing dense classiﬁcation over the image with a fully
connected conditional random ﬁeld (CRF) model [ 12]. By
replacing the fully connected layers of the CNN with convo-
lutional layers [ 24], the computational burden is signiﬁcantly
lower than a naive sliding window approach.
In summary, we make two new contributions:
We introduce a new material dataset, MINC , and 3-
stage crowdsourcing pipeline for efﬁciently collecting
millions of click labels (Section 3.2).
Our new semantic segmentation method combines a
fully-connected CRF with unary predictions based on
CNN learned features (Section 4.2) for simultaneous
material recognition and segmentation.
2. Prior Work
Material Databases. Much of the early work on material
recognition focused on classifying speciﬁc instances of tex-
tures or material samples. For instance, the CUReT [ 4]
database contains 61 material samples, each captured under
205 different lighting and viewing conditions. This led to
research on the task of instance-level texture or material clas-
siﬁcation [ 15,30], and an appreciation of the challenges of
building features that are invariant to pose and illumination.
Later, databases with more diverse examples from each ma-terial category began to appear, such as KTH-TIPS [ 9,2],
and led explorations of how to generalize from one example
of a material to another—from one sample of wood to a com-
pletely different sample, for instance. Real-world texture
attributes have also recently been explored [3].
In the domain of categorical material databases, Sharan et
al. released FMD [ 26] (described above). Subsequently,
Bell et al. released OpenSurfaces [ 1] which contains over
20,000 real-world scenes labeled with both materials and ob-
jects, using a multi-stage crowdsourcing pipeline. Because
OpenSurfaces images are drawn from consumer photos on
Flickr, material samples have real-world context, in contrast
to prior databases (CUReT, KTH-TIPS, FMD) which fea-
ture cropped stand-alone samples. While OpenSurfaces is a
good starting point for a material database, we substantially
expand it with millions of new labels.
Material recognition. Much prior work on material recog-
nition has focused on the classiﬁcation problem (categorizing
an image patch into a set of material categories), often using
hand-designed image features. For FMD, Liu et al. [17] in-
troduced reﬂectance-based edge features in conjunction with
other general image features. Hu et al. [10] proposed fea-
tures based on variances of oriented gradients. Qi et al. [20]
introduced a pairwise local binary pattern (LBP) feature.
Liet al. [16] synthesized a dataset based on KTH-TIPS2
and built a classiﬁer from LBP and dense SIFT. Timofte et
al. [29] proposed a classiﬁcation framework with minimal
parameter optimization. Schwartz and Nishino [ 23] intro-
duced material traits that incorporate learned convolutional
auto-encoder features. Recently, Cimpoi et al. [3] devel-
oped a CNN and improved Fisher vector (IFV) classiﬁer that
achieves state-of-the-art results on FMD and KTH-TIPS2.
Finally, it has been shown that jointly predicting objects and
materials can improve performance [10, 33].
Convolutional neural networks. While CNNs have been
around for a few decades, with early successes such as
LeNet [ 14], they have only recently led to state-of-the-
art results in object classiﬁcation and detection, leading
to enormous progress. Driven by the ILSVRC chal-
lenge [ 21], we have seen many successful CNN architec-
tures [ 32,24,28,27], led by the work of Krizhevsky et al.
on their SuperVision (a.k.a. AlexNet ) network [ 13], with
more recent architectures including GoogLeNet [ 28]. In ad-
dition to image classiﬁcation, CNNs are the state-of-the-art
for detection and localization of objects, with recent work
including R-CNNs [ 7], Overfeat [ 24], and VGG [ 27]. Fi-
nally, relevant to our goal of per-pixel material segmentation,
Farabet et al. [6] use a multi-scale CNN to predict the class
at every pixel in a segmentation. Oquab et al. [18] employ a
sliding window approach to localize patch classiﬁcation of
objects. We build on this body of work in deep learning to
solve our problem of material recognition and segmentation.
2
Brick Carpet Ceramic Fabric Foliage Food Glass Hair
Leather Metal Mirror Other Painted Paper Plastic Pol. stone
Skin Sky Stone Tile Wallpaper Water Wood
Figure 2. Example patches from all 23 categories of the Materials in Context Database (MINC ). Note that we sample patches so that the
patch center is the material in question (and not necessarily the entire patch). See Table 1 for the size of each category.
3. The Materials in Context Database (MINC)
We now describe the methodology that went into building
our new material database. Why a new database? We needed
a dataset with the following properties:
Size: It should be sufﬁciently large that learning meth-
ods can generalize beyond the training set.
Well-sampled : Rare categories should be represented
with a large number of examples.
Diversity : Images should span a wide range of appear-
ances of each material in real-world settings.
Number of categories : It should contain many differ-
ent materials found in the real world.
3.1. Sources of data
We decided to start with the public, crowdsourced
OpenSurfaces dataset [ 1] as the seed for MINC since it is
drawn from Flickr imagery of everyday, real-world scenes
with reasonable diversity. Furthermore, it has a large number
of categories and the most samples of all prior databases.
While OpenSurfaces data is a good start, it has a few lim-
itations. Many categories in OpenSurfaces are not well sam-
pled. While the largest category, wood , has nearly 20K sam-
ples, smaller categories, such as water , have only tens of ex-
amples. This imbalance is due to the way the OpenSurfaces
dataset was annotated; workers on Amazon Mechanical Turk
(AMT) were free to choose any material subregion to seg-
ment. Workers often gravitated towards certain common
types of materials or salient objects, rather than being en-
couraged to label a diverse set of materials. Further, the
images come from a single source (Flickr).
We decided to augment OpenSurfaces with substantially
more data, especially for underrepresented material cate-gories, with the initial goal of gathering at least 10K samples
per material category. We decided to gather this data from
another source of imagery, professional photos on the inte-
rior design website Houzz (houzz.com). Our motivation for
using this different source of data was that, despite Houzz
photos being more “staged” (relative to Flickr photos), they
actually represent a larger variety of materials. For instance,
Houzz photos contain a wide range of types of polished
stone. With these sources of image data, we now describe
how we gather material annotations.
3.2. Segments, Clicks, and Patches
What speciﬁc kinds of material annotations make for a
good database? How should we collect these annotations?
The type of annotations to collect is guided in large part by
the tasks we wish to generate training data for. For some
tasks such as scene recognition, whole-image labels can
sufﬁce [ 31,34]. For object detection, labeled bounding
boxes as in PASCAL are often used [ 5]. For segmentation or
scene parsing tasks, per-pixel segmentations are required [ 22,
8]. Each style of annotation comes with a cost proportional
to its complexity. For materials, we decided to focus on two
problems, guided by prior work:
Patch material classiﬁcation. Given an image patch,
what kind of material is it at the center?
Full scene material classiﬁcation. Given a full im-
age, produce a full per-pixel segmentation and labeling.
Also known as semantic segmentation orscene parsing
(but in our case, focused on materials). Note that classi-
ﬁcation can be a component of segmentation, e.g., with
sliding window approaches.
3
(a) Which images
contain wood ?(b) Click on 3
points of wood(c) What is this
material?Figure 3. AMT pipeline schematic for collecting clicks. (a)
Workers ﬁlter by images that contain a certain material, (b)work-
ers click on materials, and (c)workers validate click locations by
re-labeling each point. Example responses are shown in orange.
Segments. OpenSurfaces contains material segmentations—
carefully drawn polygons that enclose same-material regions.
To form the basis of MINC, we selected OpenSurfaces seg-
ments with high conﬁdence (inter-worker agreement) and
manually curated segments with low conﬁdence, giving a
total of 72K shapes. To better balance the categories, we
manually segmented a few hundred extra samples for sky,
foliage andwater .
Since some of the OpenSurfaces categories are difﬁcult
for humans, we consolidated these categories. We found
that many AMT workers could not disambiguate stone from
concrete ,clear plastic from opaque plastic , and granite
from marble . Therefore, we merged these into stone ,plastic ,
andpolished stone respectively. Without this merging, many
ground truth examples in these categories would be incorrect.
The ﬁnal list of 23 categories is shown in Table 1. The
category other is different in that it was created by combining
various smaller categories.
Clicks. Since we want to expand our dataset to millions
of samples, we decided to augment OpenSurfaces segments
by collecting clicks : single points in an image along with a
material label, which are much cheaper and faster to collect.
Figure 3 shows our pipeline for collecting clicks.
Initially, we tried asking workers to click on examples
of a given material in a photo. However, we found that
workers would get frustrated if the material was absent in
too many of the photos. Thus, we added an initial ﬁrst
stage where workers ﬁlter out such photos. To increase
the accuracy of our labels, we verify the click labels by
asking different workers to specify the material for each
click without providing them with the label from the previous
stage.
To ensure that we obtain high quality annotations and
avoid collecting labels from workers who are not making an
effort, we include secret known answers (sentinels) in the
ﬁrst and third stages, and block workers with an accuracy
below 50% and 85% respectively. We do not use sentinels
in the second stage since it would require per-pixel ground
truth labels, and it turned out not to be necessary. Workers
generally performed all three tasks so we could identify bad
workers in the ﬁrst or third task.Patches Category Patches Category Patches Category
564,891 Wood 114,085 Polished stone 35,246 Skin
465,076 Painted 98,891 Carpet 29,616 Stone
397,982 Fabric 83,644 Leather 28,108 Ceramic
216,368 Glass 75,084 Mirror 26,103 Hair
188,491 Metal 64,454 Brick 25,498 Food
147,346 Tile 55,364 Water 23,779 Paper
142,150 Sky 39,612 Other 14,954 Wallpaper
120,957 Foliage 38,975 Plastic
Table 1. MINC patch counts by category. Patches were created
from both OpenSurfaces segments and our newly collected clicks .
See Section 3.2 for details.
Material clicks were collected for both OpenSurfaces
images and the new Houzz images. This allowed us to use
labels from OpenSurfaces to generate the sentinel data; we
included 4 sentinels per task. With this streamlined pipeline
we collected 2,341,473 annotations at an average cost of
$0.00306 per annotation (stage 1: $0.02 / 40 images, stage
2: $0.10 / 50 images, 2, stage 3: $0.10 / 50 points).
Patches. Labeled segments and clicks form the core of
MINC . For training CNNs and other types of classiﬁers,
it is useful to have data in the form of ﬁxed-sized patches.
We convert both forms of data into a uniﬁed dataset format:
square image patches. We use a patch center andpatch scale
(a multiplier of the smaller image dimension) to deﬁne the
image subregion that makes a patch. For our patch classi-
ﬁcation experiments, we use 23.3% of the smaller image
dimension. Increasing the patch scale provides more context
but reduces the spatial resolution. Later in Section 5 we
justify our choice with experiments that vary the patch scale
for AlexNet.
We place a patch centered around each click label. For
each segment , if we were to place a patch at every interior
pixel then we would have a very large and redundant dataset.
Therefore, we Poisson-disk subsample each segment , sepa-
rating patch centers by at least 9.1% of the smaller image
dimension. These segments generated 655,201 patches (an
average of 9.05 patches per segment ). In total, we gener-
ated 2,996,674 labeled patches from 436,749 images. Patch
counts are shown in Table 1, and example patches from
various categories are illustrated in Figure 2.
4. Material recognition in real-world images
Our goal is to train a system that recognizes the material
at every pixel in an image. We split our training procedure
into multiple stages and analyze the performance of the
network at each stage. First, we train a CNN that produces a
single prediction for a given input patch. Then, we convert
the CNN into a sliding window and predict materials on a
dense grid across the image. We do this at multiple scales
and average to obtain a unary term. Finally, a dense CRF
[12] combines the unary term with fully connected pairwise
reasoning to output per-pixel material predictions. The entire
system is depicted in Figure 1, and described more below.
4
Sky
Water
FoliageTile
(a) Multiscale input (b) Probability map (12 of 23 shown) (c) Predicted labelsSliding
CNNDense
CRF
Wood
Plastic Sliding
CNNSliding
CNN
Avg.
Glass
Hair
Fabric
Polished
stone
Stone
 PaperWaterSky
Foliage
TilePaperWood
StoneWood
TileFigure 4. Pipeline for full scene material classiﬁcation. An image (a)is resized to multiple scales [1=p
2;1;p
2]. The same sliding CNN
predicts a probability map (b)across the image for each scale; the results are upsampled and averaged. A fully connected CRF predicts a
ﬁnal label for each pixel (c). This example shows predictions from a single GoogLeNet converted into a sliding CNN (no average pooling).
4.1. Training procedure
MINC contains 3 million patches that we split into train-
ing, validation and test sets. Randomly splitting would result
in nearly identical patches (e.g., from the same OpenSur-
faces segment) being put in training and test, thus inﬂating
the test score. To prevent correlation, we group photos into
clusters of near-duplicates, then assign each cluster to one
of train, validate or test. We make sure that there are at least
75segments of each category in the test set to ensure there
are enough segments to evaluate segmentation accuracy. To
detect near-duplicates, we compare AlexNet CNN features
computed from each photo (see the supplemental for details).
For exact duplicates, we discard all but one of the copies.
We train all of our CNNs by ﬁne-tuning the network
starting from the weights obtained by training on 1.2 mil-
lion images from ImageNet (ILSVRC2012). When training
AlexNet , we use stochastic gradient descent with batchsize
128, dropout rate 0.5, momentum 0.9, and a base learning
rate of 103that decreases by a factor of 0.25 every 50,000
iterations. For GoogLeNet , we use batchsize 69, dropout 0.4,
and learning rate t= 104p
1t=250000 for iteration t.
Our training set has a different number of examples per
class, so we cycle through the classes and randomly sample
an example from each class. Failing to properly balance the
examples results in a 5.7% drop in mean class accuracy (on
the validation set). Further, since it has been shown to reduce
overﬁtting, we randomly augment samples by taking crops
(227227out of 256256), horizontal mirror ﬂips, spatial
scales in the range [1=p
2;p
2], aspect ratios from 3:4 to 4:3,
and amplitude shifts in [0:95;1:05]. Since we are looking at
local regions, we subtract a per-channel mean (R: 124, G:
117, B: 104) rather than a mean image [13].
4.2. Full scene material classiﬁcation
Figure 4 shows an overview of our method for simul-
taneously segmenting and recognizing materials. Given a
CNN that can classify individual points in the image, we
convert it to a sliding window detector and densely classify
a grid across the image. Speciﬁcally, we replace the last
fully connected layers with convolutional layers, so that the
network is fully convolutional and can classify images ofany shape. After conversion, the weights are ﬁxed and not
ﬁne-tuned. With our converted network, the strides of each
layer cause the network to output a prediction every 32 pix-
els. We obtain predictions every 16 pixels by shifting the
input image by half-strides (16 pixels). While this appears to
require 4x the computation, Sermanet et al. [24] showed that
the convolutions can be reused and only the pool5 through
fc8 layers need to be recomputed for the half-stride shifts.
Adding half-strides resulted in a minor 0.2% improvement
in mean class accuracy across segments (after applying the
dense CRF, described below), and about the same mean class
accuracy at click locations.
The input image is resized so that a patch maps to a
256x256 square. Thus, for a network trained at patch scale
s, the resized input has smaller dimension d= 256=s. Note
thatdis inversely proportional to scale, so increased context
leads to lower spatial resolution. We then add padding so
that the output probability map is aligned with the input
when upsampled. We repeat this at 3 different scales (smaller
dimensiond=p
2,d,dp
2), upsample each output probability
map with bilinear interpolation, and average the predictions.
To make the next step more efﬁcient, we upsample the output
to a ﬁxed smaller dimension of 550.
We then use the dense CRF of Kr¨ahenb ¨uhlet al. [12] to
predict a label at every pixel, using the following energy:
E(xjI) =X
i i(xi) +X
i<j ij(xi;xj) (1)
 i(xi) =logpi(xi) (2)
 ij(xi;xj) =wp(xi6=xj)k(fifj) (3)
where iis the unary energy (negative log of the aggre-
gated softmax probabilities) and  ijis the pairwise term
that connects every pair of pixels in the image. We use a
single pairwise term with a Potts label compatibility term
weighted by wpand unit Gaussian kernel k. For the fea-
turesfi, we convert the RGB image to L*a*b*and use color
(IL
i;Ia
i;Ib
i)and position (px;py)as pairwise features for
each pixel: fi=h
px
i
pd;py
i
pd;IL
i
L;Ia
i
ab;Ib
i
abi
, wheredis the
smaller image dimension. Figure 4 shows an example unary
termpiand the resulting segmentation x.
5
Figure 5. Varying patch scale . We train/test patches of different
scales (the patch locations do not vary). The optimum is a trade-off
between context and spatial resolution. CNN: AlexNet.
Architecture Validation Test
AlexNet [13] 82.2% 81.4%
GoogLeNet [28] 85.9% 85.2%
VGG-16 [27] 85.6% 84.8%
Table 2. Patch material classiﬁcation results. Mean class accu-
racy for different CNNs trained on MINC. See Section 5.1.
Sky 97.3% Food 90.4% Wallpaper 83.4% Glass 78.5%
Hair 95.3% Leather 88.2% Tile 82.7% Fabric 77.8%
Foliage 95.1% Other 87.9% Ceramic 82.7% Metal 77.7%
Skin 93.9% Pol. stone 85.8% Stone 82.7% Mirror 72.0%
Water 93.6% Brick 85.1% Paper 81.8% Plastic 70.9%
Carpet 91.6% Painted 84.2% Wood 81.3%
Table 3. Patch test accuracy by category. CNN: GoogLeNet . See
the supplemental material for a full confusion matrix.
5. Experiments and Results
5.1. Patch material classiﬁcation
In this section, we evaluate the effect of many different
design decisions for training methods for material classiﬁca-
tion and segmentation, including various CNN architectures,
patch sizes, and amounts of data.
CNN Architectures. Our ultimate goal is full material seg-
mentation, but we are also interested in exploring which
CNN architectures give the best results for classifying sin-
gle patches. Among the networks and parameter varia-
tions we tried we found the best performing networks were
AlexNet [13],VGG-16 [27] and GoogLeNet [28].AlexNet
andGoogLeNet are re-implementations by BVLC [ 11], and
VGG-16 is conﬁguration D (a 16 layer network) of [ 27].
All models were obtained from the Caffe Model Zoo [ 11].
Our experiments use AlexNet for evaluating material classi-
ﬁcation design decisions and combinations of AlexNet and
GoogLeNet for evaluating material segmentation. Tables 2
and 3 summarize patch material classiﬁcation results on our
dataset. Figure 10 shows correct and incorrect predictions
made with high conﬁdence.
Input patch scale. To classify a point in an image we must
decide how much context to include around it. The context,
expressed as a fraction of image size, is the patch scale. A
priori, it is not clear which scale is best since small patches
have better spatial resolution, but large patches have more
0.5e6 1e6 1.5e6 2e6 2.5e6
Number of training patches0.600.650.700.750.800.85Mean class accuracy (test set)66.1%74.8%77.9%79.3%80.2%80.9% 81.2% 81.4%Equal size
79.6%Figure 6. Varying database size. Patch accuracy when trained on
random subsets of MINC .Equal size is using equal samples per
category (size determined by smallest category). CNN: AlexNet.
Peak accuracy
per category
Figure 7. Accuracy vs patch scale by category . Dots: peak accu-
racy for each category; colored lines: sky,wallpaper ,mirror ; gray
lines: other categories. CNN: AlexNet . While most materials are
optimally recognized at 23.3% or 32% patch scale, recognition of
sky,wallpaper andmirror improve with increasing context.
contextual information. Holding patch centers ﬁxed we var-
ied scale and evaluated classiﬁcation accuracy with AlexNet .
Results and a visualization of patch scales are shown in Fig-
ure 5. Scale 32% performs the best. Individual categories
had peaks at middle scale with some exceptions; we ﬁnd
thatmirror ,wallpaper andskyimprove with increasing con-
text (Figure 7). We used 23.3% (which has nearly the same
accuracy but higher spatial resolution) for our experiments.
Dataset size. To measure the effect of size on patch clas-
siﬁcation accuracy we trained AlexNet with patches from
randomly sampled subsets of all 369,104 training images
and tested on our full test set (Figure 6). As expected, using
more data improved performance. In addition, we still have
not saturated performance with 2.5 million training patches;
even higher accuracies may be possible with more training
data (though with diminishing returns).
Dataset balance. Although we’ve shown that more data is
better we also ﬁnd that a balanced dataset is more effective.
We trained AlexNet with all patches of our smallest category
(wallpaper ) and randomly sampled the larger categories
(the largest, wood , being 40x larger) to be equal size. We
then measured mean class accuracy on the same full test
set. As shown in Figure 6, “Equal size” is more accurate
than a dataset of the same size and just 1.7% lower than
the full training set (which is 9x larger). This result further
demonstrates the value of building up datasets in a balanced
manner, focusing on expanding the smallest, least common
categories.
6
Brick
WoodCarpet
WaterWallpaperTileStoneSkySkinPolished stonePlasticPaperPainted
OtherMirrorMetalLeatherHairGlassFoodFoliageFabricCeramicFigure 8. Full-scene material classiﬁcation examples: high-accuracy test set predictions by our method. CNN: GoogLeNet (with the
average pooling layer removed). Right: legend for material colors. See Table 4 for quantitative evaluation.
Input image (a)Labels from CRF (b)Labels from CRF
(test set) trained on segments trained on clicks
Figure 9. Optimizing for click accuracy leads to sloppy bound-
aries. In(a), we optimize for mean class accuracy across segments,
resulting in high quality boundaries. In (b), we optimize for mean
class accuracy at click locations. Since the clicks are not neces-
sarily close to object boundaries, there is no penalty for sloppy
boundaries. CNN: GoogLeNet (without average pooling).
5.2. Full scene material segmentation
The full test set for our patch dataset contains 41,801
photos, but most of them contain only a few labels. Since
we want to evaluate the per-pixel classiﬁcation performance,
we select a subset of 5,000 test photos such that each photo
contains a large number of segments and clicks, and small
categories are well sampled. We greedily solve for the best
such set of photos. We similarly select 2,500 of 25,844
validation photos. Our splits for all experiments are included
online with the dataset. To train the CRF for our model, we
try various parameter settings ( p,ab,L,wp) and select the
model that performs best on the validation set. In total, we
evaluate 1799 combinations of CNNs and CRF parameters.
See the supplemental material for a detailed breakdown.
We evaluate multiple versions of GoogLeNet : both the
original architecture and a version with the average pooling
layer (at the end) changed to 5x5, 3x3, and 1x1 (i.e. no
average pooling). We evaluate AlexNet trained at multiple
patch scales (Figure 5). When using an AlexNet trained
at a different scale, we keep the same scale for testing. We
also experiment with ensembles of GoogLeNet andAlexNet ,Architecture(a)Segments only (b)Clicks only
Class Total Class Total
AlexNet Scale: 11.6% 64.3% 72.6% 79.9% 77.2%
AlexNet Scale: 23.3% 69.6% 76.6% 83.3% 81.1%
AlexNet Scale: 32.0% 70.1% 77.1% 83.2% 80.7%
AlexNet Scale: 46.5% 69.6% 75.4% 80.8% 77.7%
AlexNet Scale: 66.2% 67.7% 72.0% 77.2% 72.6%
GoogLeNet 7x7 avg. pool 64.4% 71.6% 63.6% 63.4%
GoogLeNet 5x5 avg. pool 67.6% 74.6% 70.9% 69.8%
GoogLeNet 3x3 avg. pool 70.4% 77.7% 76.1% 74.7%
GoogLeNet No avg. pool 70.4% 78.8% 79.1% 77.4%
Ensemble 2 CNNs 73.1% 79.8% 84.5% 83.1%
Ensemble 3 CNNs 73.1% 79.3% 85.9% 83.5%
Ensemble 4 CNNs 72.1% 78.4% 85.8% 83.2%
Ensemble 5 CNNs 71.7% 78.3% 85.5% 83.2%
Table 4. Full scene material classiﬁcation results. Mean class
and total accuracy on the test set. When training, we optimize the
CRF parameters for mean class accuracy, but report both mean class
and total accuracy (mean accuracy across all examples). In one
experiment (a), we train and test only on segments; in a separate
experiment (b), we train and test only on clicks. Accuracies for
segments are averaged across all pixels that fall in that segment.
combined with either arithmetic or geometric mean.
Since we have two types of data, clicks andsegments , we
run two sets of experiments: (a) we train and test only on
segments, and in a separate experiment (b) we train and test
only on clicks. These two training objectives result in very
different behavior, as illustrated in Figure 9. In experiment
(a), the accuracy across segments are optimized, producing
clean boundaries. In experiment (b), the CRF maximizes
accuracy only at click locations, thus resulting in sloppy
boundaries. As shown in Table 4, the numerical scores for
the two experiments are also very different: segments are
more challenging than clicks. While clicks are sufﬁcient to
train a CNN, they are not sufﬁcient to train a CRF.
Focusing on segmentation accuracy, we see from Ta-
ble 4(a) that our best single model is GoogLeNet without
average pooling (6% better than with pooling). The best
ensemble is 2 CNNs: GoogLeNet (no average pooling) and
AlexNet (patch scale: 46.5%), combined with arithmetic
mean. Larger ensembles perform worse since we are aver-
7
Correct
Fabric (99%) Foliage (99%) Food (99%) Leather (99%)Correct
Metal (99%) Mirror (99%) Painted (99%) Plastic (99%)Incorrect
T: Wood T: Polished stone T: Water T: Fabric
P: Stone (90%) P: Water (35%) P: Carpet (27%) P: Foliage (67%)
Figure 10. High conﬁdence predictions. Top two rows: correct
predictions. Bottom row: incorrect predictions ( T: true, P: pre-
dicted). Percentages indicate conﬁdence (the predictions shown are
at least this conﬁdent). CNN: GoogLeNet.
aging worse CNNs. In Figure 8, we show example labeling
results on test images.
5.3. Comparing MINC to FMD
Compared to FMD, the size and diversity of MINC is
valuable for classifying real-world imagery. Table 5 shows
the effect of training on all of FMD and testing on MINC
(and vice versa). The results suggests that training on FMD
alone is not sufﬁcient for real-world classiﬁcation. Though it
may seem that our dataset is “easy,” since the best classiﬁca-
tions scores are lower for FMD than for MINC , we ﬁnd that
difﬁculty is in fact closely tied to dataset size (Section 5.1).
Taking 100 random samples per category, AlexNet achieves
54.20.7% on MINC (64.6 1.3% when considering only
the 10 FMD categories) and 66.5% on FMD.
5.4. Comparing CNNs with prior methods
Cimpoi [ 3] is the best prior material classiﬁcation method
on FMD. We ﬁnd that by replacing DeCAF with oversam-
pled AlexNet features we can improve on their FMD results.
We then show that on MINC , a ﬁnetuned CNN is even better.
To improve on [ 3], we take their SIFT IFV, combine it
with AlexNet fc7 features, and add oversampling [ 13] (see
supplemental for details). With a linear SVM we achieve
69.60.3% on FMD. Previous results are listed in Table 6.
Having found that SIFT IFV+fc7 is the new best on
FMD, we compare it to a ﬁnetuned CNN on a sub-
set of MINC (2500 patches per category, one patch per
photo). Fine-tuning AlexNet achieves 76.00.2% whereasTest
FMD MINC
TrainFMD 66.5% 26.1%
MINC 41.7% 85.0%(10 categories
in common)
Table 5. Cross-dataset experiments. We train on one dataset and
test on another dataset. Since MINC contains 23 categories, we
limit MINC to the 10 categories in common. CNN: AlexNet.
Method Accuracy Trials
Sharan et al. [25] 57.10.6% 14 splits
Cimpoi et al. [3] 67.10.4% 14 splits
Fine-tuned AlexNet 66.51.5% 5 folds
SIFT IFV+fc7 69.60.3% 10 splits
Table 6. FMD experiments. By replacing DeCAF features with
oversampled AlexNet features we improve on the best FMD result.
SIFT IFV+fc7 achieves 67.4 0.5% with a linear SVM
(oversampling, 5 splits). This experiment shows that
a ﬁnetuned CNN is a better method for MINC than
SIFT IFV+fc7.
6. Conclusion
Material recognition is a long-standing, challenging prob-
lem. We introduce a new large, open, material database,
MINC , that includes a diverse range of materials of every-
day scenes and staged designed interiors, and is at least an
order of magnitude larger than prior databases. Using this
large database we conduct an evaluation of recent deep learn-
ing algorithms for simultaneous material classiﬁcation and
segmentation, and achieve results that surpass prior attempts
at material recognition.
Some lessons we have learned are:
Training on a dataset which includes the surrounding
context is crucial for real-world material classiﬁcation.
Labeled clicks are cheap and sufﬁcient to train a CNN
alone. However, to obtain high quality segmentation
results, training a CRF on polygons results in much
better boundaries than training on clicks.
Many future avenues of work remain. Expanding the
dataset to a broader range of categories will require new
ways to mine images that have more variety, and new an-
notation tasks that are cost-effective. Inspired by attributes
for textures [ 3], in the future we would like to identify mate-
rial attributes and expand our database to include them. We
also believe that further exploration of joint material and ob-
ject classiﬁcation and segmentation will be fruitful [ 10] and
lead to improvements in both tasks. Our database, trained
models, and all experimental results are available online at
http://minc.cs.cornell.edu/.
Acknowledgements. This work was supported in part by
Google, Amazon AWS for Education, a NSERC PGS-D
scholarship, the National Science Foundation (grants IIS-
1149393, IIS-1011919, IIS-1161645), and the Intel Science
and Technology Center for Visual Computing.
8
References
[1]S. Bell, P. Upchurch, N. Snavely, and K. Bala. OpenSurfaces:
A richly annotated catalog of surface appearance. ACM Trans.
on Graphics (SIGGRAPH) , 32(4), 2013. 1, 2, 3
[2]B. Caputo, E. Hayman, and P. Mallikarjuna. Class-speciﬁc
material categorisation. In ICCV , pages 1597–1604, 2005. 2
[3]M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and
A. Vedaldi. Describing textures in the wild. In CVPR , pages
3606–3613. IEEE, 2014. 2, 8
[4]K. J. Dana, B. Van Ginneken, S. K. Nayar, and J. J. Koen-
derink. Reﬂectance and texture of real-world surfaces. ACM
Transactions on Graphics (TOG) , 18(1):1–34, 1999. 2
[5]M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn,
and A. Zisserman. The Pascal Visual Object Classes (VOC)
Challenge. IJCV , 88(2):303–338, June 2010. 3
[6]C. Farabet, C. Couprie, L. Najman, and Y . LeCun. Learning
hierarchical features for scene labeling. PAMI , 35(8):1915–
1929, 2013. 2
[7]R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich
feature hierarchies for accurate object detection and semantic
segmentation. In CVPR , 2014. 2
[8]S. Gould, R. Fulton, and D. Koller. Decomposing a scene
into geometric and semantically consistent regions. In ICCV ,
2009. 3
[9]E. Hayman, B. Caputo, M. Fritz, and J. olof Eklundh. On the
signiﬁcance of real-world conditions for material classiﬁca-
tion. In ECCV , 2004. 2
[10] D. Hu, L. Bo, and X. Ren. Toward robust material recognition
for everyday objects. In BMVC , pages 1–11. Citeseer, 2011.
1, 2, 8
[11] Y . Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Gir-
shick, S. Guadarrama, and T. Darrell. Caffe: Convolutional
architecture for fast feature embedding. In Proceedings of
the ACM International Conference on Multimedia , pages 675–
678. ACM, 2014. 6
[12] P. Kr ¨ahenb ¨uhl and V . Koltun. Parameter learning and con-
vergent inference for dense random ﬁelds. In ICML , pages
513–521, 2013. 2, 4, 5
[13] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classiﬁcation with deep convolutional neural networks. In
Advances in neural information processing systems , pages
1097–1105, 2012. 2, 5, 6, 8
[14] Y . LeCun, B. Boser, J. S. Denker, D. Henderson, R. E.
Howard, W. Hubbard, and L. D. Jackel. Backpropagation
applied to handwritten zip code recognition. Neural computa-
tion, 1(4):541–551, 1989. 2
[15] T. Leung and J. Malik. Representing and recognizing the vi-
sual appearance of materials using three-dimensional textons.
IJCV , 43(1):29–44, June 2001. 2
[16] W. Li and M. Fritz. Recognizing materials from virtual exam-
ples. In ECCV , pages 345–358. Springer, 2012. 2
[17] C. Liu, L. Sharan, E. H. Adelson, and R. Rosenholtz. Explor-
ing features in a bayesian framework for material recognition.
InCVPR , pages 239–246. IEEE, 2010. 1, 2
[18] M. Oquab, L. Bottou, I. Laptev, and J. Sivic. Learning and
transferring mid-level image representations using convolu-
tional neural networks. In CVPR , 2014. 2[19] G. Patterson, C. Xu, H. Su, and J. Hays. The SUN Attribute
Database: Beyond Categories for Deeper Scene Understand-
ing. IJCV , 108(1-2):59–81, 2014. 1
[20] X. Qi, R. Xiao, J. Guo, and L. Zhang. Pairwise rotation
invariant co-occurrence local binary pattern. In ECCV , pages
158–171. Springer, 2012. 1, 2
[21] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma,
Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg,
and L. Fei-Fei. ImageNet Large Scale Visual Recognition
Challenge, 2014. 1, 2
[22] B. C. Russell, A. Torralba, K. P. Murphy, and W. T. Free-
man. LabelMe: A database and web-based tool for image
annotation. IJCV , 77(1-3):157–173, May 2008. 3
[23] G. Schwartz and K. Nishino. Visual material traits: Rec-
ognizing per-pixel material context. In Proceedings of the
International Conference on Computer Vision Workshops (IC-
CVW) , pages 883–890. IEEE, 2013. 2
[24] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus,
and Y . LeCun. Overfeat: Integrated recognition, localization
and detection using convolutional networks. In International
Conference on Learning Representations (ICLR2014) . CBLS,
April 2014. 2, 5
[25] L. Sharan, C. Liu, R. Rosenholtz, and E. Adelson. Recog-
nizing materials using perceptually inspired features. IJCV ,
2013. 1, 8
[26] L. Sharan, R. Rosenholtz, and E. Adelson. Material percep-
tion: What can you see in a brief glance? Journal of Vision ,
9(8):784–784, 2009. 1, 2
[27] K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. arXiv preprint
arXiv:1409.1556 , 2014. 2, 6
[28] C. Szegedy, W. Liu, Y . Jia, P. Sermanet, S. Reed, D. Anguelov,
D. Erhan, V . Vanhoucke, and A. Rabinovich. Going deeper
with convolutions. CVPR , 2015. 2, 6
[29] R. Timofte and L. J. Van Gool. A training-free classiﬁcation
framework for textures, writers, and materials. In BMVC ,
pages 1–12, 2012. 2
[30] M. Varma and A. Zisserman. A statistical approach to texture
classiﬁcation from single images. IJCV , 62(1-2):61–81, Apr.
2005. 2
[31] J. Xiao, K. A. Ehinger, J. Hays, A. Torralba, and A. Oliva.
SUN Database: Exploring a large collection of scene cate-
gories. IJCV , 2014. 1, 3
[32] M. D. Zeiler and R. Fergus. Visualizing and understanding
convolutional networks. In ECCV , pages 818–833. Springer,
2014. 2
[33] S. Zheng, M.-M. Cheng, J. Warrell, P. Sturgess, V . Vineet,
C. Rother, and P. H. Torr. Dense semantic image segmentation
with objects and attributes. In CVPR , pages 3214–3221. IEEE,
2014. 2
[34] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva.
Learning deep features for scene recognition using Places
database. NIPS , 2014. 1, 3
9


Summary:
Contour detection has been a fundamental component in many image segmentation
and object detection systems. Most previous work utilizes low-level features
such as texture or saliency to detect contours and then use them as cues for a
higher-level task such as object detection. However, we claim that recognizing
objects and predicting contours are two mutually related tasks. Contrary to
traditional approaches, we show that we can invert the commonly established
pipeline: instead of detecting contours with low-level cues for a higher-level
recognition task, we exploit object-related features as high-level cues for
contour detection.
  We achieve this goal by means of a multi-scale deep network that consists of
five convolutional layers and a bifurcated fully-connected sub-network. The
section from the input layer to the fifth convolutional layer is fixed and
directly lifted from a pre-trained network optimized over a large-scale object
classification task. This section of the network is applied to four different
scales of the image input. These four parallel and identical streams are then
attached to a bifurcated sub-network consisting of two independently-trained
branches. One branch learns to predict the contour likelihood (with a
classification objective) whereas the other branch is trained to learn the
fraction of human labelers agreeing about the contour presence at a given point
(with a regression criterion).
  We show that without any feature engineering our multi-scale deep learning
approach achieves state-of-the-art results in contour detection.

Gedas Bertasius
Jianbo Shi
Lorenzo Torresani

Accepted to CVPR 2015

DeepEdge: A Multi-Scale Bifurcated Deep Network
for Top-Down Contour Detection
Gedas Bertasius
University of Pennsylvania
gberta@seas.upenn.eduJianbo Shi
University of Pennsylvania
jshi@seas.upenn.eduLorenzo Torresani
Dartmouth College
lt@dartmouth.edu
Abstract
Contour detection has been a fundamental component
in many image segmentation and object detection systems.
Most previous work utilizes low-level features such as tex-
ture or saliency to detect contours and then use them as cues
for a higher-level task such as object detection. However,
we claim that recognizing objects and predicting contours
are two mutually related tasks. Contrary to traditional ap-
proaches, we show that we can invert the commonly estab-
lished pipeline: instead of detecting contours with low-level
cues for a higher-level recognition task, we exploit object-
related features as high-level cues for contour detection.
We achieve this goal by means of a multi-scale deep net-
work that consists of ﬁve convolutional layers and a bifur-
cated fully-connected sub-network. The section from the in-
put layer to the ﬁfth convolutional layer is ﬁxed and directly
lifted from a pre-trained network optimized over a large-
scale object classiﬁcation task. This section of the network
is applied to four different scales of the image input. These
four parallel and identical streams are then attached to
a bifurcated sub-network consisting of two independently-
trained branches. One branch learns to predict the con-
tour likelihood (with a classiﬁcation objective) whereas the
other branch is trained to learn the fraction of human la-
belers agreeing about the contour presence at a given point
(with a regression criterion).
We show that without any feature engineering our multi-
scale deep learning approach achieves state-of-the-art re-
sults in contour detection.
1. Introduction
Contour detection is typically considered a low-level
problem, and used to aid higher-level tasks such as object
detection [1, 3, 31, 24]. However, it can be argued that
the tasks of detecting objects and predicting contours are
closely related. For instance, given the contours we can
easily infer which objects are present in the image. Con-
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 100.10.20.30.40.50.60.70.80.91
RecallPrecision

[F=.80] Human
[F=.753] DeepEdge
[F=.753] N4−fields [10]
[F=.747] MCG [2]
[F=.746] SE [8]
[F=.739] SCG [27]
[F=.737] PMI [14]
[F=.727] Sketch Tokens [19]
[F=.726] gPb−owt−ucm [1]Figure 1: Contour detection accuracy on the BSDS500
dataset. Our method attains higher average precision com-
pared to prior methods and state-of-the-art F-score. At low
recall, DeepEdge achieves nearly 100% precision..
versely, if we are given exact locations of the objects we
could predict contours just as easily. A commonly estab-
lished pipeline in computer vision starts with with low-level
contour prediction and then moves up to higher-level object
detection. However, since we claim that these two tasks are
mutually related, we propose to invert this process. Instead
of using contours as low-level cues for object detection, we
want to use object-speciﬁc information as high-level cues
for contour detection. Thus, in a sense our scheme can be
viewed as a top-down approach where object-level cues in-
form the low-level contour detection process.
In this work, we present a uniﬁed multi-scale deep learn-
ing approach that uses higher-level object information to
1arXiv:1412.1123v3  [cs.CV]  23 Apr 2015
predict contours. Speciﬁcally, we present a front-to-end
convolutional architecture where contours are learned di-
rectly from raw pixels. Our proposed deep learning archi-
tecture reuses features computed by the ﬁrst ﬁve convolu-
tional layers of the network of Krizhevsky et al. [17]. We re-
fer to this network as the KNet . Because the KNet has been
trained for object classiﬁcation, reusing its features enable
our method to incorporate object-level information for con-
tour prediction. In the experimental section, we will show
that this high-level object information greatly improves con-
tour detection results.
Furthermore, our deﬁned architecture operates on mul-
tiple scales simultaneously and combines local and global
information from the image, which leads to signiﬁcantly
improved contour detection accuracy rates.
We connect the features computed by the convolutional
layers of the KNet at four different scales of the input with
alearned subnetwork that bifurcates into two branches (the
architecture of our model is illustrated in Fig. 2).
What should the learning objective be? When a human
observer decides if a pixel is a boundary edge, a number
of supporting evidence is used with object level reasoning.
While it is impossible to record such information, we do
have the fraction of observers in agreement for each pixel.
We argue that a learning objective that predicts the fraction
of human labelers in agreement can mimic human reasoning
better.
Thus, in the bifurcated sub-network we optimize the two
branches with different learning objectives. The weights in
one branch are optimized with an edge classiﬁcation objec-
tive, while the other branch is trained to predict the frac-
tion of human labelers in agreement, i.e., using a regression
criterion. We show that predictions from the classiﬁcation
branch yield high edge recall, while the outputs of the re-
gression branch have high precision. Thus, fusing these two
outputs allows us to obtain excellent results with respect to
both metrics and produce state-of-the-art F-score and aver-
age precision.
In summary, the use of higher-level object features, inde-
pendent optimization of edge classiﬁcation and regression
objectives, as well as a uniﬁed multi-scale architecture are
the key characteristics that allow our method to achieve the
state-of-the-art in contour detection (see Fig. 1).
2. Related Work
Deep Learning. In the recent years, deep convolutional
networks have achieved remarkable results in a wide array
of computer vision tasks [32, 25, 33, 17]. However, thus
far, applications of convolutional networks focused on high-
level vision tasks such as face recognition, image classiﬁca-
tion, pose estimation or scene labeling [32, 25, 33, 17]. Ex-
cellent results in these tasks beg the question whether con-
volutional networks could perform equally well in lower-
Figure 2: Visualization of multi-scale DeepEdge network
architecture. To extract candidate contour points, we run the
Canny edge detector. Then, around each candidate point,
we extract patches at four different scales and simultane-
ously run them through the ﬁve convolutional layers of the
KNet [17]. We connect these convolutional layers to two
separately-trained network branches. The ﬁrst branch is
trained for classiﬁcation, while the second branch is trained
as a regressor. At testing time, the scalar outputs from these
two sub-networks are averaged to produce the ﬁnal score.
level vision tasks such as contour detection. In this paper,
we present a convolutional architecture that achieves state-
of-the-art results in a contour detection task, thus demon-
strating that convolutional networks can be applied success-
fully for lower-level vision tasks as well.
Edge Detection. Most of the contour detection methods
can be divided into two branches: local and global meth-
ods. Local methods perform contour detection by reason-
ing about small patches inside the image. Some recent
local methods include sketch tokens [18] and structured
edges [7], Both of these methods are trained in a super-
vised fashion using a random forest classiﬁer. Sketch to-
kens [18] pose contour detection as a multi-class classiﬁca-
tion task and predicts a label for each of the pixels individ-
ually. Structured edges [7], on the other hand, attempt to
predict the labels of multiple pixels simultaneously.
Global methods predict contours based on the informa-
tion from the full image. Some of the most successful ap-
proaches in this genre are the MCG detector [2], gPb de-
tector [1] and sparse code gradients [26]. While sparse
code gradients use supervised SVM learning [4], both gPb
and MCG rely on some form of spectral methods. Other
spectral-based methods include Normalized Cuts [30] and
PMI [13].
Recently, there have also been attempts to apply deep
learning methods to the task of contour detection. While
SCT [20] is a sparse coding approach, both N4ﬁelds [9]
and DeepNet [16] use Convolutional Neural Networks
(CNNs) to predict contours. N4ﬁelds rely on dictionary
a) Input Image
 c)2ndlayer
 d)3rdlayer
 e)4thlayer
 e)5thlayer
Figure 3: Visualization of the activation values at the selected convolutional ﬁlters of the KNet (ﬁlters are resized to the
original image dimensions). The ﬁlters in the second layer ﬁre on oriented edges inside the image. The third and fourth
convolutional layers produce an outline of the shape of the object. The ﬁfth layer ﬁres on the speciﬁc parts of the object.
learning and the use of the Nearest Neighbor algorithm
within a CNN framework while DeepNet uses a traditional
CNN architecture to predict contours.
In comparison to these prior approaches, our work offers
several contributions. First, we deﬁne a novel multi-scale
bifurcated CNN architecture that enables our network to
achieve state-of-the-art contour detection results. Second,
we avoid manual feature engineering by learning contours
directly from raw data. Finally, we believe that we are the
ﬁrst to propose the use of high-level object features for con-
tour detection, thus inverting the traditional pipeline rely-
ing on low-level cues for mid-level feature extraction. Our
experiments show that this top-down approach for contour
detection yields state-of-the-art results.
3. The DeepEdge Network
In this section, we describe our proposed deep learn-
ing approach for contour detection. For simplicity, we ﬁrst
present our architecture in the single-scale scenario (subsec-
tion 3.1) and then discuss how to take advantage of multiple
scales (subsection 3.2).
3.1. Single-Scale Architecture
Selection of Candidate Edge Points. To extract a set
of candidate contours with high recall we apply the Canny
edge detector [5] to the input image. For each of these
points we then extract a patch of ﬁxed size such that our
candidate point is the center of the patch. Patches that do
not ﬁt into the image boundaries are padded with the mirror
reﬂections of itself.
Extraction of High-Level Features. We then resize the
patch of ﬁxed size to match the input dimensions of the
KNet [17] and use this network to extract object-level fea-
tures. The KNet is an appropriate model for our setting as it
has been trained over a large number of object classes (the
1000 categories of the ImageNet dataset [28]) and thus cap-
tures features that are generic and useful for many object
categories. While such features have been optimized for the
task of object class recognition, they have been shown to
be highly effective for other image-analysis tasks, includ-ing object detection [10], attribute prediction [34], and im-
age style recognition [15]. The network was trained on 1:2
million images and it includes more than 60million param-
eters. Its architecture consists of 5convolutional layers and
3fully connected layers. As we intend to use the KNet as
a feature extractor for boundary detection, we utilize only
the ﬁrst 5convolutional layers, which preserve explicit lo-
cation information before the “spatial scrambling” of the
fully connection layers (note that a spatially variant repre-
sentation is crucially necessary to predict the presence of
contours at individual pixels). The ﬁrst two KNet convo-
lutional layers learn low-level information. As we move
into the deeper layers, however, we observe that the network
learns higher-level object information. The second convolu-
tional layer seems to encode coherent edge structures. The
third convolutional layer ﬁres at locations corresponding to
prototypical object shapes. The fourth layer appears to gen-
erate high responses for full shapes of the object, whereas
the ﬁfth layer ﬁres on the speciﬁc object parts (See Fig. 3).
In order to obtain a representation that captures this hier-
archical information, we perform feature extraction at each
of the ﬁve convolutional layers, as shown in Fig. 5. Specif-
ically, we consider a small sub-volume of the feature map
stack produced at each layer. The sub-volume is centered
at the center of the patch in order to assess the presence of
a contour in a small area around the candidate point. We
then perform max,average , and center pooling on this sub-
volume. This yields a feature descriptor of size 3Fwhere
Fis the number of feature maps computed by the convolu-
tional layer. While max and average pooling are well estab-
lished operations in deep learning, we deﬁne center pooling
as selecting the center-value from each of the feature maps.
The motivation for center pooling is that for each candidate
point we want to predict the contour presence at that partic-
ular point. Because the candidate point is located at the cen-
ter of the input patch, center pooling extracts the activation
value from the location that corresponds to our candidate
point location.
A Bifurcated Sub-Network. We connect the feature
maps computed via pooling from the ﬁve convolutional
layers to two separately-trained network branches. Each
Figure 4: A few samples of ground truth data illustrating the difference between the classiﬁcation (ﬁrst row) and the regression
(second row) objectives. The classiﬁcation branch is trained to detect contours that are marked by at least one of the human
annotators. Conversely, the regression branch is optimized to the contour values that depict the fraction of human annotators
agreeing on the contour.
branch consists of two fully-connected layers. The ﬁrst
branch is trained using binary labels, i.e., to perform con-
tour classiﬁcation. This branch is making less selective pre-
dictions by classifying whether a given point is a contour or
not. In a sense, this classiﬁcation branch abstracts details
related to the edge structure (orientation, strength, etc) and
simply tries to predict the presence/absence of an edge at
a given point. Due to such abstractions, the classiﬁcation
branch produces contour predictions with high recall.
The second branch is optimized as a regressor to predict
the fraction of human labelers agreeing about the contour
presence at a particular point. Due to a regression objective,
this branch is much more selective than the ﬁrst branch. In-
tuitively, the second branch is trained to learn the structural
differences between the contours that are marked by a dif-
ferent fraction of human labelers. For instance, the area that
was labeled as a contour by 80% of human labelers must
be signiﬁcantly different than the area that was labeled as
a contour by 20% human labelers. The regression branch
is trying to learn such differences by predicting the frac-
tion of human labelers who would mark a particular point
as a contour. Thus, in a sense, we are training the regres-
sion branch to implicitly mimic how human labelers rea-
son about the contour presence at a given point. In the ex-
perimental section, we demonstrate that due to its selectiv-
ity, the regression branch produces contour predictions with
very high precision. In Fig. 4, we present several samples
of ground truth data that illustrate the different properties of
our two end-objectives.
The number of hidden layers in the ﬁrst and second fully
connected layers of both branches are 1024 and512, re-
spectively. Both branches optimize the sum of squared dif-
ference loss over the (binary or continuous) labels. At test-ing time, the scalar outputs computed from these two sub-
networks are averaged to produce a ﬁnal score indicative of
the probability that the candidate point is a contour. Visual-
ization of this architecture is presented in Fig. 5.
In order to train our sub-network, we generate patch ex-
amples and labels using training images with ground truth
annotations from multiple human labelers. To generate the
binary labels, we ﬁrst sample 40;000positive examples that
were marked as contours by at least one of the labelers. To
generate negative examples we consider the points that were
selected as candidate contour points by the Canny edge de-
tector but that have not been marked as contours by any of
the human labelers. These are essentially false positives.
For training, we use a random subset of 40;000 of such
points in order to have equal proportion of negative and
positive examples. These 80;000examples are then used
to train our classiﬁcation sub-network.
In addition to the binary labels, we also generate con-
tinuous labels that are used to train the regression network.
For this purpose, we deﬁne the regression label of a point to
be the fraction of human labelers that marked the point as a
contour. These 80;000examples with continuous labels are
then also used to train our regression sub-network.
3.2. Multi-Scale Architecture
In the previous section we presented a convolutional ar-
chitecture for contour prediction utilizing a single scale.
However, in practice, we found that a multi-scale approach
works much better. In this section, we show how to modify
the single-scale architecture so that it can exploit multiple
scales simultaneously.
Rather than extracting a patch at a single scale as we did
in the previous section, in a multi-scale setting we extract
|{z}
|{z}Fixed WeightsLearned WeightsClassiﬁcation Branch
Regression Branch
}
}Figure 5: Detailed illustration of our proposed architecture in a single-scale setting. First, an input patch, centered around the
candidate point, goes through ﬁve convolutional layers of the KNet . To extract high-level features, at each convolutional layer
we extract a small sub-volume of the feature map around the center point, and perform max,average , and center pooling on
this sub-volume. The pooled values feed a bifurcated sub-network. At testing time, the scalar outputs computed from the
branches of a bifurcated sub-networks are averaged to produce a ﬁnal contour prediction.
patches around the candidate point for different patch sizes
so that they cover different spatial extents of the image. We
then resize the patches to ﬁt the KNet input and pump them
in parallel through the ﬁve convolutional layers. Our high-
level features are then built by performing max, average and
center pooling in a small sub-volume of the feature map at
each convolutional layer and at each scale. This effectively
increases the dimensionality of the feature vector by a factor
equal to the number of scales compared to the single-scale
setting. These pooled features are then connected as before
to the two separately-trained network branches. A visual-
ization of our multi-scale architecture is shown in Fig. 2.
3.3. Implementation Details
In this section, we describe additional implementation
details of our model. Our deep network is implemented us-
ing the software library Caffe [14].
We use four different scales for our patches. The sizes
of these patches are 6464;128128;196196and a
full-sized image. All of the patches are then resized to the
KNet input dimensions of 227227.
When extracting high-level features from the convolu-
tional layers of KNet , we use sub-volumes of convolutionalfeature maps having spatial sizes 77;55;33;33,
and33for the convolutional layers 1;2;3;4;5, respec-
tively. Note that we shrink the size of the subvolume as we
go deeper in the network since the feature maps get smaller
due to pooling. Our choice of subvolume sizes is made to
ensure we are roughly considering the same spatial extent
of the original image at each layer.
As illustrated in Fig. 5, during the training the weights
in the convolutional layers are ﬁxed and only the weights in
the fully connected layers of the two branches are learned.
To train our model we use the learning rate of 0:1, the
dropout fraction of 0:5,50number of epochs, and the size
of the batch equal to 100.
As described earlier, to train classiﬁcation and regres-
sion branches we sample 80;000examples with binary la-
bels. We also generate continuous labels for these 80;000
examples. In addition, we sample a hold-out dataset of size
40;000. This hold-out dataset is used for the hard-positive
mining step [22].
For the ﬁrst 25epochs we train classiﬁcation and regres-
sion branches independently on the original 80;000sam-
ple training dataset. After the ﬁrst 25epochs, we test both
branches on the hold-out dataset and detect false negative
predictions made by each branch. We then use these false
negative examples along with the same number of randomly
selected true negative examples to augment our original
80;000training dataset. For the remaining 25epochs, we
train both branches on this augmented dataset.
The motivation for the hard-positive mining step is to re-
duce the number of false negative predictions produced by
both branches. By augmenting the original 80;000sized
training data with false negative examples, we are forcing
both branches to focus on hard positive examples, and thus,
effectively reducing the number of false negative predic-
tions.
4. Experiments
In this section, we present our results on the BSDS500
dataset [23], which is arguably the most established bench-
mark for contour detection. This dataset contains 200train-
ing images, 100 validation images, and 200 test images.
Contour detection accuracy is evaluated using three stan-
dard measures: ﬁxed contour threshold (ODS), per-image
best threshold (OIS), and average precision (AP).
In section 4.1 we quantitatively compare our approach
to the state-of-the-art. In sections 4.2-4.5 we study how the
performance of our system changes as we modify some of
the architecture choices (number of scales, feature maps,
pooling scheme, training objective). This will cast addi-
tional insight into the factors that critically contribute to the
high accuracy of our system.
4.1. Comparison with Other Methods
We compare the results produced by our approach and
previously proposed contour detection methods. Table 1
summarizes the results. We note that our algorithm achieves
contour detection accuracy that is higher or equal to state-
of-the-art results according to two of the three metrics.
Fig. 1 shows the precision and recall curve for the meth-
ods considered in our comparison. It also lists the F-score
for each method (in the legend). We observe that there is the
accuracy margin separating our approach from prior tech-
niques. In particular, for low-recall our method achieves al-
most perfect precision rate. It also produces state-of-the-art
F-score.
4.2. Single Scale versus Multiple Scales
In this section we study the beneﬁts of a multi-scale ar-
chitecture. Results in Table 2 report accuracy for differ-
ent numbers and choices of scales. The ﬁrst four rows in
the table illustrate the results achieved using a single-scale
approach. Speciﬁcally, these four cases show performance
obtained when training and testing our system with an input
patch of size 6464;128128;196196or a full-sized im-
age, respectively. Note that adding information from multi-
ple scales leads to signiﬁcantly higher F-scores and higherMethod ODS OIS AP
Felz., Hutt. [8] 0.610 0.640 0.560
Mean Shift [6] 0.640 0.680 0.560
Ncuts [30] 0.640 0.680 0.450
SCT [20] 0.710 0.720 0.740
gPb-owt-ucm [1] 0.726 0.757 0.696
Sketch Tokens [18] 0.727 0.746 0.780
PMI [13] 0.737 0.771 0.783
DeepNet [16] 0.738 0.759 0.758
SCG [26] 0.739 0.758 0.773
SE [7] 0.746 0.767 0.803
MCG [2] 0.747 0.779 0.759
N4-ﬁelds [9] 0.753 0.769 0.784
DeepEdge 0.753 0.772 0.807
Table 1: Edge detection results on the BSDS500 bench-
mark. Our DeepEdge method achieves state-of-the-art con-
tour detections results according to both F-score and AP
metrics.
Scale ODS OIS AP
64 0.71 0.73 0.76
128 0.72 0.74 0.78
196 0.71 0.73 0.76
Full Image 0.67 0.69 0.57
64, 128 0.72 0.75 0.78
64, 128,196 0.72 0.75 0.78
64,128,196,Full Image 0.75 0.77 0.81
Table 2: Results illustrating the effect of using a multi-scale
architecture. Considering multiple scales for contour detec-
tion yields signiﬁcantly higher accuracy relative to a single
scale approach.
average precisions. Thus, these results suggest that a multi-
scale approach is highly advantageous in comparison to a
single scale setting.
4.3. Advantages of Higher-Level Features
In this section, we examine the validity of our earlier
claim that higher-level object-features enhance contour de-
tection accuracy. In Table 3, we present individual contour
detection results using features from the different convolu-
tional layers of KNet . Note that the 4thconvolutional layer
produces the most effective features when considering one
layer at a time. From our earlier discussion we know that
the4thconvolutional layer encodes higher-level object in-
formation related to shape and speciﬁc object parts. This
indicates that object speciﬁc cues are particularly beneﬁcial
for contour detection accuracy.
We also observe that by incorporating features from all
Input Image
 Canny Edges
 Raw DeepEdges
 Thresholded DeepEdges
 Ground Truth Edges
Figure 6: Qualitative results produced by our method. Notice how our method learns to distinguish between strong and
weak contours. For instance, in the last row of predictions, contours corresponding to zebra stripes are assigned much lower
probabilities than contours that correspond to the actual object boundaries separating the zebras from the background.
Conv. Layers ODS OIS AP
1st0.66 0.68 0.69
2nd0.71 0.74 0.76
3rd0.74 0.75 0.79
4th0.74 0.76 0.79
5th0.73 0.74 0.77
All 0.75 0.77 0.81
Table 3: This table shows the advantage of using higher-
level features from the KNet convolutional layers. Individ-
ually, the 4thconvolutional layer produces the best contour
prediction results, which implies that higher-level object in-
formation is indeed beneﬁcial for contour detection. Com-
bining the features from all convolutional layers leads to
state-of-the-art results.
the convolutional layers, our method achieves state-of-the-
art contour detection results. This suggests that the features
computed by different layers are complementary and that
considering information from the entire hierarchy is advan-
tageous.4.4. Pooling Scheme
When presenting the architecture of our model, we dis-
cussed three different types of pooling: max,average , and
center pooling. These three techniques were used to pool
the values from the sub-volumes extracted around the cen-
ter point in each convolutional ﬁlter as illustrated in Fig. 5.
We now show how each type of pooling affects contour
detection results. Table 4 illustrates that, individually, cen-
ter pooling yields the best contour detection results. This is
expected because the candidate point for which we are try-
ing to predict a contour probability is located at the center
of the input patch.
However, we note that combining all three types of pool-
ing, achieves better contour detection results than any single
pooling technique alone.
4.5. Bifurcation and Training Objective
Next, we want to show that that the two independently-
trained classiﬁcation and regression branches in the bifur-
cated sub-network provide complementary information that
yields improved contour detection accuracy. In Table 5, we
present contour detection results achieved by using predic-
tions from the individual branches of the bifurcated sub-
network.
Pooling Type ODS OIS AP
Average 0.73 0.75 0.78
Max 0.69 0.72 0.73
Center 0.74 0.76 0.8
Avg+Max+Cen 0.75 0.77 0.81
Table 4: Effect of different pooling schemes on contour de-
tection results. Center pooling produces better results than
max or average pooling. Combining all three types of pool-
ing further improves the results.
Branch ODS OIS AP
Classiﬁcation 0.75 0.76 0.78
Regression 0.74 0.76 0.80
Classiﬁcation+Regression 0.75 0.77 0.81
Table 5: Contour detection accuracy of the two branches
in our bifurcated sub-network. The classiﬁcation branch
yields solid F-score results whereas the regression branch
achieves high average precision. Averaging the outputs
from these two branches further improve the results.
From these results, we observe that using predictions
just from the classiﬁcation branch produces high F-score
whereas using predictions only from the regression branch
yields high average precision. Combining the predictions
from both branches improves the results according to both
metrics thus, supporting our claim that separately optimiz-
ing edge classiﬁcation and regression objectives is beneﬁ-
cial to contour detection.
4.6. Qualitative Results
Finally, we present qualitative results produced by our
method. In Figure 6 we show for each input image exam-
ple, the set of candidate points produced by the Canny edge
detector, the un-thresholded predictions of our method, the
thresholded predictions, and the ground truth contour map
computed as an average of the multiple manual annotations.
To generate the thresholded predictions, we use a probabil-
ity threshold of 0:5.
Note that our method successfully distinguishes between
strong and weak contours. Speciﬁcally, observe that in the
last row of Figure 6, our method assigns lower probability
to contours corresponding to zebra stripes compared to the
contours of the actual object boundary separating the zebras
from the background. Thus, in the thresholded version of
the prediction, the weak contours inside the zebra bodies
are removed and we obtain contour predictions that look
very similar to the ground truth.
Due to locality of our method, it may be beneﬁcial
to apply spectral methods [21, 30] or conditional randomﬁelds [27] on top of our method to further improve its per-
formance.
4.7. Computational Cost
In its current form, our method requires about 60KKNet
evaluations ( 15Kper scale) to extract the features. Based
on the runtimes reported in [14], if executed on a GPU our
method would take about 5 minutes and could be made
faster using the approach described in [12].
An alternative way to dramatically reduce the runtime of
DeepEdge is to interpolate the entries of the feature maps
produced by applying the KNet to the full image rather than
to individual patches. Such an approach would reduce the
number of CNN evaluations needed from 60Kto4(one
for each scale), which would allow our method to run in
real time even on CPUs. We note that interpolation of fea-
tures in deep layers has been used successfully in several
recent vision papers [29, 11, 19]. Thus, we believe that such
an approach could yield nearly equivalent contour detection
accuracy, up to a small possible degradation caused by in-
terpolation.
Since in this work we were primarily interested in study-
ing the effective advantage enabled by object-level features
in contour detection, we have not invested any effort in opti-
mizing the implementation of our method. This will be one
of our immediate goals in the future.
5. Conclusions
In this work, we presented a multi-scale bifurcated deep
network for top-down contour detection. In the past, con-
tour detection has been approached as a bottom-up task
where low-level features are engineered ﬁrst, then contour
detection is performed, and ﬁnally contours may be used
as cues for object detection. However, due to a close re-
lationship between object and contour detection tasks, we
proposed to invert this pipeline and perform contour detec-
tion in a top-down fashion. We demonstrated how to use
higher-level object cues to predict contours and showed that
considering higher-level object-features leads to a substan-
tial gain in contour detection accuracy.
Additionally, we demonstrated that our multi-scale ar-
chitecture is beneﬁcial to contour prediction as well. By
considering multiple scales, our method incorporates local
and global information around the candidate contour points,
which leads to signiﬁcantly better contour detection results.
Furthermore, we showed that independent optimization of
contour classiﬁcation and regression objectives improves
contour prediction accuracy as well. As our experiments in-
dicate, DeepEdge achieves higher average precision results
compared to any prior or concurrent work.
In conclusion, our results suggest that pure CNN sys-
tems can be applied successfully to contour detection and
possibly to many other low-level vision tasks.
6. Acknowledgements
We thank Piotr Teterwak, Du Tran and Mohammad Haris
Baig for helpful discussions. This research was funded in
part by NSF award CNS-1205521.
References
[1] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. Contour
detection and hierarchical image segmentation. IEEE Trans.
Pattern Anal. Mach. Intell. , 33(5):898–916, May 2011. 1, 2,
6
[2] P. Arbelaez, J. Pont-Tuset, J. Barron, F. Marqu ´es, and J. Ma-
lik. Multiscale combinatorial grouping. In Computer Vision
and Pattern Recognition (CVPR) , 2014. 2, 6
[3] E. Borenstein. Combining top-down and bottom-up segmen-
tation. In In Proceedings IEEE workshop on Perceptual Or-
ganization in Computer Vision, CVPR , page 46, 2004. 1
[4] C. J. C. Burges. A tutorial on support vector machines for
pattern recognition. Data Mining and Knowledge Discovery ,
2:121–167, 1998. 2
[5] J. Canny. A computational approach to edge detection. IEEE
Trans. Pattern Anal. Mach. Intell. , 8(6):679–698, June 1986.
3
[6] D. Comaniciu, P. Meer, and S. Member. Mean shift: A robust
approach toward feature space analysis. IEEE Transactions
on Pattern Analysis and Machine Intelligence , 24:603–619,
2002. 6
[7] P. Doll ´ar and C. L. Zitnick. Fast edge detection using struc-
tured forests. PAMI , 2015. 2, 6
[8] P. F. Felzenszwalb and D. P. Huttenlocher. Efﬁcient
graph-based image segmentation. Int. J. Comput. Vision ,
59(2):167–181, Sept. 2004. 6
[9] Y . Ganin and V . S. Lempitsky. N4-ﬁelds: Neural network
nearest neighbor ﬁelds for image transforms. ACCV , 2014.
2, 6
[10] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich fea-
ture hierarchies for accurate object detection and semantic
segmentation. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition (CVPR) , 2014. 3
[11] B. Hariharan, P. A. Arbel ´aez, R. B. Girshick, and J. Malik.
Hypercolumns for object segmentation and ﬁne-grained lo-
calization. CoRR , abs/1411.5752, 2014. 8
[12] F. N. Iandola, M. W. Moskewicz, S. Karayev, R. B. Girshick,
T. Darrell, and K. Keutzer. Densenet: Implementing efﬁcient
convnet descriptor pyramids. CoRR , abs/1404.1869, 2014. 8
[13] P. Isola, D. Zoran, D. Krishnan, and E. H. Adelson. Crisp
boundary detection using pointwise mutual information. In
ECCV , 2014. 2, 6
[14] Y . Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Gir-
shick, S. Guadarrama, and T. Darrell. Caffe: Convolu-
tional architecture for fast feature embedding. arXiv preprint
arXiv:1408.5093 , 2014. 5, 8
[15] S. Karayev, A. Hertzmann, H. Winnemoeller, A. Agar-
wala, and T. Darrell. Recognizing image style. CoRR ,
abs/1311.3715, 2013. 3[16] J. J. Kivinen, C. K. Williams, N. Heess, and D. Technolo-
gies. Visual boundary prediction: A deep neural prediction
network and quality dissection. AISTATS , 1(2):9, 2014. 2, 6
[17] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classiﬁcation with deep convolutional neural networks. In
F. Pereira, C. Burges, L. Bottou, and K. Weinberger, edi-
tors, Advances in Neural Information Processing Systems 25 ,
pages 1097–1105. Curran Associates, Inc., 2012. 2, 3
[18] J. Lim, C. L. Zitnick, and P. Doll ´ar. Sketch tokens: A learned
mid-level representation for contour and object detection. In
CVPR , 2013. 2, 6
[19] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional
networks for semantic segmentation. CoRR , abs/1411.4038,
2014. 8
[20] M. Maire, S. X. Yu, and P. Perona. Reconstructive sparse
code transfer for contour detection and semantic labeling. In
Asian Conference on Computer Vision (ACCV) , 2014. 2, 6
[21] J. Malik, S. Belongie, T. Leung, and J. Shi. Contour and tex-
ture analysis for image segmentation. Int. J. Comput. Vision ,
43(1):7–27, June 2001. 8
[22] T. Malisiewicz, A. Gupta, and A. A. Efros. Ensemble of
exemplar-svms for object detection and beyond. In ICCV ,
2011. 5
[23] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database
of human segmented natural images and its application to
evaluating segmentation algorithms and measuring ecologi-
cal statistics. In Proc. 8th Int’l Conf. Computer Vision , vol-
ume 2, pages 416–423, July 2001. 6
[24] A. Opelt and A. Zisserman. A boundary-fragment-model for
object detection. In In ECCV , pages 575–588, 2006. 1
[25] P. H. O. Pinheiro and R. Collobert. Recurrent convolutional
neural networks for scene parsing. CoRR , abs/1306.2795,
2013. 2
[26] X. Ren and L. Bo. Discriminatively Trained Sparse Code
Gradients for Contour Detection. In Advances in Neural In-
formation Processing Systems , December 2012. 2, 6
[27] X. Ren, C. C. Fowlkes, and J. Malik. Scale-invariant contour
completion using condition random ﬁelds. Technical report.
8
[28] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,
S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein,
A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual
Recognition Challenge, 2014. 3
[29] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus,
and Y . LeCun. Overfeat: Integrated recognition, localiza-
tion and detection using convolutional networks. CoRR ,
abs/1312.6229, 2013. 8
[30] J. Shi and J. Malik. Normalized cuts and image segmenta-
tion. IEEE Transactions on Pattern Analysis and Machine
Intelligence , 22:888–905, 1997. 2, 6, 8
[31] J. Shotton. Contour-based learning for object detection. In
In Proc. ICCV , pages 503–510, 2005. 1
[32] Y . Taigman, M. Yang, M. Ranzato, and L. Wolf. Deepface:
Closing the gap to human-level performance in face veriﬁca-
tion. In Conference on Computer Vision and Pattern Recog-
nition (CVPR) , 2014. 2
[33] A. Toshev and C. Szegedy. Deeppose: Human pose estima-
tion via deep neural networks. CoRR , abs/1312.4659, 2013.
2
[34] N. Zhang, M. Paluri, M. Ranzato, T. Darrell, and L. Bourdev.
Panda: Pose aligned networks for deep attribute modeling.
CoRR , abs/1311.5591, 2013. 3